K3s can automatically provision PersistentVolumeClaims (PVCs) for local storage, but it doesn’t create the underlying host directories for you.
Let’s see K3s in action. Imagine you have a simple Nginx deployment that needs persistent storage for its configuration.
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: nginx-config-pvc
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-deployment
spec:
replicas: 1
selector:
matchLabels:
app: nginx
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: nginx:latest
ports:
- containerPort: 80
volumeMounts:
- name: nginx-config-volume
mountPath: /etc/nginx/conf.d
volumes:
- name: nginx-config-volume
persistentVolumeClaim:
claimName: nginx-config-pvc
When you apply this YAML to your K3s cluster:
kubectl apply -f nginx-app.yaml
Kubernetes will attempt to find a PersistentVolume that matches the nginx-config-pvc’s requirements. Since we’re talking about local storage, K3s has a built-in local-path provisioner. This provisioner looks for a specific directory on the host node to create the actual storage. By default, this is /var/lib/rancher/k3s/storage/.
The local-path provisioner will see the nginx-config-pvc, create a corresponding PersistentVolume object, and then create a directory on the node where the pod is scheduled. If the pod is scheduled on node-1, K3s will create /var/lib/rancher/k3s/storage/nginx-config-pvc-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx on node-1. The PVC will then be bound to this PV.
The problem arises if the node doesn’t have the /var/lib/rancher/k3s/storage/ directory (or if it’s not writable by the K3s agent). The local-path provisioner will fail to create the directory, and your PVC will remain in a Pending state.
The mental model here is that K3s’s local-path provisioner is a controller that watches for PVCs requesting local storage. When it sees one, it creates a corresponding PersistentVolume and attempts to create a directory on the node’s filesystem to back that PV. It’s not magic; it needs a place to put the data.
The key levers you control are:
storageClassName: You can explicitly set this in your PVC definition if you have multiple local storage configurations or if you want to ensure you’re using thelocal-pathprovisioner. The default is usuallylocal-path.- Node Directory: The root directory on the host node where K3s expects to create storage. This is configurable.
- Pod Scheduling: The
local-pathprovisioner binds a PV to a node when the PVC is created. If your pod is rescheduled to a different node, K3s will try to create the directory on the new node.
The one thing most people don’t realize is that the local-path provisioner doesn’t automatically ensure the parent directory (/var/lib/rancher/k3s/storage/ by default) exists and has the correct permissions before it tries to create subdirectories. It assumes the base path is set up correctly.
To fix a PVC stuck in Pending due to local path issues, you’d typically:
-
Check PVC Status:
kubectl get pvc nginx-config-pvcYou’ll likely see
Status: Pending. -
Check Events:
kubectl describe pvc nginx-config-pvcLook for events indicating a failure to create a directory or a permission denied error.
-
Verify Node Directory: SSH into the node where the pod is scheduled (or where K3s is trying to provision storage). Check if
/var/lib/rancher/k3s/storage/exists. If it doesn’t, create it:sudo mkdir -p /var/lib/rancher/k3s/storage/ -
Check Permissions: Ensure the K3s agent user (often
rootor a specific user likek3s) has write permissions to this directory.sudo chown -R k3s:k3s /var/lib/rancher/k3s/storage/ # Or if running as root: sudo chmod -R 755 /var/lib/rancher/k3s/storage/The exact user/group might depend on your K3s installation. You can often find it by checking the K3s service file or the processes running.
-
Recreate PVC (if necessary): Sometimes, deleting and reapplying the PVC is needed for the provisioner to retry.
kubectl delete pvc nginx-config-pvc kubectl apply -f nginx-app.yaml
If you encounter issues with the local-path provisioner, the next thing you’ll likely run into is managing storage across multiple nodes or dealing with data persistence beyond the lifecycle of a single K3s agent.