K3s can automatically provision PersistentVolumeClaims (PVCs) for local storage, but it doesn’t create the underlying host directories for you.

Let’s see K3s in action. Imagine you have a simple Nginx deployment that needs persistent storage for its configuration.

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: nginx-config-pvc
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:latest
        ports:
        - containerPort: 80
        volumeMounts:
        - name: nginx-config-volume
          mountPath: /etc/nginx/conf.d
      volumes:
      - name: nginx-config-volume
        persistentVolumeClaim:
          claimName: nginx-config-pvc

When you apply this YAML to your K3s cluster:

kubectl apply -f nginx-app.yaml

Kubernetes will attempt to find a PersistentVolume that matches the nginx-config-pvc’s requirements. Since we’re talking about local storage, K3s has a built-in local-path provisioner. This provisioner looks for a specific directory on the host node to create the actual storage. By default, this is /var/lib/rancher/k3s/storage/.

The local-path provisioner will see the nginx-config-pvc, create a corresponding PersistentVolume object, and then create a directory on the node where the pod is scheduled. If the pod is scheduled on node-1, K3s will create /var/lib/rancher/k3s/storage/nginx-config-pvc-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx on node-1. The PVC will then be bound to this PV.

The problem arises if the node doesn’t have the /var/lib/rancher/k3s/storage/ directory (or if it’s not writable by the K3s agent). The local-path provisioner will fail to create the directory, and your PVC will remain in a Pending state.

The mental model here is that K3s’s local-path provisioner is a controller that watches for PVCs requesting local storage. When it sees one, it creates a corresponding PersistentVolume and attempts to create a directory on the node’s filesystem to back that PV. It’s not magic; it needs a place to put the data.

The key levers you control are:

  • storageClassName: You can explicitly set this in your PVC definition if you have multiple local storage configurations or if you want to ensure you’re using the local-path provisioner. The default is usually local-path.
  • Node Directory: The root directory on the host node where K3s expects to create storage. This is configurable.
  • Pod Scheduling: The local-path provisioner binds a PV to a node when the PVC is created. If your pod is rescheduled to a different node, K3s will try to create the directory on the new node.

The one thing most people don’t realize is that the local-path provisioner doesn’t automatically ensure the parent directory (/var/lib/rancher/k3s/storage/ by default) exists and has the correct permissions before it tries to create subdirectories. It assumes the base path is set up correctly.

To fix a PVC stuck in Pending due to local path issues, you’d typically:

  1. Check PVC Status:

    kubectl get pvc nginx-config-pvc
    

    You’ll likely see Status: Pending.

  2. Check Events:

    kubectl describe pvc nginx-config-pvc
    

    Look for events indicating a failure to create a directory or a permission denied error.

  3. Verify Node Directory: SSH into the node where the pod is scheduled (or where K3s is trying to provision storage). Check if /var/lib/rancher/k3s/storage/ exists. If it doesn’t, create it:

    sudo mkdir -p /var/lib/rancher/k3s/storage/
    
  4. Check Permissions: Ensure the K3s agent user (often root or a specific user like k3s) has write permissions to this directory.

    sudo chown -R k3s:k3s /var/lib/rancher/k3s/storage/
    # Or if running as root:
    sudo chmod -R 755 /var/lib/rancher/k3s/storage/
    

    The exact user/group might depend on your K3s installation. You can often find it by checking the K3s service file or the processes running.

  5. Recreate PVC (if necessary): Sometimes, deleting and reapplying the PVC is needed for the provisioner to retry.

    kubectl delete pvc nginx-config-pvc
    kubectl apply -f nginx-app.yaml
    

If you encounter issues with the local-path provisioner, the next thing you’ll likely run into is managing storage across multiple nodes or dealing with data persistence beyond the lifecycle of a single K3s agent.

Want structured learning?

Take the full K3s course →