Fix Kubernetes Pod Eviction from Disk Pressure (2026)

Your pods are getting kicked out of Kubernetes because the nodes they’re running on are running out of disk space. This is happening because the kubelet, the agent on each node that manages pods, is configured to evict pods when disk usage on critical partitions hits a certain threshold, preventing the node from becoming completely unusable.

Here are the common reasons and how to fix them:

1. Excessive Container Image Cache

Diagnosis: Check image usage on the node:

sudo du -sh /var/lib/docker/image/*

(If using containerd, replace /var/lib/docker/image with /var/lib/containerd/images)

Cause: Over time, the node accumulates many container images that are no longer in use by any pods. Docker (or containerd) keeps these images cached, consuming significant disk space.

Fix: Manually prune unused images.

sudo docker image prune -a

(For containerd, use sudo ctr images prune -a)

This command removes all dangling images (those not tagged and not referenced by any container) and all images without at least one container associated with them.

2. Large Log Files

Diagnosis: Check log file sizes:

sudo du -sh /var/log/*

Specifically look for large kubelet.log, docker.log, or application-specific logs.

Cause: Pods or the kubelet itself might be generating excessive logs that aren’t being rotated or collected properly, filling up /var/log.

Fix: Configure log rotation for system logs and ensure container log management is in place. For system logs, edit /etc/logrotate.conf and /etc/logrotate.d/* to set appropriate rotation policies (e.g., daily rotation, keep 7 days of logs). For container logs, you can set limits on kubelet’s --container-log-max-files and --container-log-max-size flags. A common setting is:

--container-log-max-files=5 --container-log-max-size=100m

This limits each container’s log files to 5 files, each up to 100MB.

3. Persistent Volume Claims (PVCs) Exceeding Node Capacity

Diagnosis: Identify which PVCs are mounted on the affected node and their current usage.

kubectl get pods --all-namespaces -o wide | grep <node-name>
kubectl get pvc --all-namespaces -o wide | grep <pod-name-on-node>
# Then, on the node:
sudo du -sh /var/lib/kubelet/pods/<pod-uid>/volumes/kubernetes.io~csi/<volume-name>/mount

(The exact path to the mount point can vary based on CSI driver and Kubernetes version)

Cause: Applications using Persistent Volumes (PVs) stored directly on the node’s local disk (e.g., hostPath or local StorageClass) are consuming more space than allocated or available on that node.

Fix:

Increase Node Disk Size: If the node’s disk is genuinely undersized for its workload, you’ll need to provision a larger disk for the node.
Resize PVC: If the PVC itself can be resized (depending on the StorageClass and underlying storage), you can increase its capacity.
```
kubectl patch pvc <pvc-name> -n <namespace> -p '{"spec": {"resources": {"requests": {"storage": "50Gi"}}}}'
```
(Replace 50Gi with the desired new size).
Migrate Data: Move data from the local PV to a more scalable storage solution (e.g., network-attached storage, cloud provider block storage).

This works because the eviction is triggered by the kubelet monitoring the disk usage of partitions it cares about. By managing PVs, you ensure that applications don’t over-consume the node’s local storage.

4. Kubelet’s Disk Pressure Thresholds Too Low

Diagnosis: Check kubelet configuration for imageGCPolicy and evictionHard settings.

# On the node, find kubelet configuration:
# Common locations: /var/lib/kubelet/config.yaml or /etc/kubernetes/kubelet.conf
# Or check the kubelet systemd service:
sudo systemctl status kubelet
# Look for --config flag or command-line arguments

Cause: The evictionHard thresholds configured for the kubelet are too aggressive. For example, if nodefs.available is set to 10%, it means eviction starts when only 10% of the root filesystem is free, which can be reached quickly.

Fix: Adjust evictionHard thresholds to be more lenient. Edit the kubelet configuration file (e.g., /var/lib/kubelet/config.yaml) and modify the evictionHard section. Increase the available disk space percentage. For example, change from nodefs.available: "10%" to nodefs.available: "20%".

apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
# ... other settings
evictionHard:
  memory.available: "200Mi"
  nodefs.available: "20%" # Increased from potentially 10%
  imagefs.available: "20%" # Increased from potentially 10%
# ... other settings

Remember to restart the kubelet service after making changes: sudo systemctl restart kubelet. This gives pods more buffer space before eviction is triggered.

5. Large `kubelet` State Directory

Diagnosis: Check the size of the kubelet’s state directory.

sudo du -sh /var/lib/kubelet/

Cause: The kubelet stores various pieces of state, including pod volumes, mounted volumes, and plugin data. If many pods have been scheduled and removed, or if volumes are not being properly unmounted and cleaned up, this directory can grow very large.

Fix: While direct cleanup is risky and should be done with extreme caution, ensure that storage drivers (like CSI drivers) are correctly cleaning up their resources. If a particular volume type is suspect, investigate its specific cleanup mechanisms. In some cases, a kubelet restart might help clear temporary state, but this is not a guaranteed fix for persistent growth. The underlying issue is usually related to volume lifecycle management by the storage provisioner.

6. Journald accumulating logs

Diagnosis: Check journald disk usage:

sudo journalctl --disk-usage

Cause: If systemd-journald is configured to store logs without size limits, it can consume considerable disk space, especially on busy nodes.

Fix: Configure systemd-journald to limit its disk usage. Edit /etc/systemd/journald.conf and set SystemMaxUse to a reasonable value (e.g., 1G or 2G).

[Journal]
#... other settings
SystemMaxUse=2G

Then restart the journald service: sudo systemctl restart systemd-journald. This prevents journald from growing indefinitely and consuming all available disk space.

Once these are addressed, you might encounter ImagePullBackOff if the node’s disk was so full that image pulling failed previously.