The Kubernetes scheduler is refusing to place pods onto nodes because the nodes have taints that the pods do not tolerate.
Common Causes and Fixes
-
Node taints preventing scheduling:
- Diagnosis:
kubectl describe node <node-name>. Look for theTaintssection. You’ll see entries likenode-role.kubernetes.io/master:NoScheduleor custom taints. - Fix: Add a toleration to your pod’s spec. For example, to tolerate a
node-role.kubernetes.io/master:NoScheduletaint, add:
This tells the scheduler that this pod is allowed to run on nodes with this specific taint.spec: tolerations: - key: "node-role.kubernetes.io/master" operator: "Exists" effect: "NoSchedule" - Why it works: Taints are applied to nodes to repel pods. Tolerations are applied to pods to allow them to be scheduled onto tainted nodes. Without a matching toleration, the pod will not be scheduled.
- Diagnosis:
-
NoExecutetaint effect:- Diagnosis: Same as above, but observe the
effectisNoExecute. This means not only will new pods not be scheduled, but existing pods on the node that don’t tolerate the taint will be evicted. - Fix: Add a toleration with the matching
effect. For example, forkey: "special-gpu:NoExecute", add:
This explicitly allows the pod to run on (or remain on) nodes with thisspec: tolerations: - key: "special-gpu" operator: "Equal" value: "true" effect: "NoExecute"NoExecutetaint. - Why it works: The
NoExecuteeffect is a stronger form of repulsion, actively evicting pods. Tolerating it prevents eviction and allows scheduling.
- Diagnosis: Same as above, but observe the
-
Incorrect taint key or value:
- Diagnosis: Double-check the exact
keyandvalue(ifoperatorisEqual) in the node’s taints against your pod’s tolerations. Typos are common. For instance, a node might havegpu=trueand your toleration isgpu=true. - Fix: Correct the toleration to match the taint. If the node has
gpu=trueand the taint iskey: "gpu", operator: "Equal", value: "true", your toleration should be identical.spec: tolerations: - key: "gpu" operator: "Equal" value: "true" effect: "NoSchedule" # Or whatever effect is on the node - Why it works: Kubernetes performs an exact match for
Equaloperators. Any discrepancy, including case sensitivity, will cause the toleration to fail.
- Diagnosis: Double-check the exact
-
Using
Existsoperator incorrectly:- Diagnosis: If a taint has a
value(e.g.,dedicated=gpu:NoSchedule), usingoperator: "Exists"in your toleration will match any pod with that key, regardless of its value. This might not be what you intended if you need to match a specific value. - Fix: Use
operator: "Equal"if you need to match a specific value associated with the taint key. For a taintdedicated=gpu:NoSchedule, the toleration should be:spec: tolerations: - key: "dedicated" operator: "Equal" value: "gpu" effect: "NoSchedule" - Why it works: The
Existsoperator checks for the presence of the taint key, whileEqualchecks for both the key and its associated value.
- Diagnosis: If a taint has a
-
Taints applied by node labels or controllers:
- Diagnosis: Some Kubernetes components (like
kubeadmfor control-plane nodes, or cloud provider integrations) automatically apply taints. For example,kubeadmaddsnode-role.kubernetes.io/control-plane:NoScheduleto control-plane nodes. - Fix: Understand why the taint is there. If it’s a control-plane node, you likely should tolerate it for specific pods (like monitoring agents). If it’s a custom taint applied by a controller, consult that controller’s documentation.
- Why it works: Taints are not always manually applied. Understanding the source helps determine if you should tolerate it or if the taint itself is misconfigured.
- Diagnosis: Some Kubernetes components (like
-
Node not ready or unhealthy:
- Diagnosis:
kubectl get nodes. If the node status is notReady, it might not be scheduling pods, regardless of taints. Checkkubectl describe node <node-name>forConditionslikeReady: False. - Fix: Troubleshoot the node’s health. This could involve checking kubelet logs (
journalctl -u kubelet), container runtime status, or network connectivity to the control plane. - Why it works: A node must be in a
Readystate for the scheduler to consider it for pod placement. Unhealthy nodes are cordoned by default, effectively preventing new pods.
- Diagnosis:
The next error you’ll likely hit is a CrashLoopBackOff if the pod itself has an issue after finally being scheduled.