K3s nodes are just machines waiting for work, and "labels" and "taints" are how you tell K3s which machines are right for which jobs.
Let’s see it in action. Imagine you have a K3s cluster with a few nodes. You want to run a specific application, let’s call it my-gpu-app, only on nodes that have a GPU. You also have some general-purpose nodes.
First, you’d label your GPU nodes:
kubectl label node k3s-gpu-node-1 gpu.nvidia.com/present=true
kubectl label node k3s-gpu-node-2 gpu.nvidia.com/present=true
Now, kubectl get nodes --show-labels on k3s-gpu-node-1 would show gpu.nvidia.com/present=true.
Then, you’d define your my-gpu-app pod to require this label:
apiVersion: v1
kind: Pod
metadata:
name: my-gpu-app
spec:
containers:
- name: gpu-container
image: nvidia/cuda:11.0-base-ubuntu20.04
resources:
limits:
nvidia.com/gpu: 1
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: gpu.nvidia.com/present
operator: In
values:
- "true"
When you kubectl apply -f my-gpu-app.yaml, K3s’s scheduler looks at your pods and nodes. It sees my-gpu-app needs a node with gpu.nvidia.com/present=true. It scans its available nodes. k3s-gpu-node-1 and k3s-gpu-node-2 have this label, so they’re candidates. Other nodes without this label are ignored for this pod.
Now, what if you want to prevent certain pods from running on a node? That’s where taints come in. Let’s say you have a node specifically for system-level agents, and you don’t want regular application pods landing there. You’d taint that node:
kubectl taint node k3s-system-node dedicated=system:NoSchedule
This taint means: "This node is dedicated to system tasks. Don’t schedule any pods here unless they have a toleration for dedicated=system."
If you tried to schedule a regular pod (like a simple Nginx pod) on k3s-system-node without a toleration, it wouldn’t land there. The scheduler sees the taint and rejects the pod.
Here’s the full mental model:
- Labels: Key-value pairs attached to nodes (and pods). They are descriptive. You use them to select nodes for specific pods. Think of them as "this node has these characteristics."
- Taints: Applied to nodes. They repel pods. A node with a taint will not accept pods unless those pods "tolerate" the taint. Think of them as "this node is off-limits for certain pods."
- Tolerations: Applied to pods. They allow pods to "ignore" certain taints on nodes. If a pod has a toleration for
key=value:Effect, it can be scheduled on a node that has the corresponding taint. - Node Affinity (and Anti-Affinity): This is how pods express their preferences or requirements for nodes.
nodeSelector: A simple way to match labels. If a pod has anodeSelectorfordisktype=ssd, it will only be scheduled on nodes with thedisktype=ssdlabel.nodeAffinity: More powerful.requiredDuringSchedulingIgnoredDuringExecution: LikenodeSelectorbut more flexible. The pod must be scheduled on a node matching the rules, but if the node’s labels change after the pod is running, the pod isn’t evicted. This is what we used for the GPU app.preferredDuringSchedulingIgnoredDuringExecution: The scheduler will try to find nodes matching these rules, but it’s not a hard requirement. It’s a "nice to have."
podAffinityandpodAntiAffinity: Similar concepts but based on labels of other pods already running on nodes, not node labels themselves.
The key interaction is between taints on nodes and tolerations on pods. A node might have a taint like gpu=true:NoSchedule. Without a toleration, no pod can land there. If you add a toleration to a pod like:
tolerations:
- key: "gpu"
operator: "Equal"
value: "true"
effect: "NoSchedule"
Then that specific pod can be scheduled on the tainted node.
The effect of a taint is important:
NoSchedule: Prevents new pods from being scheduled onto the tainted node. Existing pods are unaffected.PreferNoSchedule: The scheduler will try to avoid scheduling pods onto the tainted node, but it’s not guaranteed.NoExecute: Prevents new pods from being scheduled and evicts any existing pods that do not tolerate the taint. This is the most aggressive.
One common pitfall is mixing up labels and taints. Labels are for finding nodes (selection), while taints are for repelling pods (exclusion). You use nodeAffinity (or nodeSelector) in your pod spec to require a label, and you use tolerations in your pod spec to allow a pod onto a tainted node.
If you’ve set up node labels and affinity rules correctly, the next thing you’ll likely encounter is managing pod resources, which involves requests and limits for CPU, memory, and other hardware like GPUs.