K3s Upgrade: Automate Node Upgrades Safely (2026)

K3s upgrades, especially on production clusters, can feel like threading a needle in a hurricane. You’re not just updating software; you’re orchestrating a delicate dance where nodes must be drained, updated, and rejoined without impacting running workloads. The goal is to automate this process, making it repeatable and less prone to human error, while ensuring zero downtime for your applications.

Let’s see this in action. Imagine a simple Nginx deployment running across your K3s cluster. We want to upgrade K3s from v1.28.5 to v1.29.0 without users noticing.

Here’s a baseline deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
spec:
  replicas: 3
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:latest
        ports:
        - containerPort: 80

And a Service to expose it:

apiVersion: v1
kind: Service
metadata:
  name: nginx-service
spec:
  selector:
    app: nginx
  ports:
    - protocol: TCP
      port: 80
      targetPort: 80
  type: LoadBalancer

When we initiate a K3s upgrade, the key is to isolate nodes one by one. We’ll mark a node as unschedulable, then gracefully evict all pods running on it, update the K3s binary on that node, and finally mark it as schedulable again. This ensures that at any given moment, at least one node remains healthy and serving traffic.

The most effective way to automate this is using a combination of kubectl and shell scripting. You’ll need to iterate through your nodes, drain them, perform the upgrade, and uncordon them.

Here’s a conceptual script outline:

#!/bin/bash

# Get list of all worker nodes (excluding control-plane if applicable)
NODES=$(kubectl get nodes -l "node-role.kubernetes.io/control-plane!=" -o jsonpath='{.items[*].metadata.name}')

K3S_VERSION="v1.29.0" # Target K3s version

for NODE in $NODES; do
  echo "--- Upgrading node: $NODE ---"

  # 1. Cordon the node to prevent new pods from being scheduled
  echo "Cordoning node $NODE..."
  kubectl cordon $NODE
  if [ $? -ne 0 ]; then
    echo "Error: Failed to cordon node $NODE. Skipping."
    continue
  fi

  # 2. Drain the node, evicting all pods gracefully
  echo "Draining node $NODE..."
  kubectl drain $NODE --ignore-daemonsets --delete-local-data
  if [ $? -ne 0 ]; then
    echo "Error: Failed to drain node $NODE. Skipping upgrade for this node."
    # Optionally, uncordon here if drain fails to allow rescheduling elsewhere
    kubectl uncordon $NODE
    continue
  fi

  # 3. Perform the K3s upgrade on the node (this part is manual or uses a separate agent)
  echo "Performing K3s upgrade on $NODE to $K3S_VERSION..."
  # Example: SSH into the node and run the K3s install script with the desired version
  # ssh user@$NODE "curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION=$K3S_VERSION sh -"
  echo "Manually upgrade K3s on $NODE to $K3S_VERSION. Press Enter when done."
  read -r

  # 4. Uncordon the node to allow new pods to be scheduled
  echo "Uncordoning node $NODE..."
  kubectl uncordon $NODE
  if [ $? -ne 0 ]; then
    echo "Error: Failed to uncordon node $NODE. Manual intervention required."
    continue
  fi

  echo "--- Node $NODE upgrade complete ---"
  sleep 30 # Give the cluster a moment to stabilize
done

echo "--- All nodes have been upgraded ---"

The kubectl drain $NODE --ignore-daemonsets --delete-local-data command is crucial. --ignore-daemonsets prevents the drain from failing due to DaemonSet pods (like kube-proxy or node-exporter) which are expected to run on every node. --delete-local-data ensures that pods using emptyDir volumes, which would lose their data upon eviction, are also considered for eviction, preventing the drain from hanging on them.

The actual upgrade command (curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION=$K3S_VERSION sh -) is the core of the K3s update. By setting INSTALL_K3S_VERSION, you tell the K3s installer script to download and install that specific version. If you’re using a different installation method (e.g., systemd service files, Ansible), you’d adapt this step accordingly.

After the upgrade, K3s automatically handles the rejoining of the node to the cluster. The control plane recognizes the updated agent and continues to schedule workloads. The Service, in our Nginx example, will seamlessly route traffic to the remaining healthy pods and then to pods on the newly upgraded node once it’s ready.

Crucially, the kubectl cordon $NODE command marks the node as unschedulable, preventing any new pods from landing on it while it’s being drained and upgraded. Conversely, kubectl uncordon $NODE reverses this, allowing the scheduler to place new pods on the now-healthy, upgraded node.

One aspect that often trips people up is the local-data part of the drain. If you have pods that must retain their emptyDir data across node upgrades (which is rare for emptyDir but possible), you’d need a more sophisticated strategy, perhaps involving persistent volumes or application-level state replication. For most stateless or stateful applications using persistent storage, --delete-local-data is safe and necessary for a smooth drain.

The next hurdle you’ll likely encounter after a successful K3s upgrade is ensuring your application images are also updated to compatible versions, or that your CI/CD pipeline is ready to deploy them against the new K3s version.