You can upgrade your Kubernetes cluster without downtime by performing a rolling upgrade of your control plane and worker nodes.
Here’s how to do it:
First, let’s set up a simple cluster to demonstrate. We’ll use kind for this example, but the principles apply to any Kubernetes environment.
cat <<EOF | kind create cluster --name upgrade-demo --config=-
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
- role: worker
- role: worker
EOF
This creates a two-worker node cluster running the latest stable Kubernetes version kind defaults to. Now, imagine we want to upgrade to a newer version, say v1.28.0.
The core idea is to upgrade one component at a time, ensuring that the cluster remains functional throughout the process. We’ll start with the control plane.
Upgrading the Control Plane
In a managed Kubernetes service (like GKE, EKS, AKS), this is often handled for you, but understanding the process is key. For self-managed clusters, you’ll typically upgrade the kubeadm configuration and then run kubeadm upgrade apply.
If you were using kubeadm directly, you’d first upgrade the kubeadm binary on your control plane node(s):
# On your control plane node(s)
sudo apt-mark unhold kubeadm && \
sudo apt-get update && sudo apt-get install -y kubeadm=1.28.0-00 && \
sudo apt-mark hold kubeadm
Then, you’d initiate the upgrade:
sudo kubeadm upgrade apply v1.28.0
This command intelligently upgrades the static pods that make up your control plane (API server, controller-manager, scheduler, etcd if co-located) to the new version. It checks for compatibility and ensures your etcd cluster is healthy.
Why this works: kubeadm is designed to manage the lifecycle of a Kubernetes cluster. When you run upgrade apply, it fetches the new static pod manifests, applies them, and then waits for the new control plane components to become healthy before proceeding. This ensures that your cluster’s brain is upgraded in a controlled manner.
Upgrading Worker Nodes
Once the control plane is stable on the new version, you can start upgrading your worker nodes. This involves upgrading the kubelet and kube-proxy components on each node. Again, kubeadm helps here.
You’ll first upgrade the kubeadm binary on your worker nodes:
# On your worker node(s)
sudo apt-mark unhold kubeadm && \
sudo apt-get update && sudo apt-get install -y kubeadm=1.28.0-00 && \
sudo apt-mark hold kubeadm
Then, you’ll upgrade the kubelet and kube-proxy on each worker node:
# On your worker node(s)
sudo apt-mark unhold kubelet kube-proxy && \
sudo apt-get update && sudo apt-get install -y kubelet=1.28.0-00 kube-proxy=1.28.0-00 && \
sudo apt-mark hold kubelet kube-proxy
Crucially, after upgrading the binaries, you need to restart the kubelet service to pick up the new configuration and version:
sudo systemctl daemon-reload
sudo systemctl restart kubelet
Why this works: The kubelet is the primary agent on each worker node. By upgrading its binary and restarting the service, you ensure it communicates with the upgraded control plane using the new API version and features. kube-proxy handles network rules, and upgrading it ensures it’s compatible with the new Kubernetes version’s networking requirements.
Rolling Upgrade Strategy
The key to avoiding downtime is to perform this upgrade one node at a time.
-
Drain the node: Before upgrading a worker node, you must drain it. This cordons the node (prevents new pods from being scheduled) and evicts existing pods gracefully.
kubectl drain <node-name> --ignore-daemonsets --delete-local-data -
Upgrade the node: Perform the
kubeadmupgrade steps andkubelet/kube-proxybinary updates as described above. Restartkubelet. -
Uncordon the node: Once the node is back up and running with the new version, you can uncordon it.
kubectl uncordon <node-name>
Repeat this process for each worker node.
Why this works: By draining and then uncordoning nodes one by one, you ensure that your applications always have available capacity on other healthy nodes. Kubernetes’ built-in scheduler and controllers handle the rescheduling of pods from drained nodes to active ones, maintaining service availability.
Considerations for etcd
If your etcd is not managed by kubeadm (e.g., you’re running a highly available etcd cluster separately), you’ll need to follow etcd’s specific upgrade procedures. Typically, this also involves a rolling upgrade of etcd members, ensuring quorum is maintained at all times. Each etcd member is upgraded individually, and the cluster waits for the member to rejoin successfully before proceeding to the next.
The Next Hurdle
After successfully upgrading your entire cluster, you might encounter issues with older Custom Resource Definitions (CRDs) or admission webhooks that are not yet compatible with the new API versions used by the upgraded control plane components.