Istio’s control plane upgrades can be performed with a canary strategy, allowing you to roll out new versions to a subset of your traffic before a full rollout.
Let’s see how this looks in practice. Imagine we have Istio version 1.18.2 running, and we want to upgrade to 1.20.0.
First, we’ll install the new control plane components alongside the old ones. This involves applying the new Istio operator configuration, but crucially, we’ll specify a revision for the new control plane. This revision acts as a unique identifier for this specific control plane version.
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
revision: 1-20-0 # This is the key to canarying
meshConfig:
accessLogFile: /dev/stdout
defaultConfig:
proxyMetadata:
ISTIO_META_DNS_CAPTURE: "true"
ISTIO_META_DNS_INJECT: "true"
components:
pilot:
k8s:
env:
- name: PILOT_TRACE_SAMPLING
value: "100"
When you apply this with istioctl install -f new-istio-operator.yaml, Istio will deploy a new set of control plane pods (istiod) tagged with the 1-20-0 revision. Your existing services will still be pointed to the old control plane revision (e.g., istio-system namespace with no explicit revision label).
The magic happens when you want to start directing workloads to the new control plane. You do this by annotating your istiod service. Initially, only the old revision is active.
kubectl label namespace istio-system istio-injection- --overwrite
To enable the new revision, you’ll annotate the istiod service in the istio-system namespace.
kubectl annotate service istiod -n istio-system \
"alpha.istio.io/kubernetesService"='{"revision":"1-20-0"}' \
--overwrite
This annotation tells Istio’s sidecar injector to start injecting sidecars that are configured to communicate with the istiod instance associated with the 1-20-0 revision. However, this annotation alone doesn’t immediately switch all existing pods. New pods created or pods that are redeployed will get the new sidecar.
To gradually shift traffic, you’ll incrementally enable the istio-injection=enabled label on namespaces. You can start with a single namespace:
kubectl label namespace my-app-staging istio-injection=enabled --overwrite
Pods in my-app-staging will now have sidecars that talk to the 1-20-0 revision of Istiod. You can monitor your application’s behavior in this staging namespace. If everything looks good, you can expand the rollout to more namespaces.
To fully roll out, you would eventually label all relevant namespaces with istio-injection=enabled. Once all namespaces are using the new revision, you can remove the old istiod deployment and the annotation from the istiod service.
kubectl delete deployment istiod-1-18-2 -n istio-system
kubectl annotate service istiod -n istio-system "alpha.istio.io/kubernetesService" --unset
The underlying mechanism relies on the istio-sidecar-injector webhook. When a pod is created or updated in a namespace with istio-injection=enabled, the injector consults the istiod service’s alpha.istio.io/kubernetesService annotation. If the annotation points to a specific revision, the injector configures the sidecar to communicate with that revision’s istiod. For namespaces without the annotation, or where the annotation is absent, it defaults to the istio-system namespace’s istiod service.
A common pitfall is forgetting to update the istio-injection label on namespaces. If you only update the istiod service annotation and don’t relabel namespaces or restart pods, existing sidecars won’t automatically switch to the new control plane. The injector’s decision is based on the namespace’s labels at pod creation time.
The next challenge you’ll likely face is understanding how to manage multiple Istio control plane revisions simultaneously for different sets of workloads, enabling more granular control over your service mesh’s evolution.