The Istio control plane’s certificate authority (CA) is issuing certificates that aren’t yet valid for some workloads, causing mTLS connections to fail.
This usually happens because the CA’s clock is out of sync with the workload pods, or because the Istio CA itself is misconfigured.
Cause 1: Clock Skew on the Istio CA Pod(s)
The most frequent culprit is that the system clock on the pod running the Istio CA (typically istiod) has drifted significantly from Coordinated Universal Time (UTC). Certificate validity periods are strictly enforced, and if the CA’s clock is ahead, it will issue certificates with a "not yet valid" start date.
-
Diagnosis: Exec into the
istiodpod and check its current time.kubectl exec -it -n istio-system <istiod-pod-name> -- dateCompare this output to a reliable UTC time source (e.g.,
date -uon your local machine or an online NTP check). A difference of more than a few seconds is problematic. -
Fix: Synchronize the clock on the
istiodpod. This is best achieved by ensuring your Kubernetes nodes have NTP configured correctly. If nodes are synced, the pod’s time should generally follow. As a temporary workaround or if node syncing is difficult, you can restart theistiodpod, which often forces it to resynchronize its time from the node.kubectl delete pod -n istio-system <istiod-pod-name>This forces Kubernetes to reschedule the
istioddeployment, and a new pod will start with a potentially more accurate time. -
Why it works: Correctly synchronized clocks ensure that the "Not Before" timestamp in the issued certificate aligns with the actual current time when the client attempts to validate it.
Cause 2: Incorrect validityDuration in Istio Operator Configuration
The Istio operator configuration allows you to specify the validity duration of certificates issued by the CA. If this value is set too high, it can lead to issues, though more commonly, it’s a symptom of other problems. However, if it’s set to something extremely large, or if there’s a misunderstanding of its interaction with clock skew, it can manifest as this error.
-
Diagnosis: Examine your Istio operator configuration. This is often a
IstioOperatorcustom resource in theistio-systemnamespace.kubectl get iop -n istio-system -o yamlLook for a section like
spec.meshConfig.caCertificatesor similar, and specifically thevalidityDurationfield. -
Fix: Reset
validityDurationto a reasonable default or a smaller value if you’re experiencing issues. A common default is24h. If you suspect this is the cause, try removing it or setting it to24h.apiVersion: install.istio.io/v1alpha1 kind: IstioOperator metadata: name: istio namespace: istio-system spec: meshConfig: caCertificates: - secretName: cacerts # validityDuration: 24h # Uncomment or set to a reasonable valueApply the change and then restart
istiod.kubectl apply -f your-istio-operator.yaml kubectl delete pod -n istio-system <istiod-pod-name> -
Why it works: A shorter, well-defined validity duration reduces the window where clock skew between the CA and clients can cause validation failures.
Cause 3: Clock Skew on the Client Workload Pods
It’s not just the CA that needs accurate time; the client pods establishing the mTLS connection also need their clocks synchronized. If a client pod’s clock is too far ahead of UTC, it will consider a certificate valid now even if the CA issued it with a future start date.
-
Diagnosis: Similar to Cause 1, exec into a client pod that’s experiencing the error and check its time.
kubectl exec -it -n <your-namespace> <client-pod-name> -- dateCompare this to UTC.
-
Fix: Ensure that the Kubernetes nodes hosting these client pods have NTP configured correctly. If the nodes are synced, the pods will generally inherit accurate time. Restarting the client pods can also help them resynchronize.
kubectl delete pod -n <your-namespace> <client-pod-name> -
Why it works: All participants in the mTLS handshake must agree on the current time for certificate validation to succeed.
Cause 4: Istio CA Certificate Rotation Issues
Istio automatically rotates its root and intermediate CA certificates. If this rotation process encounters errors or if there’s a delay in new certificates being distributed and picked up by istiod and the sidecars, it can lead to a state where clients are trying to validate against an old CA or a CA whose new certificate is not yet valid.
-
Diagnosis: Check the creation and validity dates of the CA secrets in the
istio-systemnamespace.kubectl get secret cacerts -n istio-system -o yaml kubectl get secret istio-ca-certificates -n istio-system -o yaml # For newer Istio versionsLook for the
data.ca-cert.pemanddata.root-cert.pemfields and decode them to check their validity periods.kubectl get secret cacerts -n istio-system -o jsonpath='{.data.ca-cert\.pem}' | base64 --decode | openssl x509 -noout -dates kubectl get secret cacerts -n istio-system -o jsonpath='{.data.root-cert\.pem}' | base64 --decode | openssl x509 -noout -dates -
Fix: If the rotation seems stuck or has failed, you might need to manually trigger a rotation or restart
istiodto pick up any pending changes. Often, a simpleistiodrestart is enough. If secrets are genuinely corrupt or missing, you may need to reapply Istio configurations or, in extreme cases, reinstall.kubectl delete pod -n istio-system <istiod-pod-name> -
Why it works: Ensures that the CA certificates used by
istiodand distributed to sidecars are current and correctly signed, with valid start dates.
Cause 5: Incorrect ca.crt or root-cert.pem in Secrets
Manual modifications to the cacerts or related secrets in istio-system can corrupt the CA chain. If the ca.crt or root-cert.pem within these secrets are incorrect, malformed, or have invalid dates, it will break mTLS.
-
Diagnosis: Decode the certificates from the relevant secret and inspect them using
openssl.kubectl get secret cacerts -n istio-system -o jsonpath='{.data.ca-cert\.pem}' | base64 --decode > ca.crt kubectl get secret cacerts -n istio-system -o jsonpath='{.data.root-cert\.pem}' | base64 --decode > root.crt openssl x509 -in ca.crt -text -noout openssl x509 -in root.crt -text -nooutPay close attention to the "Not Before" and "Not After" dates.
-
Fix: Restore the correct certificates. If you have a backup or can regenerate them via Istio installation/upgrade, do so. The safest approach is often to let Istio manage these secrets. If you suspect manual edits, delete the secret and let
istiodrecreate it.kubectl delete secret cacerts -n istio-system kubectl delete pod -n istio-system <istiod-pod-name> -
Why it works: Replaces corrupted or invalid CA certificates with a known good chain, allowing
istiodto issue and clients to validate certificates correctly.
After resolving these, you’ll likely encounter UPSTREAM_DISCONNECTED errors if your service mesh is not properly configured for health checks or if network policies are blocking traffic.