The Istio control plane’s certificate authority (CA) is issuing certificates that aren’t yet valid for some workloads, causing mTLS connections to fail.

This usually happens because the CA’s clock is out of sync with the workload pods, or because the Istio CA itself is misconfigured.

Cause 1: Clock Skew on the Istio CA Pod(s)

The most frequent culprit is that the system clock on the pod running the Istio CA (typically istiod) has drifted significantly from Coordinated Universal Time (UTC). Certificate validity periods are strictly enforced, and if the CA’s clock is ahead, it will issue certificates with a "not yet valid" start date.

  • Diagnosis: Exec into the istiod pod and check its current time.

    kubectl exec -it -n istio-system <istiod-pod-name> -- date
    

    Compare this output to a reliable UTC time source (e.g., date -u on your local machine or an online NTP check). A difference of more than a few seconds is problematic.

  • Fix: Synchronize the clock on the istiod pod. This is best achieved by ensuring your Kubernetes nodes have NTP configured correctly. If nodes are synced, the pod’s time should generally follow. As a temporary workaround or if node syncing is difficult, you can restart the istiod pod, which often forces it to resynchronize its time from the node.

    kubectl delete pod -n istio-system <istiod-pod-name>
    

    This forces Kubernetes to reschedule the istiod deployment, and a new pod will start with a potentially more accurate time.

  • Why it works: Correctly synchronized clocks ensure that the "Not Before" timestamp in the issued certificate aligns with the actual current time when the client attempts to validate it.

Cause 2: Incorrect validityDuration in Istio Operator Configuration

The Istio operator configuration allows you to specify the validity duration of certificates issued by the CA. If this value is set too high, it can lead to issues, though more commonly, it’s a symptom of other problems. However, if it’s set to something extremely large, or if there’s a misunderstanding of its interaction with clock skew, it can manifest as this error.

  • Diagnosis: Examine your Istio operator configuration. This is often a IstioOperator custom resource in the istio-system namespace.

    kubectl get iop -n istio-system -o yaml
    

    Look for a section like spec.meshConfig.caCertificates or similar, and specifically the validityDuration field.

  • Fix: Reset validityDuration to a reasonable default or a smaller value if you’re experiencing issues. A common default is 24h. If you suspect this is the cause, try removing it or setting it to 24h.

    apiVersion: install.istio.io/v1alpha1
    kind: IstioOperator
    metadata:
      name: istio
      namespace: istio-system
    spec:
      meshConfig:
        caCertificates:
          - secretName: cacerts
            # validityDuration: 24h # Uncomment or set to a reasonable value
    

    Apply the change and then restart istiod.

    kubectl apply -f your-istio-operator.yaml
    kubectl delete pod -n istio-system <istiod-pod-name>
    
  • Why it works: A shorter, well-defined validity duration reduces the window where clock skew between the CA and clients can cause validation failures.

Cause 3: Clock Skew on the Client Workload Pods

It’s not just the CA that needs accurate time; the client pods establishing the mTLS connection also need their clocks synchronized. If a client pod’s clock is too far ahead of UTC, it will consider a certificate valid now even if the CA issued it with a future start date.

  • Diagnosis: Similar to Cause 1, exec into a client pod that’s experiencing the error and check its time.

    kubectl exec -it -n <your-namespace> <client-pod-name> -- date
    

    Compare this to UTC.

  • Fix: Ensure that the Kubernetes nodes hosting these client pods have NTP configured correctly. If the nodes are synced, the pods will generally inherit accurate time. Restarting the client pods can also help them resynchronize.

    kubectl delete pod -n <your-namespace> <client-pod-name>
    
  • Why it works: All participants in the mTLS handshake must agree on the current time for certificate validation to succeed.

Cause 4: Istio CA Certificate Rotation Issues

Istio automatically rotates its root and intermediate CA certificates. If this rotation process encounters errors or if there’s a delay in new certificates being distributed and picked up by istiod and the sidecars, it can lead to a state where clients are trying to validate against an old CA or a CA whose new certificate is not yet valid.

  • Diagnosis: Check the creation and validity dates of the CA secrets in the istio-system namespace.

    kubectl get secret cacerts -n istio-system -o yaml
    kubectl get secret istio-ca-certificates -n istio-system -o yaml # For newer Istio versions
    

    Look for the data.ca-cert.pem and data.root-cert.pem fields and decode them to check their validity periods.

    kubectl get secret cacerts -n istio-system -o jsonpath='{.data.ca-cert\.pem}' | base64 --decode | openssl x509 -noout -dates
    kubectl get secret cacerts -n istio-system -o jsonpath='{.data.root-cert\.pem}' | base64 --decode | openssl x509 -noout -dates
    
  • Fix: If the rotation seems stuck or has failed, you might need to manually trigger a rotation or restart istiod to pick up any pending changes. Often, a simple istiod restart is enough. If secrets are genuinely corrupt or missing, you may need to reapply Istio configurations or, in extreme cases, reinstall.

    kubectl delete pod -n istio-system <istiod-pod-name>
    
  • Why it works: Ensures that the CA certificates used by istiod and distributed to sidecars are current and correctly signed, with valid start dates.

Cause 5: Incorrect ca.crt or root-cert.pem in Secrets

Manual modifications to the cacerts or related secrets in istio-system can corrupt the CA chain. If the ca.crt or root-cert.pem within these secrets are incorrect, malformed, or have invalid dates, it will break mTLS.

  • Diagnosis: Decode the certificates from the relevant secret and inspect them using openssl.

    kubectl get secret cacerts -n istio-system -o jsonpath='{.data.ca-cert\.pem}' | base64 --decode > ca.crt
    kubectl get secret cacerts -n istio-system -o jsonpath='{.data.root-cert\.pem}' | base64 --decode > root.crt
    openssl x509 -in ca.crt -text -noout
    openssl x509 -in root.crt -text -noout
    

    Pay close attention to the "Not Before" and "Not After" dates.

  • Fix: Restore the correct certificates. If you have a backup or can regenerate them via Istio installation/upgrade, do so. The safest approach is often to let Istio manage these secrets. If you suspect manual edits, delete the secret and let istiod recreate it.

    kubectl delete secret cacerts -n istio-system
    kubectl delete pod -n istio-system <istiod-pod-name>
    
  • Why it works: Replaces corrupted or invalid CA certificates with a known good chain, allowing istiod to issue and clients to validate certificates correctly.

After resolving these, you’ll likely encounter UPSTREAM_DISCONNECTED errors if your service mesh is not properly configured for health checks or if network policies are blocking traffic.

Want structured learning?

Take the full Istio course →