The error: failed to connect: upstream error in Istio means the Envoy proxy acting as a client couldn’t find any healthy instances of the service it was trying to reach. This usually points to a problem with Istio’s control plane configuration or the health of the application pods themselves.

Common Causes and Fixes:

  1. Application Pods are Not Ready or Crashing:

    • Diagnosis: Check the status of your application pods:
      kubectl get pods -n <your-namespace>
      
      Look for pods in CrashLoopBackOff, Error, or Pending states. Then, inspect individual pod logs:
      kubectl logs <pod-name> -n <your-namespace>
      
    • Fix: Address the underlying application issue. This might involve fixing bugs, increasing resource limits (CPU/memory), or resolving dependency problems. Ensure your application starts quickly and passes its readiness and liveness probes.
    • Why it works: If the application pods aren’t running correctly, Envoy has no endpoints to send traffic to, resulting in the "no healthy upstream" error.
  2. Incorrect Kubernetes Service Definition:

    • Diagnosis: Verify that your Kubernetes Service object correctly targets your application pods.
      kubectl get service <service-name> -n <your-namespace> -o yaml
      
      Pay close attention to the selector field and ensure it matches the labels on your application pods. Also, check the ports to confirm the targetPort matches the port your application is listening on.
    • Fix: Correct the selector or targetPort in your Service definition to accurately reflect your pod labels and application listening port. Apply the corrected YAML:
      kubectl apply -f <your-service-yaml> -n <your-namespace>
      
    • Why it works: The Kubernetes Service acts as an internal DNS and load balancer. If its selector is wrong, it won’t discover your pods, and Envoy won’t know where to send traffic.
  3. Istio Sidecar Not Injected or Not Running:

    • Diagnosis: Check if the Istio sidecar proxy (Envoy) is injected into your application pods.
      kubectl get pods -n <your-namespace> -o jsonpath='{.items[*].spec.containers[*].name}'
      
      You should see at least two containers per pod: your application container and istio-proxy. Also, check the status of the istio-proxy container for any errors.
    • Fix: If the sidecar is missing, enable automatic sidecar injection for the namespace:
      kubectl label namespace <your-namespace> istio-injection=enabled
      
      If it’s present but not running, check its logs for errors:
      kubectl logs <pod-name> -c istio-proxy -n <your-namespace>
      
      And restart the pod to ensure it gets a fresh sidecar.
    • Why it works: The Istio sidecar intercepts all inbound and outbound traffic for the pod. If it’s not running, network requests can’t be managed by Istio, leading to connectivity issues.
  4. Istio VirtualService or DestinationRule Misconfiguration:

    • Diagnosis: Examine your Istio VirtualService and DestinationRule configurations for the affected service.
      kubectl get virtualservice <service-name> -n <your-namespace> -o yaml
      kubectl get destinationrule <service-name> -n <your-namespace> -o yaml
      
      Ensure the host in the VirtualService matches the service name you’re trying to reach (e.g., my-service.my-namespace.svc.cluster.local). Check that the DestinationRule correctly defines subsets (if used) and that these subsets point to valid Kubernetes Service endpoints.
    • Fix: Correct any typos, incorrect hostnames, or misconfigured subset definitions in your Istio configuration. Apply the corrected YAML.
    • Why it works: VirtualServices route traffic, and DestinationRules define how to connect to the destination. If these are misconfigured, Istio might be trying to route traffic to an incorrect host or an undefined subset, leading to no healthy upstream.
  5. Network Policies Blocking Traffic:

    • Diagnosis: Check if any Kubernetes NetworkPolicy objects are in place that might be preventing the Istio sidecar (or the application pod) from communicating with the upstream service’s pods or its Kubernetes Service.
      kubectl get networkpolicies -n <your-namespace>
      
      Review policies in both the source and destination namespaces.
    • Fix: Modify or create NetworkPolicy rules to explicitly allow traffic between the necessary pods and services. For example, to allow ingress to your application pods:
      apiVersion: networking.k8s.io/v1
      kind: NetworkPolicy
      metadata:
        name: allow-ingress-from-frontend
        namespace: <your-namespace>
      spec:
        podSelector:
          matchLabels:
            app: my-app
        policyTypes:
        - Ingress
        ingress:
        - from:
          - podSelector:
              matchLabels:
                app: frontend
      
    • Why it works: Network policies are firewalls at the pod level. If they are too restrictive, they can block legitimate traffic between services, even if Istio is configured correctly.
  6. Istio Control Plane Component Issues (Pilot/istiod):

    • Diagnosis: Check the health of Istio’s control plane components, primarily istiod.
      kubectl get pods -n istio-system
      
      Ensure istiod pods are running and healthy. Check their logs for errors:
      kubectl logs <istiod-pod-name> -n istio-system
      
    • Fix: If istiod is unhealthy, investigate why. This could involve resource constraints, configuration errors during installation, or upstream communication problems within the Kubernetes cluster. Restarting the istiod pods or the Istio control plane might resolve transient issues.
    • Why it works: istiod is responsible for pushing configuration (like VirtualServices and DestinationRules) to the Envoy sidecars. If istiod is not functioning, Envoy won’t receive up-to-date endpoint information or routing rules.

After resolving these issues, you might encounter a timeout error if the upstream service is still slow to respond or if load balancing is misconfigured in a DestinationRule subset.

Want structured learning?

Take the full Istio course →