The Istio ingress gateway is refusing requests because it can’t find any healthy upstream pods for the target service.
This is happening because the Istio control plane (istiod) is not detecting any healthy pods for your application service, or the gateway itself is misconfigured to look for the wrong service.
Cause 1: Pods are actually unhealthy
Diagnosis: Check the status of your application pods.
kubectl get pods -n <your-namespace> -l app=<your-app-label> -o wide
Look for pods that are CrashLoopBackOff, Error, or in Pending with no clear reason.
Fix: Address the underlying issue causing your pods to be unhealthy. This could be:
- Application errors: Check your application logs:
kubectl logs <pod-name> -n <your-namespace> - Resource limits: If pods are OOMKilled, increase memory/CPU limits in your deployment spec.
- Readiness probes failing: Ensure your readiness probe path is correct and returning a 2xx status code. Example:
initialDelaySeconds: 5,periodSeconds: 10,httpGet: { path: /healthz, port: 8080 } - Node issues: If pods are
Pendingdue to insufficient resources or node taints, resolve those.
Why it works: Istio’s EDS (Endpoint Discovery Service) will only add endpoints to a service cluster if the corresponding pods are marked as Running and passing their readiness probes. If no pods meet these criteria, Istio has no endpoints to send traffic to.
Cause 2: Service definition is incorrect or missing
Diagnosis: Verify your Kubernetes Service definition.
kubectl get service <your-service-name> -n <your-namespace> -o yaml
Ensure the selector in the service definition correctly matches the labels on your application pods.
Fix: Correct the selector in your Service YAML to match your pod labels.
apiVersion: v1
kind: Service
metadata:
name: my-app-service
namespace: my-namespace
spec:
selector:
app: my-app # This MUST match the labels on your pods
ports:
- protocol: TCP
port: 80
targetPort: 8080
Why it works: The Kubernetes Service acts as an internal load balancer and provides the IP address for Istio’s internal service discovery. If the selector is wrong, the Service won’t discover any pods, and subsequently, Istio won’t either.
Cause 3: Istio VirtualService targets the wrong service or port
Diagnosis: Inspect your VirtualService configuration.
kubectl get virtualservice <your-virtual-service-name> -n <your-namespace> -o yaml
Check the hosts and route.destination.host fields.
Fix: Ensure the hosts in your VirtualService match the host you are sending to the gateway (e.g., my-app.example.com), and that route.destination.host correctly points to your Kubernetes Service name (e.g., my-app-service.my-namespace.svc.cluster.local or just my-app-service if in the same namespace). Also, verify route.destination.port.number matches the port exposed by your Kubernetes Service.
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: my-app-vs
namespace: istio-system # Or the namespace where your VS resides
spec:
hosts:
- my-app.example.com
gateways:
- my-gateway # Your Istio gateway resource
http:
- route:
- destination:
host: my-app-service.my-namespace.svc.cluster.local # Correct service name
port:
number: 80 # Correct service port
Why it works: The VirtualService tells Istio how to route incoming gateway traffic to your internal services. If it’s misdirected, Istio will try to find endpoints for a service that doesn’t exist or isn’t configured correctly.
Cause 4: Istio Gateway resource is misconfigured
Diagnosis: Examine your Gateway resource.
kubectl get gateway <your-gateway-name> -n <your-namespace> -o yaml
Ensure the selector for the gateway points to your Istio ingress gateway pods.
Fix: Correct the selector in your Gateway YAML to match the labels of your Istio ingress gateway pods. Typically, this is app: istio-ingressgateway and it’s deployed in the istio-system namespace.
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
name: my-gateway
namespace: istio-system # Or the namespace where your Gateway resource resides
spec:
selector:
istio: ingressgateway # This label must match your ingress gateway deployment
servers:
- port:
number: 80
name: http
protocol: HTTP
hosts:
- my-app.example.com
Why it works: The Gateway resource configures the Istio ingress gateway itself. If the gateway’s selector is incorrect, it might not be running or properly configured to serve traffic, leading to it being unable to route requests even if the VirtualService is correct.
Cause 5: Network policy blocking Istio sidecar or ingress gateway
Diagnosis: Check Kubernetes NetworkPolicy resources.
kubectl get networkpolicy -n <your-namespace>
Look for policies that might restrict ingress to your application pods or egress from the Istio ingress gateway.
Fix: Add or modify NetworkPolicy rules to allow traffic from the Istio ingress gateway (or the istio-ingressgateway service account) to your application pods on the correct port.
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-istio-ingress
namespace: <your-namespace>
spec:
podSelector:
matchLabels:
app: <your-app-label>
policyTypes:
- Ingress
ingress:
- from:
- namespaceSelector:
matchLabels:
istio: ingressgateway # Or a label that identifies your ingress gateway namespace
podSelector:
matchLabels:
app: istio-ingressgateway # Label of the ingress gateway pods
ports:
- protocol: TCP
port: 8080 # The targetPort of your application service
Why it works: Network policies enforce network segmentation. If a policy is too restrictive, it can prevent the ingress gateway (or even the sidecar proxies) from reaching your application pods, even if Istio’s configuration is otherwise correct.
Cause 6: Istiod health and connectivity issues
Diagnosis: Check the health of your istiod pods.
kubectl get pods -n istio-system -l app=istiod
Ensure istiod pods are Running and healthy. Also, check logs for errors.
Fix: If istiod is unhealthy, investigate the reasons. This might involve restarting the pods, checking resource allocation for istiod, or ensuring the Kubernetes API server is accessible from istiod.
Why it works: istiod is the central control plane component responsible for pushing configuration (like VirtualServices and Gateways) and service/endpoint information to the Envoy proxies in the data plane (sidecars and ingress gateway). If istiod is down or unhealthy, it cannot update the Envoy proxies, which will eventually lose their endpoint information or fail to receive new configurations.
The next error you’ll likely see is related to the VirtualService not being found if you fixed the service configuration but not the VS.