The Istio ingress gateway is failing to route traffic to your services because it can’t find a matching VirtualService for the incoming request’s host and path.
This usually means one of two things: either the VirtualService isn’t being applied to the correct namespace, or its configuration doesn’t precisely match the request.
Common Causes and Fixes
1. VirtualService in the Wrong Namespace
-
Diagnosis: Check the namespace of your
VirtualServiceresource. Istio’sGatewayresource is namespace-scoped, and theVirtualServicemust be in the same namespace as theGatewayit’s intended to bind to, or theVirtualServicemust explicitly reference theGateway’s namespace if they are different (which is less common for basic setups).kubectl get virtualservices -ALook for your
VirtualServiceand confirm its namespace. If yourGatewayis inistio-system, yourVirtualServicemust also be inistio-system(or specifygateways: - "istio-system/my-gateway"within theVirtualServiceif it’s in another namespace). -
Fix: If the
VirtualServiceis in the wrong namespace, move it to the namespace where yourGatewayresides.# Example: If VirtualService is in 'default' and Gateway is in 'istio-system' kubectl get virtualservice my-virtual-service -n default -o yaml | \ sed "s/namespace: default/namespace: istio-system/" | \ kubectl apply -f - # Then delete the old one kubectl delete virtualservice my-virtual-service -n defaultThis ensures the
VirtualServiceis discoverable by theGatewayin the same scope.
2. Mismatched Host in VirtualService and Request
-
Diagnosis: The
hostsfield in yourVirtualServicemust exactly match theHostheader of the incoming HTTP request. If you’re accessingmy-app.example.com, yourVirtualServiceneedshosts: ["my-app.example.com"]. Wildcards are supported (e.g.,hosts: ["*.example.com"]).# Capture traffic hitting the ingress gateway and inspect Host header kubectl tcpdump -nL -i <ingress-gateway-node-interface> port 80 or port 443 -s 0 -w - | \ tcpdump -r - 'tcp port 80 or tcp port 443' -vvvAlternatively, use Istio’s traffic mirroring or
istioctl proxy-config listenerson the ingress gateway pod to see what hosts it’s advertising. -
Fix: Update the
hostsfield in yourVirtualServiceto match the expectedHostheader.apiVersion: networking.istio.io/v1alpha3 kind: VirtualService metadata: name: my-app-vs namespace: default # Must match Gateway namespace or be specified in Gateway spec: hosts: - "my-app.example.com" # Ensure this matches your request's Host header gateways: - my-ingress-gateway # Reference the Gateway http: - route: - destination: host: my-app-service port: number: 80This provides the precise binding Istio needs to direct traffic.
3. Mismatched Path in VirtualService’s match Rule
-
Diagnosis: If your
VirtualServicehas anhttpsection withmatchrules (e.g.,uri.prefix: "/api"), the incoming request’s URI must also match this prefix. A request to/won’t match a rule expecting/api.# Again, use tcpdump or Istio's debugging tools on the ingress gateway # to inspect the request URI. kubectl logs <ingress-gateway-pod> -c istio-proxy -n istio-system -fLook for
RST_STREAMor404responses and check the upstream cluster name. If it’sPassthroughorBlackHole, the request didn’t match any defined route. -
Fix: Adjust the
matchrules in yourVirtualServiceto accurately reflect the request path.apiVersion: networking.istio.io/v1alpha3 kind: VirtualService metadata: name: my-app-vs namespace: default spec: hosts: - "my-app.example.com" gateways: - my-ingress-gateway http: - match: - uri: prefix: "/api" # This must match the request path, e.g., /api/users route: - destination: host: my-api-service port: number: 8080This ensures that requests with paths starting with
/apiare correctly routed.
4. Empty or Incorrect Gateway Reference in VirtualService
-
Diagnosis: The
VirtualServicemust explicitly reference theGatewayit’s supposed to be bound to using thegatewaysfield. If this field is missing, or references aGatewaythat doesn’t exist or is in the wrong namespace, theVirtualServicewon’t be activated.kubectl get gateway my-ingress-gateway -n istio-system -o yaml kubectl get virtualservice my-app-vs -n default -o yamlVerify the
gatewaysfield in theVirtualServicepoints to the correctGatewayname and namespace. -
Fix: Add or correct the
gatewaysfield in yourVirtualService.apiVersion: networking.istio.io/v1alpha3 kind: VirtualService metadata: name: my-app-vs namespace: default spec: hosts: - "my-app.example.com" gateways: - istio-system/my-ingress-gateway # Explicitly specify namespace/name http: - route: - destination: host: my-app-service port: number: 80This linkage is crucial for Istio to associate the routing rules with the ingress point.
5. Gateway Resource Not Applied or Incorrectly Configured
-
Diagnosis: The
Gatewayresource itself must be correctly applied and configured to listen on the expected ports and hosts. If theGatewayisn’t active, noVirtualServicecan bind to it.kubectl get gateway my-ingress-gateway -n istio-system -o yaml kubectl describe gateway my-ingress-gateway -n istio-systemCheck the
selectoron theGatewayto ensure it matches the labels on your ingress gateway pods (usuallyistio: ingressgateway). Also, verify theserverssection defines the correct ports (e.g.,port: { number: 80, name: http }). -
Fix: Ensure your
Gatewayresource is applied and has the correct selector and server configuration.apiVersion: networking.istio.io/v1alpha3 kind: Gateway metadata: name: my-ingress-gateway namespace: istio-system spec: selector: istio: ingressgateway # Must match ingress gateway pod labels servers: - port: number: 80 name: http protocol: HTTP hosts: - "*" # Or your specific domainThis makes the ingress gateway ready to receive traffic and bind
VirtualServices.
6. Typo in Destination Service Hostname
-
Diagnosis: Within the
VirtualService’shttproute, thedestination.hostmust be the actual service name as known by Istio’s internal service registry (which is typically the Kubernetes service name). A typo here means Istio tries to route to a non-existent service.kubectl get svc -n <your-service-namespace> kubectl get virtualservice my-app-vs -n default -o yamlCompare the
destination.hostin yourVirtualServicewith the output ofkubectl get svc. -
Fix: Correct the
destination.hostin theVirtualServiceto match the actual Kubernetes service name.apiVersion: networking.istio.io/v1alpha3 kind: VirtualService metadata: name: my-app-vs namespace: default spec: hosts: - "my-app.example.com" gateways: - my-ingress-gateway http: - route: - destination: host: my-app-service.my-namespace.svc.cluster.local # Fully qualified or short name port: number: 80This ensures traffic is directed to the correct backing service.
7. Istio Control Plane Not Healthy
-
Diagnosis: If Istiod (the control plane) is unhealthy, it won’t be able to push configuration updates to the data plane (Envoy proxies in the gateway and sidecars). This can lead to stale configurations where new
VirtualServicesaren’t recognized.kubectl get pods -n istio-system kubectl logs <istiod-pod-name> -n istio-system -c discoveryLook for any
Error,CrashLoopBackOff, or repeated restarts in theistio-systemnamespace, particularly foristiod. -
Fix: Troubleshoot and resolve the underlying issues causing Istiod to be unhealthy. This might involve resource constraints, network policies blocking communication, or bugs. Restarting Istiod pods can sometimes resolve transient issues.
kubectl rollout restart deployment istiod -n istio-systemA healthy Istiod is critical for distributing all Istio configurations.
After fixing these, you’ll likely encounter a "503 Service Unavailable" if the backend service itself is not running or reachable.