The Istio ingress gateway failed to resolve the DNS names of external services, preventing external traffic from reaching services outside the mesh.
Cause 1: CoreDNS configuration missing hosts block for external names
Diagnosis:
Check the CoreDNS configuration loaded by Istio. This is typically found in the istio-system namespace, often as a ConfigMap named coredns or similar. Look for a hosts block that explicitly maps external DNS names to IP addresses. If it’s missing, CoreDNS will try to resolve them externally and potentially fail.
kubectl get configmap coredns -n istio-system -o yaml
Fix:
Add a hosts block to the CoreDNS configuration to manually resolve the external service’s DNS name. For example, if your external service is my-external-service.example.com and its IP is 192.0.2.100:
apiVersion: v1
kind: ConfigMap
metadata:
name: coredns
namespace: istio-system
data:
Corefile: |
.:53 {
errors
health {
lameduck 5s
}
ready
kubernetes cluster.local in-addr.arpa ip6.arpa {
pods insecure
fallthrough in-addr.arpa ip6.arpa
ttl 30
}
prometheus :9153
cache 30
forward . /etc/resolv.conf
# Add this block
hosts {
192.0.2.100 my-external-service.example.com
fallthrough
}
}
Why it works: This forces CoreDNS to return the correct IP address for the external service directly, bypassing the need for external DNS lookups that might be failing or being blocked.
Cause 2: ExternalName Service definition incorrect
Diagnosis:
Verify that the Istio ServiceEntry resource correctly defines the external service. The hosts field should match the DNS name used in your application, and the addresses field should contain the IP address of the external service.
kubectl get serviceentry -n istio-system <your-service-entry-name> -o yaml
Fix:
Ensure the ServiceEntry correctly points to the external service. For an external name resolution, the resolution should be DNS.
apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
name: my-external-service
namespace: istio-system
spec:
hosts:
- my-external-service.example.com
ports:
- number: 80
name: http
protocol: HTTP
resolution: DNS
location: MESH_EXTERNAL
Why it works: This tells Istio that my-external-service.example.com is an external service that should be resolved via DNS. Istio’s Envoy proxies will then attempt to resolve this name.
Cause 3: Network Policy blocking DNS lookups
Diagnosis:
Check Kubernetes NetworkPolicy resources in the istio-system namespace that might be restricting egress traffic from the Istio pods (especially CoreDNS and the ingress gateway) to external DNS servers (UDP/TCP port 53).
kubectl get networkpolicy -n istio-system
Fix:
If a restrictive NetworkPolicy is found, add an egress rule allowing DNS traffic to your cluster’s upstream DNS servers or to 0.0.0.0/0 on port 53.
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-dns-egress
namespace: istio-system
spec:
podSelector: {} # Applies to all pods in istio-system
policyTypes:
- Egress
egress:
- to:
- ipBlock:
cidr: 0.0.0.0/0 # Or your specific upstream DNS server CIDR
except:
- 10.0.0.0/8 # Exclude internal cluster IPs if necessary
ports:
- protocol: UDP
port: 53
- protocol: TCP
port: 53
Why it works: This explicitly permits the DNS queries originating from Istio components to reach external DNS resolvers, enabling them to obtain the IP address for the external service.
Cause 4: Istio Sidecar Injection not enabled for the calling pod
Diagnosis:
Ensure that the Kubernetes pods making requests to the external service have the Istio sidecar injected. If the sidecar is not present, traffic will bypass Istio’s proxy and won’t benefit from the ServiceEntry or CoreDNS configurations.
kubectl get pods -n <your-app-namespace> -o json | jq '.items[].spec.containers[].name'
Look for a container named istio-proxy in the output for your application pods.
Fix:
Enable automatic sidecar injection for the namespace where your application pod resides by setting the istio-injection label.
kubectl label namespace <your-app-namespace> istio-injection=enabled
Then, redeploy your application pods for the sidecar to be injected.
Why it works: The Istio sidecar intercepts all network traffic from the pod. When configured via ServiceEntry, it can then correctly route or resolve external service requests through Istio’s control plane.
Cause 5: Incorrect ServiceEntry resolution type
Diagnosis:
Double-check the resolution field in your ServiceEntry. If it’s set to STATIC but you intend for the IP to be resolved dynamically via DNS, this will cause failures.
kubectl get serviceentry <your-service-entry-name> -n istio-system -o yaml
Fix:
Ensure the resolution field is set to DNS for external services that rely on DNS resolution.
spec:
# ... other fields
resolution: DNS
# ...
Why it works: DNS resolution tells Istio’s Envoy proxy to perform a DNS lookup for the host specified in the ServiceEntry when traffic is routed to it.
Cause 6: External DNS server unreachable or misconfigured
Diagnosis:
If CoreDNS is configured to forward to external DNS servers (e.g., forward . /etc/resolv.conf), and those upstream servers are unreachable from the Kubernetes cluster, DNS resolution will fail. You can test this by exec’ing into a CoreDNS pod and running dig or nslookup.
kubectl exec -it -n istio-system <coredns-pod-name> -- dig my-external-service.example.com
Fix:
Update the forward directive in the CoreDNS Corefile to point to reachable external DNS servers, or ensure network connectivity from your cluster to the existing upstream DNS servers.
# In istio-system/coredns ConfigMap, data.Corefile
.:53 {
# ...
forward . 8.8.8.8 8.8.4.4 { # Example: Google Public DNS
max_concurrent 100
}
# ...
}
Why it works: By directing CoreDNS to known-good and reachable DNS servers, it can successfully perform the necessary lookups for the external service.
The next error you’ll likely encounter if all external name resolution issues are fixed is related to TLS handshake failures if the external service requires HTTPS and the ServiceEntry or application isn’t configured for it.