The Kubernetes Ingress controller failed to forward incoming requests to the correct backend Service, likely due to a misconfiguration in the Ingress resource itself or an issue with the controller’s internal state.

Common Causes and Fixes

1. Incorrect serviceName or servicePort in Ingress Resource

  • Diagnosis:

    kubectl get ingress <your-ingress-name> -n <your-namespace> -o yaml
    

    Examine the spec.rules.http.paths.backend section for the service.name and service.port.number (or name). Also, check the target Service:

    kubectl get service <your-service-name> -n <your-namespace> -o yaml
    

    Verify that the spec.selector in the Service matches your Pod labels and that the spec.ports correctly define the port and targetPort.

  • Fix: Edit the Ingress resource:

    kubectl edit ingress <your-ingress-name> -n <your-namespace>
    

    Ensure spec.rules.http.paths.backend.service.name exactly matches the name of your Service and spec.rules.http.paths.backend.service.port.number or name matches the port defined in your Service. If using port.name, ensure it’s spelled identically in both the Ingress and Service definitions.

  • Why it works: The Ingress controller uses these fields to dynamically configure its routing rules. A mismatch means the controller attempts to send traffic to a non-existent or incorrectly configured Service.

2. Missing or Incorrect path in Ingress Resource

  • Diagnosis:

    kubectl get ingress <your-ingress-name> -n <your-namespace> -o yaml
    

    Check the spec.rules.http.paths section. For example, if you expect traffic to example.com/api, the path should be /api. The default path is /.

  • Fix: Edit the Ingress resource:

    kubectl edit ingress <your-ingress-name> -n <your-namespace>
    

    Add or correct the path field. For a specific path, use the exact string (e.g., path: /api). For a wildcard, use pathType: Prefix and path: /. Ensure pathType is set appropriately (Exact, Prefix, or ImplementationSpecific). Prefix is common for directory-like paths.

  • Why it works: The Ingress controller uses the path and pathType to match incoming request URLs to specific backend Services. Incorrect path matching means requests never reach the intended destination.

3. Ingress Controller Not Watching the Correct Namespace

  • Diagnosis: Check the configuration of your Ingress controller deployment. The controller deployment (e.g., Nginx Ingress Controller, Traefik) often has arguments or environment variables that specify which namespaces it should watch for Ingress resources.

    kubectl get deployment <ingress-controller-deployment-name> -n <ingress-controller-namespace> -o yaml
    

    Look for arguments like --watch-namespace or environment variables.

  • Fix: If the Ingress controller is configured to watch only specific namespaces, and your Ingress resource is in a different namespace, you have two options:

    • Option A (Recommended if applicable): Move your Ingress resource to a namespace the controller is already watching.
    • Option B: Reconfigure the Ingress controller deployment to watch the namespace containing your Ingress resource. This might involve editing the deployment manifest and restarting the controller pods. For the Nginx Ingress Controller, you might add --watch-namespace=<your-namespace> to the controller’s arguments.
  • Why it works: Ingress controllers are often deployed in a central namespace (like ingress-nginx or kube-system) but can be configured to monitor Ingress objects across multiple or all namespaces. If the controller isn’t aware of the namespace where your Ingress resource resides, it will never process it.

4. Incorrect Annotations for Ingress Controller Behavior

  • Diagnosis:

    kubectl get ingress <your-ingress-name> -n <your-namespace> -o yaml
    

    Review the metadata.annotations section. Different Ingress controllers (Nginx, Traefik, HAProxy) use specific annotations to control advanced routing, SSL termination, rewrite rules, etc. An incorrect or missing annotation can lead to unexpected routing. For example, an annotation meant for Nginx might be applied to a Traefik controller, or vice-versa.

  • Fix: Consult the documentation for your specific Ingress controller.

    • For Nginx Ingress Controller, common annotations include nginx.ingress.kubernetes.io/rewrite-target for path rewriting or nginx.ingress.kubernetes.io/ssl-redirect for HTTPS enforcement.
    • For Traefik, annotations might look like traefik.ingress.kubernetes.io/router.middlewares. Correct any typos, ensure the annotation key is correct for your controller, and verify its value adheres to the controller’s expected format.
  • Why it works: Annotations provide controller-specific instructions. If these instructions are malformed or irrelevant to the controller, the controller cannot interpret the desired routing behavior, leading to routing failures or incorrect application of rules.

5. Backend Pods Not Ready or Not Running

  • Diagnosis: First, ensure the Service is selecting Pods:

    kubectl get endpoints <your-service-name> -n <your-namespace>
    

    If the ENDPOINTS list is empty or shows <none>, the Service isn’t finding any healthy Pods. Then, check the status of the Pods targeted by the Service:

    kubectl get pods -n <your-namespace> -l <your-app-label-key>=<your-app-label-value>
    

    Look for Pods in Running state and check their READY count (e.g., 1/1).

  • Fix: If Pods are not running or not ready:

    • Check Pod logs: kubectl logs <pod-name> -n <your-namespace>
    • Describe Pods for events: kubectl describe pod <pod-name> -n <your-namespace>
    • Ensure the application inside the container is listening on the targetPort defined in the Service and is healthy.
    • Verify that the Pod’s readinessProbe (if configured) is passing.
  • Why it works: The Ingress controller routes traffic to Service endpoints. If the Service has no healthy endpoints because the backend Pods are unhealthy, crashing, or not yet ready, the Ingress controller has nowhere to send the traffic, and requests will fail.

6. Network Policy Blocking Traffic

  • Diagnosis: Check if any NetworkPolicy resources are applied in the namespace of your Service or Pods.

    kubectl get networkpolicy -n <your-namespace>
    

    If policies exist, inspect them to see if they allow ingress traffic from the Ingress controller’s namespace or Pods to your application Pods on the required port.

  • Fix: Edit the relevant NetworkPolicy to include a rule that allows traffic from the Ingress controller’s Pods (often identified by labels) or from the Service’s namespace to your application Pods on the targetPort. Example allowing ingress from pods with label app.kubernetes.io/name: ingress-nginx on port 80:

    apiVersion: networking.k8s.io/v1
    kind: NetworkPolicy
    metadata:
      name: allow-ingress
      namespace: <your-app-namespace>
    spec:
      podSelector:
        matchLabels:
          app: your-app
      policyTypes:
        - Ingress
      ingress:
        - from:
            - podSelector:
                matchLabels:
                  app.kubernetes.io/name: ingress-nginx # Label of your ingress controller pods
          ports:
            - protocol: TCP
              port: 80 # The targetPort your app listens on
    
  • Why it works: NetworkPolicy objects act as firewalls at the Pod level. If a policy restricts ingress traffic, it can prevent the Ingress controller’s requests from reaching your application Pods, even if all other configurations are correct.

The next error you’ll likely encounter is a 503 Service Unavailable or a 404 Not Found if the Ingress controller is running but cannot establish a connection to any healthy backend endpoints.

Want structured learning?

Take the full Kubernetes course →