Kubernetes Network Policy is failing to block traffic because the CNI plugin managing network rules doesn’t support or enforce the policy correctly.
Common Causes and Fixes:
-
CNI Plugin Doesn’t Support NetworkPolicy:
- Diagnosis: Check your cluster’s CNI plugin. Many basic CNIs (like
bridgeor olderflannelversions withoutvxlanorhost-gwbackend) do not implement KubernetesNetworkPolicyresources.
If you seekubectl get ds -n kube-system # Look for your CNI daemonset, e.g., calico-node, flannel, cilium, weave-netflannelwith a configuration that doesn’t explicitly mentionvxlanorhost-gw, or a custom CNI that isn’t listed as supportingNetworkPolicy, this is your problem. - Fix: Replace your CNI with one that fully supports
NetworkPolicy. Popular choices include Calico, Cilium, or Weave Net. For example, to switch to Calico:- Uninstall your current CNI.
- Install Calico. The official installation manifest is usually found on their GitHub releases page.
# Example for a typical Calico install (check official docs for latest version) kubectl apply -f https://raw.githubusercontent.com/projectcalico/calico/v3.27.0/manifests/calico.yaml - Why it works: Network policies are implemented by the CNI plugin. If the plugin isn’t designed to understand and enforce these policies, they will have no effect. A compliant CNI translates
NetworkPolicyobjects into actual firewall rules (like iptables or eBPF) on each node.
- Diagnosis: Check your cluster’s CNI plugin. Many basic CNIs (like
-
NetworkPolicy is Applied to the Wrong Namespace:
- Diagnosis: Network policies are namespace-scoped. A policy defined in
namespace-aonly affects pods withinnamespace-a. If your pods are indefaultand your policy is inkube-system, it won’t do anything.# Check the namespace of your policy kubectl get networkpolicy <policy-name> -n <policy-namespace> -o yaml # Check the namespace of the pods you expect to be affected kubectl get pods --all-namespaces - Fix: Ensure the
metadata.namespacefield in yourNetworkPolicydefinition matches the namespace of the pods you intend to control.apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: deny-all-ingress namespace: my-app-namespace # <-- Ensure this is correct spec: podSelector: matchLabels: app: my-app policyTypes: - Ingress ingress: [] # Empty ingress means deny all ingress by default - Why it works: Kubernetes resources, including NetworkPolicies, are namespaced. The control plane and the CNI plugin look for policies within the same namespace as the target pods to apply the rules.
- Diagnosis: Network policies are namespace-scoped. A policy defined in
-
podSelectoris Too Broad or Incorrect:- Diagnosis: The
podSelectorin aNetworkPolicydetermines which pods the policy applies to. If it’s empty ({}) or matches all pods, the policy applies to all pods in the namespace. If it doesn’t match the labels of the pods you want to restrict, the policy won’t affect them.
To check the labels of your pods:# Policy applying to all pods in 'my-app-namespace' apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: restrict-external namespace: my-app-namespace spec: podSelector: {} # <-- This matches ALL pods in the namespace ingress: - from: - podSelector: matchLabels: app: backend # Only allows traffic from pods with label app=backendkubectl get pods -n <your-namespace> --show-labels - Fix: Adjust the
podSelectorto accurately target the pods that should be subject to the policy.apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: restrict-external namespace: my-app-namespace spec: podSelector: matchLabels: app: frontend # <-- This policy now ONLY applies to pods with label app=frontend ingress: - from: - podSelector: matchLabels: app: backend - Why it works: The
podSelectoracts as a filter. The CNI plugin uses these labels to identify the specific network interfaces and endpoints on which to enforce the defined ingress and egress rules.
- Diagnosis: The
-
Missing
policyTypesor IncorrectpolicyTypes:- Diagnosis: A
NetworkPolicycan specifyingress,egress, or both. IfpolicyTypesis omitted, it defaults toIngressifingressrules are present, andEgressifegressrules are present. If you intend to block all ingress traffic, you must explicitly definepolicyTypes: ["Ingress"]and an emptyingress: []section. If you want to block all egress, you needpolicyTypes: ["Egress"]and an emptyegress: []section.# This policy *allows* all ingress by default if policyTypes is omitted or only egress is specified apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: allow-all-ingress-by-mistake namespace: my-app-namespace spec: podSelector: matchLabels: app: my-app # policyTypes is missing, and ingress is defined, so it defaults to Ingress only ingress: - from: [] # Empty from *with* ingress defined means "allow all ingress" - Fix: Explicitly set
policyTypesto include what you want to control. To block all ingress, usepolicyTypes: ["Ingress"]and an emptyingress: []list. To block all egress, usepolicyTypes: ["Egress"]and an emptyegress: []list.apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: deny-all-ingress namespace: my-app-namespace spec: podSelector: matchLabels: app: my-app policyTypes: - Ingress # <-- Explicitly state we are controlling ingress ingress: [] # <-- An empty ingress rule denies all ingress traffic - Why it works: The
policyTypesfield tells the CNI which direction of traffic (ingress, egress, or both) the policy rules apply to. Without it, or with an incorrect setting, the CNI might not apply rules as expected, or might default to an "allow all" behavior for unspecified types.
- Diagnosis: A
-
Traffic is Not Pod-to-Pod:
- Diagnosis:
NetworkPolicyprimarily controls traffic between pods within the cluster. Traffic originating from outside the cluster (e.g., via a LoadBalancer Service, NodePort, or directly to a node’s IP) or traffic to external services is not affected by default. If your traffic is coming from a LoadBalancer, yourNetworkPolicywon’t block it unless you have specific CNI features or additional configurations.# Check the type of the service exposing your application kubectl get svc <your-service-name> -n <your-namespace> -o yaml # Look for 'type: LoadBalancer' or 'type: NodePort' - Fix: For external traffic, you need to configure your Ingress controller or Service to restrict access. For traffic to external endpoints, you need to define
egressrules withtoselectors that explicitly allow connections to external IPs or CIDRs, or use a default-deny egress policy and then allow only necessary outbound connections.# Example: Allow egress to specific external CIDRs apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: allow-external-dns namespace: my-app-namespace spec: podSelector: matchLabels: app: my-app policyTypes: - Egress egress: - to: - ipBlock: cidr: 8.8.8.8/32 # Allow egress to Google DNS ports: - protocol: UDP port: 53 - to: - ipBlock: cidr: 1.1.1.1/32 # Allow egress to Cloudflare DNS ports: - protocol: UDP port: 53 - Why it works:
NetworkPolicyoperates at the pod network layer. External traffic bypasses this layer initially. Ingress controllers and LoadBalancer implementations handle traffic before it reaches the pod network, and external network routing handles egress traffic.
- Diagnosis:
-
CNI Firewall Rules Not Applying (iptables/eBPF issues):
- Diagnosis: In some cases, the CNI might be configured correctly, but the underlying firewall rules (often
iptablesfor older CNIs oreBPFfor newer ones) are not being applied or are being overwritten. This can happen if other components on the node are also manipulatingiptablesor if the CNI agent has crashed or is misconfigured.
If you see no rules, or rules that don’t correspond to your# For iptables-based CNIs (e.g., older flannel, weave) # Check the KUBE-NETWORK-POLICY chain in the filter table sudo iptables -t filter -L KUBE-NETWORK-POLICY -n # For eBPF-based CNIs (e.g., Cilium, Calico with eBPF mode) # This is more complex and CNI-specific. For Cilium, you might check: cilium status # Check Cilium agent health # And potentially examine eBPF maps if you're deep in troubleshootingNetworkPolicydefinitions, this is the issue. - Fix: Restart the CNI daemonset pods on the affected nodes. If the problem persists, consider reinstalling the CNI or debugging the CNI agent logs.
# Find the CNI daemonset pods kubectl get pods -n kube-system -l <cni-label-selector> # Delete them to force a restart (the daemonset will recreate them) kubectl delete pod <cni-pod-name> -n kube-system - Why it works: The CNI agent on each node is responsible for programmatically adding and managing the firewall rules that enforce
NetworkPolicy. A restart ensures it re-evaluates its state and applies the correct rules based on the cluster’sNetworkPolicyobjects.
- Diagnosis: In some cases, the CNI might be configured correctly, but the underlying firewall rules (often
You’ll next run into issues with NetworkPolicy affecting DNS resolution if you’re too restrictive with egress.