Kubernetes Pod Networking Deep Dive

The most surprising thing about Kubernetes pod networking is that Kubernetes itself doesn’t actually do networking. It delegates all the complex, low-level packet forwarding, IP address management, and network policy enforcement to separate components called Container Network Interface (CNI) plugins.

Let’s see this in action. Imagine you have a simple deployment with two pods, pod-a and pod-b, on the same Kubernetes node.

# pod-a.yaml
apiVersion: v1
kind: Pod
metadata:
  name: pod-a
spec:
  containers:
  - name: nginx
    image: nginx:latest
    ports:
    - containerPort: 80

# pod-b.yaml
apiVersion: v1
kind: Pod
metadata:
  name: pod-b
spec:
  containers:
  - name: busybox
    image: busybox:latest
    command: ["sleep", "3600"]

When you apply these, Kubernetes doesn’t magically connect them. Instead, the Kubelet on the node, upon seeing a new pod, calls out to the configured CNI plugin. The CNI plugin then:

1. Allocates an IP address to the pod from a pre-defined subnet for that node.
2. Creates a network interface (often a veth pair) inside the pod’s network namespace.
3. Connects that interface to the node’s network, often via a bridge (like cbr0 for bridge CNI, or more complex setups for calico or cilium).
4. Configures routing rules so that other pods on the same node (and eventually across nodes) can reach this pod’s IP, and vice-versa.

This means the actual how of pod-to-pod communication, service discovery, and network policy is entirely up to the CNI plugin you choose. The most common ones you’ll encounter are:

• bridge: This is the simplest, often used in development environments like minikube or kind. It creates a Linux bridge on each node and connects pod interfaces to it. Pods on the same node can communicate directly. Cross-node communication often relies on IP masquerading or NAT.
• host-local: This plugin doesn’t do much on its own. It’s often used as a prerequisite for other plugins, primarily for IP address allocation. It reserves IP address ranges for pods on a given node.
• calico: A very popular choice for production. Calico uses BGP or VXLAN to route traffic directly between pods on different nodes, effectively treating pods as first-class citizens on your network. It also provides robust network policy enforcement.
• cilium: Another powerful, production-ready option. Cilium leverages eBPF (extended Berkeley Packet Filter) to provide high-performance networking, load balancing, and security without modifying the kernel’s networking stack or relying on traditional iptables. It can also enforce network policies at a very granular level.
• flannel: A relatively simple CNI that focuses on providing a flat network overlay across nodes, often using VXLAN or UDP encapsulation. It’s easier to set up than Calico or Cilium but might offer less advanced features.

To see which CNI is active on your cluster, you’d typically look at the configuration of your Kubelet. On most systems, this is specified in a configuration file, often located at /var/lib/kubelet/config.yaml or managed by a systemd service file. You’ll find a networkPlugin setting or, more commonly, a cniConfDir and cniBinDir that point to where the CNI configuration and binaries reside.

For example, if you see a directory like /etc/cni/net.d/ containing a file named 10-calico.conflist or 10-flannel.conflist, that’s your clue.

Here’s a snippet you might find in /etc/cni/net.d/10-calico.conflist (simplified):

{
  "name": "k8s-pod-network",
  "cniVersion": "0.3.1",
  "plugins": [
    {
      "type": "calico",
      "logFile": "/var/log/calico/cni.log",
      "logSeverity": "INFO",
      "datastoreType": "kubernetes",
      "nodename": "__KUBERNETES_NODE_NAME__",
      "mtu": 1476,
      "ipam": {
        "type": "calico-ipam"
      },
      "policy": {
        "type": "calico"
      },
      "kubernetes": {
        "kubeconfig": "__KUBECONFIG_FILEPATH__"
      }
    },
    {
      "type": "portmap",
      "capabilities": {"portMappings": true}
    }
  ]
}

This configuration tells the CNI plugin manager (which is usually ptpd or a similar daemon) to execute the calico plugin first. The calico plugin itself is responsible for IP address management (ipam) and network policy enforcement (policy). It then might run a portmap plugin to handle host port mappings.

The mtu (Maximum Transmission Unit) setting is crucial. If you’re using an overlay network (like VXLAN with Calico or Flannel), each packet is wrapped in another packet. This encapsulation reduces the effective MTU. A common default is 1500 bytes for Ethernet. If your CNI encapsulates traffic, it often sets the MTU to something like 1476 or 1450 to account for the overlay header, preventing fragmentation issues. Mismatched MTUs across nodes or between your pods and the underlying network are a frequent cause of intermittent connectivity problems or slow transfers.

When you set up network policies, for example, to deny all ingress to a specific pod unless it comes from a particular namespace, it’s the CNI plugin that translates those high-level Kubernetes NetworkPolicy objects into low-level firewall rules (like iptables or eBPF rules) that actually drop or permit packets.

The next logical step after understanding how pods get their IPs and can talk to each other is understanding how they talk to services, which is handled by the kube-proxy component, often in conjunction with the CNI.