Pods on the same Kubernetes node can talk to each other directly using their IP addresses, but pods on different nodes need a bit more magic.

Let’s see it in action. Imagine two pods, frontend-abcde and backend-fghij, running on different nodes.

# On node1, where frontend-abcde is running
kubectl exec frontend-abcde -- curl http://backend-fghij:8080

# On node2, where backend-fghij is running
kubectl exec backend-fghij -- curl http://frontend-abcde:8080

When you run these commands, Kubernetes makes sure that the network traffic from frontend-abcde destined for backend-fghij’s IP address gets routed correctly, even though they’re on separate machines. It’s not magic, though; it’s a carefully designed network overlay.

The core problem Kubernetes networking solves is providing a flat, routable IP address space for every pod, regardless of which node it lands on. This allows pods to communicate as if they were on the same local network. The primary mechanism for this is the Container Network Interface (CNI). When a pod is created, the Kubelet on that node invokes a CNI plugin. This plugin is responsible for assigning an IP address to the pod and configuring the pod’s network namespace to use that IP.

Common CNI plugins include Calico, Flannel, Cilium, and Weave Net. Each has its own way of creating the overlay network. For example, Flannel often uses VXLAN tunnels to encapsulate pod traffic and send it between nodes. Calico, on the other hand, might use BGP to directly route pod IPs on the underlying network infrastructure.

Here’s a simplified look at the components involved:

  • Pod IP Addresses: Each pod gets a unique IP address from a cluster-wide CIDR range. These IPs are only routable within the cluster.
  • Kube-proxy: Running on each node, kube-proxy watches the Kubernetes API for Service and Endpoint changes. It programs network rules (often using iptables or IPVS) on the node to intercept traffic destined for a Service IP and direct it to the appropriate backend pod IPs.
  • CNI Plugin: As mentioned, this is the workhorse for pod-to-pod networking. It sets up the network interfaces and routing for each pod.
  • Network Policies: Kubernetes Network Policies allow you to define rules about which pods are allowed to communicate with each other, providing a firewall at the pod level.

Consider a Service named my-app that exposes a set of backend pods. When frontend-abcde wants to talk to my-app, it actually sends traffic to the Service’s ClusterIP (e.g., 10.96.0.10). kube-proxy on frontend-abcde’s node intercepts this traffic. It looks up the actual IP addresses of the pods backing my-app (e.g., 10.244.1.5 and 10.244.2.8) and then uses iptables rules to randomly select one of these pod IPs and forward the traffic. If the selected pod is on a different node, the CNI plugin on the source node ensures the packet is routed correctly to the destination node, where it’s then delivered to the target pod.

The CNI plugin’s configuration is crucial. For instance, if you’re using Flannel with VXLAN, the CNI configuration might specify a backend type and a subnet mask for the pod network. A typical Flannel configuration might look something like this:

{
  "name": "cni-plugin",
  "type": "flannel",
  "delegate": {
    "isDefaultGateway": true,
    "hairpinMode": "none",
    "network": "10.244.0.0/16",
    "backend": {
      "type": "vxlan",
      "port": 8472
    }
  }
}

This tells Flannel to use the 10.244.0.0/16 CIDR for pod IPs and to encapsulate traffic using VXLAN on UDP port 8472.

A common misconception is that pod IPs are directly routable on your physical network. They are not, unless you’re using a CNI plugin like Calico with BGP configured to peer with your physical routers. For overlay networks, the actual routing happens within the overlay, and the underlying nodes just need to be able to reach each other on their node IPs. The CNI plugin is responsible for creating the virtual network interfaces within the pod’s network namespace and ensuring that traffic destined for other pod IPs is correctly routed via the overlay. This often involves creating virtual network devices (like veth pairs) that connect the pod’s network namespace to the host’s network namespace, and then configuring routing rules on the host to send traffic across the overlay.

Once you’ve got pod-to-pod communication sorted, the next logical step is understanding how to expose those pods to the outside world using Kubernetes Services.

Want structured learning?

Take the full Kubernetes course →