Overlay Networking: Virtual Networks Over Physical (2026)

Overlay networks are how you get IP connectivity to machines that can’t directly talk to each other on the underlying physical network.

Imagine you have two servers, server-a and server-b. server-a is in Rack 1, server-b is in Rack 5. They’re on different subnets, say 10.0.1.10/24 and 10.0.5.20/24. They can’t ping each other. The physical network (switches, routers) doesn’t know how to get traffic from Rack 1 to Rack 5 for those specific IPs.

Here’s server-a trying to ping server-b:

# On server-a
ping 10.0.5.20
# Output:
# PING 10.0.5.20 (10.0.5.20) 56(84) bytes of data.
# From 10.0.1.10 icmp_seq=1 Destination Host Unreachable
# From 10.0.1.10 icmp_seq=2 Destination Host Unreachable

This is where an overlay network comes in. It carves out a virtual network on top of the physical one. Think of it like a private tunnel. server-a and server-b will get new IP addresses within this virtual network, say 192.168.1.10 and 192.168.1.20. Now they can talk to each other using these virtual IPs.

Let’s set up a simple overlay using VXLAN (Virtual Extensible LAN). We’ll use flanneld as our overlay daemon.

First, on both server-a and server-b, install flannel:

# On both servers (example for Ubuntu/Debian)
sudo apt update
sudo apt install flannel

Next, we need a way for flannel daemons to discover each other and agree on the network. A common method is using etcd, a distributed key-value store. Let’s assume you have an etcd cluster running.

On server-a, configure flanneld to point to etcd and define the VXLAN subnet:

# On server-a
cat <<EOF | sudo tee /etc/flannel/flanneld.conf
{
  "etcd-endpoints": "http://etcd-server-1:2379,http://etcd-server-2:2379",
  "etcd-prefix": "/coreos.com/network",
  "network": "10.0.0.0/16",
  "iface": "eth0"
}
EOF

The key part here is "network": "10.0.0.0/16". This tells flanneld to create a VXLAN network using this CIDR. flanneld will pick an IP from this range for server-a.

Start the flanneld service:

# On both servers
sudo systemctl enable flanneld
sudo systemctl start flanneld

After a moment, flanneld on each server will:

Contact etcd to get its assigned IP from the 10.0.0.0/16 range and the configuration for the VXLAN network.
Program its local network stack to create a VXLAN interface.
For each other flanneld instance it discovers (via etcd), it will set up a VXLAN tunnel. Traffic destined for another node’s overlay IP will be encapsulated in UDP packets and sent to that node’s physical IP.

Now, check the network interfaces on server-a:

# On server-a
ip addr show flannel.1
# Output will look something like:
# 5: flannel.1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1472 qdisc noqueue state UNKNOWN group default qlen 500
#     link/ether ee:3f:07:8c:e8:70 brd ff:ff:ff:ff:ff:ff
#     inet 10.0.1.10/16 scope global flannel.1
#        valid_lft forever preferred_lft forever

Notice inet 10.0.1.10/16. This is server-a’s IP in the overlay network. server-b will have something like 10.0.5.20/16.

Now, try pinging server-b’s overlay IP from server-a:

# On server-a
ping 10.0.5.20
# Output:
# PING 10.0.5.20 (10.0.5.20) 56(84) bytes of data.
# 64 bytes from 10.0.5.20: icmp_seq=1 ttl=64 time=2.34 ms
# 64 bytes from 10.0.5.20: icmp_seq=2 ttl=64 time=2.41 ms

Success! The flanneld daemons on both servers are handling the encapsulation and decapsulation. When server-a sends a ping to 10.0.5.20, flanneld sees this is an overlay IP. It looks up the physical IP of the node with 10.0.5.20 (let’s say 10.0.5.20 is on server-b at physical IP 10.0.5.20), encapsulates the ICMP packet inside a UDP packet with destination port 8472 (the default VXLAN port), and sends it to server-b’s physical IP. flanneld on server-b receives the UDP packet, decapsulates the ICMP packet, and delivers it to the flannel.1 interface.

The magic is that flanneld programs your routing table. When you ping 10.0.5.20, your kernel sees that 10.0.5.0/16 is routed via the flannel.1 interface. The flannel.1 interface knows to send this traffic to the flanneld process, which then handles the VXLAN encapsulation.

The physical network only sees standard IP packets between the physical IP addresses of server-a and server-b. It doesn’t know or care about the 10.0.1.10 or 10.0.5.20 IPs.

The actual IP address assigned to flannel.1 on each host is determined by flanneld based on its configuration and discovery mechanisms. For example, if you used --kube-subnet-mgr with Kubernetes, it would coordinate with the Kubernetes API server. For etcd, it uses keys like /coreos.com/network/subnets to allocate IPs from the defined network CIDR.

The MTU (Maximum Transmission Unit) on the flannel.1 interface is often reduced (e.g., 1472) to account for the VXLAN header (typically 50 bytes for header + UDP/IP overhead), preventing IP fragmentation.

This ability to create isolated, routable networks between any two endpoints, regardless of their physical location or subnet, is what makes overlay networks fundamental to modern cloud infrastructure and container orchestration.

The next step is usually managing these overlay networks at scale, often with systems like Kubernetes which use CNI (Container Network Interface) plugins to automate the setup and configuration of overlays for pods.