VXLAN essentially allows you to run a virtual network that’s completely independent of the physical network it’s running on, like having a separate, invisible network laid over your existing one.
Let’s see it in action. Imagine you have two servers, server-a and server-b, physically connected to the same IP network. We want them to talk as if they were on the same Layer 2 segment, even though they might be miles apart.
First, we need to set up a VXLAN tunnel endpoint (VTEP) on each server. This is often done via software, like the vxlan kernel module in Linux.
On server-a:
sudo ip link add vxlan0 type vxlan id 100 remote 192.168.1.2 dstport 4789
sudo ip link set vxlan0 up
sudo ip addr add 10.0.0.1/24 dev vxlan0
On server-b:
sudo ip link add vxlan0 type vxlan id 100 remote 192.168.1.1 dstport 4789
sudo ip link set vxlan0 up
sudo ip addr add 10.0.0.2/24 dev vxlan0
Here, id 100 is the VXLAN Network Identifier (VNI), which acts like a VLAN ID but for the overlay network. remote 192.168.1.2 is the IP address of the other VTEP. dstport 4789 is the standard UDP port for VXLAN. We’ve assigned IP addresses 10.0.0.1 and 10.0.0.2 to the vxlan0 interfaces, creating our virtual Layer 2 segment.
Now, if you ping 10.0.0.2 from server-a, it will work. The vxlan0 interface on server-a sees the ping destined for 10.0.0.2. It encapsulates the Ethernet frame from the ping packet into a UDP packet. The source IP of this UDP packet is server-a’s physical IP (e.g., 192.168.1.1), and the destination IP is server-b’s physical IP (192.168.1.2). This UDP packet is then sent over the physical IP network. On server-b, the kernel receives the UDP packet on port 4789, sees the VNI matches its vxlan0 configuration, decapsulates the Ethernet frame, and delivers it to the vxlan0 interface.
This mechanism solves the problem of extending Layer 2 networks across Layer 3 boundaries. Traditionally, this required complex spanning tree protocols or dedicated Layer 2 circuits. VXLAN lets you create these isolated, virtual Layer 2 segments (identified by VNIs) and run them as encapsulated UDP/IP packets over any IP network, including the internet. You can have thousands of these VNIs, each acting as a separate broadcast domain, effectively giving you multitenancy and network segmentation at scale. The physical network just needs to be able to route IP packets between the VTEPs.
The maximum transmission unit (MTU) of the underlying physical network is a critical, often overlooked, constraint. Because VXLAN adds an Ethernet header, an IP header, and a UDP header (around 50 bytes) to the original Ethernet frame, you can easily exceed the typical 1500-byte MTU of many networks. If the underlying network doesn’t support jumbo frames (e.g., MTU of 9000 bytes), you’ll experience dropped packets for larger frames, leading to performance degradation or connectivity issues that are hard to debug because the VXLAN tunnel itself appears up and healthy. You’ll often need to configure ip link set vxlan0 mtu 1450 (or similar, depending on the physical network’s MTU) on your VTEPs to account for the VXLAN overhead.
The next step is understanding how to map MAC addresses to VTEP IP addresses for inter-VXLAN communication.