VXLAN is a bit like a magic trick for your network, letting you stretch a single Layer 2 network across a vast Layer 3 infrastructure, as if they were right next to each other.
Let’s see this in action. Imagine you have two servers, server-a and server-b, on different physical subnets, but you want them to believe they’re on the same L2 segment. We’ll use a simple VXLAN setup with two network devices, leaf-1 and leaf-2, acting as VXLAN tunnel endpoints (VTEPs).
First, we need to configure the VTEPs on our switches. On leaf-1, we’d configure something like this:
interface vxlan1
vxlan udp-port 4789
vxlan vrf L2_TENANT_1
vxlan remote-peer 192.168.2.2 vrf L2_TENANT_1
vxlan vni 10001
And on leaf-2:
interface vxlan1
vxlan udp-port 4789
vxlan vrf L2_TENANT_1
vxlan remote-peer 192.168.1.2 vrf L2_TENANT_1
vxlan vni 10001
Here, vxlan udp-port 4789 is the standard UDP port for VXLAN. The vxlan vrf L2_TENANT_1 command associates this VXLAN interface with a specific Layer 3 VRF, isolating traffic. Crucially, vxlan remote-peer 192.168.2.2 vrf L2_TENANT_1 on leaf-1 tells it to send VXLAN traffic destined for VNI 10001 to the IP address 192.168.2.2 (which is leaf-2’s IP). The vxlan vni 10001 command assigns this VXLAN tunnel to Virtual Network Identifier 10001, essentially creating our L2 segment.
Now, if server-a (connected to leaf-1 on IP 10.0.1.10) sends an ARP request for server-b’s IP (10.0.1.20), leaf-1 will see this L2 frame. Since leaf-1 knows that IP 10.0.1.20 is part of VNI 10001, it will encapsulate the original L2 frame within a VXLAN header. The source IP of the outer IP header will be leaf-1’s IP (192.168.1.2), and the destination IP will be leaf-2’s IP (192.168.2.2). This VXLAN-encapsulated packet then travels across the L3 network. leaf-2 receives the packet, strips off the VXLAN header, and forwards the original L2 frame to server-b. When server-b responds, the process is reversed.
The fundamental problem VXLAN solves is the limitation of traditional L2 networks. Spanning tree protocols (STP) prevent loops but also block redundant links, severely limiting bandwidth and scalability. Extending L2 across large L3 fabrics using STP is impractical, if not impossible. VXLAN sidesteps this by tunneling L2 frames over an existing L3 underlay. The L3 network simply sees UDP packets between VTEP IP addresses, and the L2 topology is abstracted away.
Internally, VXLAN uses a 42-byte header. The most important part for us is the 24-bit Virtual Network Identifier (VNI). This VNI acts like a VLAN ID but on a much grander scale, allowing for up to 16 million unique L2 segments. This is a massive leap from the 4094 VLANs available in traditional Ethernet. The VTEPs are responsible for encapsulating and decapsulating these VXLAN packets, effectively acting as the gateways between the L2 overlay and the L3 underlay.
When a VTEP learns the MAC address of a host on its local segment, it also learns the VTEP IP address associated with that MAC. This is typically done via a control plane, most commonly BGP EVPN (Ethernet VPN). For instance, if server-a on leaf-1 sends traffic to server-b on leaf-2, leaf-1 will query its MAC address table. If it finds server-b’s MAC associated with VTEP 192.168.2.2, it will encapsulate the frame with the outer IP header pointing to 192.168.2.2. If the MAC address is not in the table, the VTEP might flood the frame within the VNI, similar to an unknown unicast flood in a traditional L2 network, but this flood is contained within the VNI and directed to other VTEPs participating in that VNI.
A common configuration detail often overlooked is the management of MAC address tables on the VTEPs. These tables store the mapping between MAC addresses and the VTEP IP addresses they are reachable from. If these tables become too large or are not properly synchronized (e.g., via BGP EVPN), traffic might be misdirected or dropped. For example, on Cisco Nexus switches, you might see commands like show nve peers to verify VTEP connectivity and show mac address-table vlan <vlan_id> or show mac address-table vxlan <vni> to inspect MAC learning.
The next step you’ll typically encounter is integrating this VXLAN overlay with routing, often involving Layer 3 Virtual Network Identifiers (L3 VNIs) and inter-VNI routing.