A Virtual IP (VIP) address, managed by keepalived, can seem like magic for achieving high availability (HA) for your haproxy load balancer. It’s not magic; it’s a sophisticated dance of network protocols and state synchronization.
Let’s watch it in action. Imagine we have two haproxy servers, haproxy-1 and haproxy-2. Both are running keepalived. We want a single VIP, say 192.168.1.100, to always be reachable, pointing to whichever haproxy is currently active.
Here’s a snippet of a typical keepalived.conf on haproxy-1:
vrrp_script chk_haproxy {
script "/usr/local/bin/check_haproxy.sh"
interval 2
weight -20
fall 2
rise 2
}
vrrp_instance VI_1 {
state BACKUP
interface eth0
virtual_router_id 51
priority 100
advert_int 1
authentication {
auth_type PASS
auth_pass mysecretpassword
}
virtual_ipaddress {
192.168.1.100/24 dev eth0
}
track_script {
chk_haproxy
}
}
And on haproxy-2, it’s almost identical, but with state MASTER and a slightly lower priority (e.g., priority 90):
vrrp_script chk_haproxy {
script "/usr/local/bin/check_haproxy.sh"
interval 2
weight -20
fall 2
rise 2
}
vrrp_instance VI_1 {
state MASTER
interface eth0
virtual_router_id 51
priority 90
advert_int 1
authentication {
auth_type PASS
auth_pass mysecretpassword
}
virtual_ipaddress {
192.168.1.100/24 dev eth0
}
track_script {
chk_haproxy
}
}
The virtual_router_id (51 in this case) must be the same on both nodes. This ID uniquely identifies the VRRP group. The advert_int (1 second) is how often they’ll send out VRRP advertisements. auth_type and auth_pass ensure only trusted nodes participate.
The core of this setup is VRRP (Virtual Router Redundancy Protocol). keepalived uses VRRP to elect a master node. The master node claims the VIP, meaning it binds the IP address to its network interface. All other nodes in the VRRP group are in backup mode. They listen for VRRP advertisements from the master. If the master stops sending these advertisements (e.g., it crashes or the network link fails), the backup nodes detect this absence and one of them (the one with the highest priority) will transition to master mode and take over the VIP.
The vrrp_script section is crucial for making this HA aware of the haproxy process itself. The check_haproxy.sh script (which you’d write to verify haproxy is running and healthy) is executed periodically. If the script fails, keepalived reduces the priority of that node. If the priority drops low enough (due to weight -20 and fall 2 failures), the backup node might even preempt the current master, causing a failover.
The virtual_ipaddress directive is where you specify the actual VIP and the network interface it should be assigned to (eth0).
When haproxy-1 is master, you’d see 192.168.1.100 assigned to its eth0:
ip addr show eth0 | grep 192.168.1.100
inet 192.168.1.100/24 brd 192.168.1.255 scope global eth0
If haproxy-1 fails, haproxy-2 (with its higher priority) will eventually become master and acquire the VIP. You’d then see the same IP on haproxy-2’s eth0.
It’s not just about IP address assignment. keepalived also uses multicast or unicast to send "heartbeat" packets. If a node doesn’t receive these heartbeats within a certain timeout, it assumes the other node is down and initiates a failover. The advert_int defines how often these heartbeats are sent.
When a failover occurs, the new master node not only claims the VIP but also needs to inform the rest of the network that the MAC address associated with that VIP has changed. It does this by sending gratuitous ARP requests. This is critical for network switches to update their ARP tables quickly, ensuring traffic continues to flow to the new master with minimal interruption.
You can monitor keepalived’s status with systemctl status keepalived and check the VRRP state with sudo ip vrrp show.
Most people focus on the IP address failover. What many overlook is the importance of the track_script directive. Without a script that actively checks the health of the application keepalived is supposed to be protecting (in this case, haproxy), you can have a situation where keepalived thinks the master is fine, but haproxy has actually crashed. The VIP would remain with the keepalived master, but no traffic would be processed. The track_script ensures that the health of the service dictates the mastership, not just the health of the keepalived process itself.
The next common issue you’ll encounter is dealing with split-brain scenarios or flapping between master and backup states due to network instability or misconfigured priorities.