High Availability Strategies: Keepalived vs. Pacemaker and DNS Failover
It was 2:15 AM on a Tuesday when PagerDuty rang. A kernel panic had frozen our primary Nginx proxy, dropping 1,200 active connections and sending incoming traffic into the void. At that hour, automated failover makes all the difference. It decides whether you resolve the incident in five seconds or spend the next two hours firefighting customer tickets. Production teams usually choose between three approaches:
- Keepalived (VRRP – Virtual Router Redundancy Protocol): Two or more nodes share a single floating Virtual IP (VIP). One node acts as MASTER and answers ARP requests. Meanwhile, the BACKUP node tracks VRRP multicast heartbeats every 1,000 ms. If those heartbeats stop or a custom health check fails, the backup claims the VIP instantly.
- Pacemaker + Corosync: A heavyweight clustering suite built for complex resource topologies, shared block devices (DRBD), and STONITH node fencing. It coordinates stateful, multi-service failovers across dozens of machines.
- DNS-Based Failover: External synthetic probes monitor endpoint health and rewrite DNS A/AAAA records through an API whenever an origin drops offline.
Pros and Cons: Picking the Right Failover Layer
Architecture decisions involve distinct trade-offs when milliseconds matter:
- Keepalived
- Pros: Extremely lightweight. It consumes less than 15MB of RAM and negligible CPU cycles. Failovers execute in 1 to 2 seconds at Layer 3/4 with zero client reconfiguration, and the configuration lives in a clean, human-readable file.
- Cons: Both nodes must share a single Layer 2 broadcast domain (VLAN or subnet), unless you configure explicit unicast peers. It also lacks native orchestration for multi-tier service dependencies.
- Pacemaker + Corosync
- Pros: Supports active-active clusters, strict startup sequences, shared storage locks, and hardware fencing to stop split-brain events dead in their tracks.
- Cons: Steep learning curve and notable memory overhead. A single misconfiguration in your fencing topology can trap your servers in an endless reboot cycle.
- DNS Failover
- Pros: Operates globally across disparate clouds, regions, and routed networks without subnet constraints.
- Cons: Crippled by client-side resolver caching. Even with a 30-second TTL, upstream ISP resolvers often ignore expiration limits. Traffic can keep hitting a dead IP for 10 to 15 minutes.
The Recommended Setup: Active-Passive Pair with Floating VIP
For reverse proxies and load balancers like HAProxy or Nginx, an active-passive Keepalived pair is the sweet spot. On a test cluster running two 2 vCPU, 4GB RAM Ubuntu 22.04 VMs, this setup handled failovers in 1.4 seconds flat. Crucially, it did so without the distributed state management headaches that come with heavier clustering stacks.
Our production deployment targets this baseline topology:
- Virtual IP (Floating VIP):
192.168.10.100 - Node 1 (Primary – MASTER):
192.168.10.11on interfaceeth0 - Node 2 (Secondary – BACKUP):
192.168.10.12on interfaceeth0 - Workload: Nginx listening on ports 80 and 443 across both nodes
Implementation Guide: Building the High Availability Cluster
Step 1: Allow Non-Local IP Binding
By default, the Linux kernel blocks processes from binding to sockets on IP addresses that are not yet assigned to an active interface. Since our floating VIP resides on only one server at any given moment, the secondary node cannot start Nginx during system boot without this override.
Run these commands on both Node 1 and Node 2:
echo "net.ipv4.ip_nonlocal_bind = 1" | sudo tee -a /etc/sysctl.d/99-keepalived.conf
sudo sysctl --systemStep 2: Install Keepalived
Grab the Keepalived package using your distribution’s native package manager:
# On Ubuntu / Debian
sudo apt update && sudo apt install -y keepalived
# On RHEL / AlmaLinux / Rocky Linux 9
sudo dnf install -y keepalivedStep 3: Create the Application Health Check Script
Never rely solely on network-level heartbeats. A host can still transmit VRRP packets while its underlying web service hangs in an unrecoverable worker deadlock. We need a health check that confirms Nginx responds to local requests with a valid status code within two seconds.
Deploy /usr/local/bin/check_service.sh on both nodes:
sudo tee /usr/local/bin/check_service.sh << 'EOF'
#!/usr/bin/env bash
set -euo pipefail
# Query the local HTTP endpoint with a strict 2-second timeout
if curl -sf --max-time 2 http://127.0.0.1:80/ > /dev/null; then
exit 0
else
exit 1
fi
EOF
sudo chmod 755 /usr/local/bin/check_service.shStep 4: Configure the Primary Node (MASTER)
On Node 1 (192.168.10.11), write the main daemon configuration. We set the initial priority to 101 and assign a negative weight of 30 to our health script. If two consecutive checks fail, Node 1’s effective priority drops from 101 to 71, dropping below Node 2’s baseline priority of 90.
sudo tee /etc/keepalived/keepalived.conf << 'EOF'
global_defs {
router_id lb-node-01
script_user root
enable_script_security
}
vrrp_script chk_web {
script "/usr/local/bin/check_service.sh"
interval 2
weight -30
fall 2
rise 2
}
vrrp_instance VI_PROD {
state MASTER
interface eth0
virtual_router_id 51
priority 101
advert_int 1
authentication {
auth_type PASS
auth_pass VrrpS3cur3Key
}
virtual_ipaddress {
192.168.10.100/24 dev eth0 label eth0:vip
}
track_script {
chk_web
}
}
EOFStep 5: Configure the Secondary Node (BACKUP)
Head to Node 2 (192.168.10.12). Notice the configuration uses state BACKUP and sets baseline priority to 90:
sudo tee /etc/keepalived/keepalived.conf << 'EOF'
global_defs {
router_id lb-node-02
script_user root
enable_script_security
}
vrrp_script chk_web {
script "/usr/local/bin/check_service.sh"
interval 2
weight -30
fall 2
rise 2
}
vrrp_instance VI_PROD {
state BACKUP
interface eth0
virtual_router_id 51
priority 90
advert_int 1
authentication {
auth_type PASS
auth_pass VrrpS3cur3Key
}
virtual_ipaddress {
192.168.10.100/24 dev eth0 label eth0:vip
}
track_script {
chk_web
}
}
EOFStep 6: Configure Firewalls and Start Services
Keepalived coordinates via VRRP (IP protocol 112), not standard TCP or UDP ports. If your local firewall drops this protocol, both nodes fail to see each other. Each node assumes the role of master and binds the same VIP, causing severe split-brain packet collisions.
Open the protocol and enable the service on both nodes:
# UFW (Ubuntu / Debian)
sudo ufw allow in on eth0 proto 112
# Firewalld (RHEL / AlmaLinux)
sudo firewall-cmd --add-protocol=vrrp --permanent
sudo firewall-cmd --reload
# Reload systemd and start the daemon
sudo systemctl daemon-reload
sudo systemctl enable --now keepalivedStep 7: Verify VIP Binding and Test Failover
Inspect the network configuration on Node 1:
ip addr show dev eth0The output should display 192.168.10.100/24 alongside your primary IP. Querying Node 2 should show only 192.168.10.12.
Now, test failover behavior by killing the web server on Node 1:
# On Node 1
sudo systemctl stop nginxWithin four seconds (two failed health checks spaced two seconds apart), check the logs on Node 2:
sudo journalctl -u keepalived -n 25 --no-pagerYou will see Keepalived transition Node 2 from BACKUP to MASTER, followed by an announcement sending gratuitous ARP frames to update your upstream switch tables. Continuous HTTP probes against 192.168.10.100 will show only one or two dropped packets before traffic resumes normally.
