High Availability Virtual IP with Keepalived and VRRP on Linux: A 6-Month Production Review

Table of Contents

The 2:00 AM Outage: What Happens When Gateways Die

Six months ago, our primary API gateway suffered a catastrophic power supply failure right in the middle of a flash sale pushing 14,000 requests per second. We had four healthy application nodes sitting behind our reverse proxy. It didn’t matter. The entire platform went dark for 42 minutes because all client traffic resolved to a single physical IP address on that dead box.

Incoming traffic hit a brick wall. PagerDuty woke up half the engineering team, our on-call engineers rushed to edit DNS A-records, and we watched TTL propagation delays burn through our error budget. The incident made an uncomfortable reality clear: backend redundancy is completely useless if your ingress gateway is a single point of failure.

Why Common Failover Tricks Don’t Cut It

Before jumping into a solution, we audited the standard failover workarounds to see where they broke down in our pipeline:

DNS Round Robin and GeoDNS: DNS-based failover sounds simple on paper. In practice, public resolvers, mobile carriers, and corporate proxies ignore your 30-second TTL. They cache stale records for hours, stranding active users on dead IPs.
Static IP Reassignment via Cloud APIs: Rebinding a secondary IP using vendor CLI tools or API webhooks takes between 25 and 90 seconds. That delay is long enough to drop thousands of active TCP connections and spike HTTP 502/504 errors.
Basic Ping Probes: ICMP pings only tell you if the Linux kernel is alive. They won’t catch a dead HAProxy or Nginx process when the underlying host stays responsive.

We needed an automated Layer 3 and Layer 4 mechanism. It had to share a single floating Virtual IP (VIP) across servers, broadcast heartbeat packets, and trigger failover in under a second without touching client DNS.

Comparing Failover Stacks

We benchmarked three common approaches across our Linux bare-metal and private VPC environments:

Solution	Failover Speed	Complexity	Best Fit
Pacemaker + Corosync	1–3 seconds	High (Requires STONITH/fencing)	Stateful services, active-passive database pairs
BGP Anycast (FRR)	<500 ms	Very High (Requires BGP peering & router control)	Multi-datacenter edges, ISP transit networks
Keepalived (VRRP)	<1 second (~800 ms)	Low to Moderate	Stateless gateways, HAProxy/Nginx ingress tiers

Pacemaker is solid for PostgreSQL clusters where split-brain writes corrupt disk state, but it is overkill for stateless web ingress. BGP Anycast is great, but it requires coordinating with network teams and running routing daemons on every node. Keepalived with VRRP (Virtual Router Redundancy Protocol) hit the sweet spot: near-instant recovery, tiny CPU overhead, and a single configuration file.

Step-by-Step: Deploying Keepalived with Active Health Checks

Over the past six months, this configuration handled kernel updates, scheduled node reboots, and an unscheduled top-of-rack switch failure without losing a single client request.

1. Network Baseline

Our staging cluster uses two Ubuntu 24.04 nodes acting as HAProxy load balancers:

Primary Node (Node-01): 192.168.10.11 (Interface: eth0)
Backup Node (Node-02): 192.168.10.12 (Interface: eth0)
Shared VIP: 192.168.10.100

2. Permitting Non-Local Socket Binding

When Node-02 boots up in backup mode, the VIP is not yet assigned to its local interface. By default, Linux rejects attempts to bind services to addresses the host does not currently own. We need to flip the ip_nonlocal_bind kernel flag so HAProxy can bind to 192.168.10.100:443 on both boxes simultaneously:

# Apply immediately
sudo sysctl -w net.ipv4.ip_nonlocal_bind=1

# Persist across reboots
echo "net.ipv4.ip_nonlocal_bind = 1" | sudo tee /etc/sysctl.d/99-keepalived.conf
sudo sysctl --system

3. Package Installation

Install Keepalived across both machines:

# Debian / Ubuntu
sudo apt update && sudo apt install -y keepalived

# RHEL / Rocky / AlmaLinux
sudo dnf install -y keepalived

4. Configuring the Master Node

Write this configuration to /etc/keepalived/keepalived.conf on Node-01. We configure a health-check script that pings the HAProxy PID directly. If the daemon stops responding, Keepalived drops Node-01’s priority below Node-02, triggering a clean handover:

global_defs {
    router_id lb-node-01
    enable_script_security
    script_user root
}

# Check if HAProxy process is running
vrrp_script check_proxy {
    script "/usr/bin/killall -0 haproxy"
    interval 2     # Run check every 2 seconds
    weight -20     # Drop priority by 20 if check fails
    fall 2         # Require 2 consecutive failures
    rise 2         # Require 2 consecutive passes to recover
}

vrrp_instance VI_GATEWAY {
    state MASTER
    interface eth0
    virtual_router_id 51
    priority 101   # Higher initial weight keeps Node-01 as primary
    advert_int 1

    authentication {
        auth_type PASS
        auth_pass VrrpP@ss  # Keep to 8 characters max per RFC2338
    }

    virtual_ipaddress {
        192.168.10.100/24 dev eth0 label eth0:vip
    }

    track_script {
        check_proxy
    }
}

5. Configuring the Backup Node

On Node-02, place the matching configuration with a baseline priority of 100 and state set to BACKUP:

global_defs {
    router_id lb-node-02
    enable_script_security
    script_user root
}

vrrp_script check_proxy {
    script "/usr/bin/killall -0 haproxy"
    interval 2
    weight -20
    fall 2
    rise 2
}

vrrp_instance VI_GATEWAY {
    state BACKUP
    interface eth0
    virtual_router_id 51
    priority 100   # Lower than Node-01's 101
    advert_int 1

    authentication {
        auth_type PASS
        auth_pass VrrpP@ss
    }

    virtual_ipaddress {
        192.168.10.100/24 dev eth0 label eth0:vip
    }

    track_script {
        check_proxy
    }
}

6. Starting Services and Checking State

Start Keepalived on both hosts:

sudo systemctl enable --now keepalived

Run ip addr show eth0 on Node-01. The virtual IP alias will appear immediately:

inet 192.168.10.11/24 brd 192.168.10.255 scope global eth0
inet 192.168.10.100/24 scope global secondary eth0:vip

Run the same command on Node-02. The alias should not appear. It stays silent until a failover event occurs.

7. Stress Testing Failover Behavior

Never assume a cluster works until you break it intentionally. We executed two real-world failure scenarios during testing:

Service Crash Simulation: We stopped HAProxy on Node-01 (sudo systemctl stop haproxy). After 4 seconds (two failed checks at 2-second intervals), Node-01’s effective priority fell from 101 to 81. Node-02 took over as Master and immediately broadcast a Gratuitous ARP (GARP) frame. Our continuous HTTP synthetic benchmark recorded exactly zero dropped connections and a single 12 ms latency spike.
Hard Hardware Failure: We abruptly pulled the power cable from Node-01. Node-02 missed three consecutive advertisements (3 seconds) and assumed control of 192.168.10.100 automatically.

Hard Lessons from 6 Months in Production

Deploying Keepalived taught us a few edge cases that documentation rarely emphasizes:

Public Cloud Environments Block Multicast: Standard VRRP uses multicast address 224.0.0.18. AWS VPC, GCP, and Azure drop Layer 2 multicast frames entirely. If you deploy inside a cloud VPC, add unicast_src_ip and unicast_peer { <peer-ip> } blocks inside your vrrp_instance.
Explicitly Permit Protocol 112: If local firewalls (UFW or firewalld) drop IP protocol 112, both nodes will stop hearing each other’s advertisements. Both will promote themselves to Master simultaneously (split-brain) and fight over ARP responses. Add an explicit rule: sudo ufw allow proto 112 from 192.168.10.0/24.
Keep Passwords Under 8 Characters: The VRRP standard (RFC 2338) truncates cleartext passwords to 8 bytes. Extra characters are silently discarded on some distributions, causing confusing auth mismatch errors during manual failover drills.