Posted inNetworking

Deploying BGP Anycast on Linux with BIRD: Resilient DNS and Web Load Balancing

Networking tutorial - IT technology blog
Networking tutorial - IT technology blog

Approach Comparison: DNS Round-Robin vs VRRP vs BGP Anycast

At 2:14 AM, the primary load balancer in rack A locked up hard. Traffic vanished into a black hole. Because public DNS resolvers cached the record for 300 seconds, over 40,000 requests dropped before traffic shifted to the secondary node. In production, your failover mechanism determines whether you sleep through a glitch or spend your morning writing an incident post-mortem.

When scaling critical services like authoritative DNS (UDP) or edge reverse proxies (TCP), infrastructure teams generally choose between three models:

  • DNS Round-Robin / GeoDNS: Simple to configure. You map one hostname to multiple A/AAAA records. However, intermediate DNS resolvers aggressively cache records and routinely ignore your 60-second TTL. If a server dies, clients continue hammering that dead IP for minutes or hours.
  • Virtual Router Redundancy Protocol (VRRP / Keepalived): Reliable for local high availability. Nodes share a Virtual IP (VIP) using heartbeat multicasts every second. While VRRP shifts IPs within 1 to 3 seconds, it remains trapped in a single Layer 2 broadcast domain. If the Top-of-Rack (ToR) switch loses power, the VIP dies with it.
  • BGP Anycast: Multiple independent servers announce the exact same IP address (a /32 IPv4 or /128 IPv6 prefix) to upstream routers via BGP. Routers use standard IP routing metrics to steer packets to the closest healthy node. When a node fails, the route withdraws, and the network fabric reroutes incoming packets within milliseconds.
Metric DNS Round-Robin VRRP (Keepalived) BGP Anycast
Failover Speed Minutes to hours (TTL-bound) 1 to 3 seconds Sub-second (with BFD) to 6s
Blast Radius Global outage for cached clients Single rack or L2 subnet Isolated to local network path
Traffic Distribution Coarse client-side random Active/Passive pair Line-rate ECMP across all nodes
DDoS Absorption Hits all targets evenly Saturates single active link Fragmented geographically at edge

Pros and Cons of BGP Anycast

BGP Anycast simplifies traffic distribution, but shifting Layer 4 balancing down to Layer 3 introduces specific operational constraints.

Advantages

  • Zero-Proxy Throughput: Upstream switches distribute traffic using Equal-Cost Multi-Path (ECMP). Packets travel directly from switch ASICs to your server network cards at 40 Gbps or 100 Gbps line rate without a software proxy middleman.
  • Automated Blast-Radius Isolation: Volumetric DDoS attacks get routed to the node topologically closest to the compromised botnet devices. A 50 Gbps flood in Frankfurt stays in Europe instead of overwhelming your Ashburn cluster.
  • Stateless Resilience: Routers drop failed nodes from their routing table as soon as the BGP session closes or a route withdrawal packet arrives.

Disadvantages and Trade-offs

  • TCP Session Resets on Route Shifts: Switches calculate ECMP paths using a 5-tuple packet hash (source IP, destination IP, source port, destination port, protocol). If one server in an ECMP group drops, the switch recalculates hashes for all surviving nodes. Long-lived TCP connections—such as a 400 MB file upload or a WebSocket stream—get routed to new servers and drop immediately with a TCP RST.
  • Route Flapping Penalties: If a buggy health check toggles BGP advertisements every few seconds, transit providers will apply BGP route dampening and suppress your prefix for up to 30 minutes.
  • BGP Peering Requirements: Your data center or hosting provider must grant you private peering sessions with their Top-of-Rack switches over eBGP or iBGP.

Recommended Architecture for BIRD on Linux

Once traffic exceeds what a pair of hardware appliances can handle, running BGP directly on Linux hosts is the industry standard approach. The cleanest setup runs BIRD 2 (Border Internet Routing Daemon) on bare metal or VM instances, cross-peered with dual Top-of-Rack switches.

Topology Overview

  • Anycast VIP: 192.0.2.53/32 bound to a dedicated dummy0 interface on every node.
  • Physical Subnet: 10.10.10.0/24 on eth0 for node management and BGP peering.
  • Node 1: IP 10.10.10.11 (AS 65001) peering with ToR Switch 1 (10.10.10.1, AS 65000).
  • Node 2: IP 10.10.10.12 (AS 65001) peering with ToR Switch 2 (10.10.10.2, AS 65000).
  • Failure Watchdog: A local agent verifies process responsiveness (DNS queries to port 53 or HTTP status 200 on port 80). If the daemon crashes, the script drops the dummy0 address. BIRD detects this via Linux netlink instantly and issues a BGP WITHDRAW.

For TCP traffic (like Nginx or Envoy), set the kernel sysctl net.ipv4.ip_nonlocal_bind = 1. This allows your web server to bind to 192.0.2.53:443 even when the dummy interface is down during initial boot.

Implementation Guide: Step-by-Step BIRD Setup on Linux

Step 1: Configure the Dummy Anycast Interface

Never bind the Anycast IP directly to your physical interface. A dummy interface decouples the service VIP from physical link-state transitions.

# Load the dummy kernel module
sudo modprobe dummy

# Create the persistent dummy interface
sudo ip link add dev dummy0 type dummy
sudo ip link set dev dummy0 up

# Assign your Anycast IP as a /32 host address
sudo ip addr add 192.0.2.53/32 dev dummy0

To persist this configuration across system reboots, create /etc/systemd/network/10-dummy0.netdev:

[NetDev]
Name=dummy0
Kind=dummy

Next, define the network configuration in /etc/systemd/network/10-dummy0.network:

[Match]
Name=dummy0

[Network]
Address=192.0.2.53/32

Enable the systemd-networkd daemon to activate the interface:

sudo systemctl enable --now systemd-networkd

Step 2: Enable Kernel Packet Forwarding and ARP Filtering

When multiple servers share the same IP inside a data center subnet, they will attempt to respond to ARP requests for that IP. Prevent duplicate ARP collisions by tightening kernel sysctl flags:

sudo tee /etc/sysctl.d/99-anycast.conf << 'EOF'
# Enable packet forwarding
net.ipv4.ip_forward = 1

# Allow services to bind to IPs not yet assigned to an interface
net.ipv4.ip_nonlocal_bind = 1

# Prevent the server from answering ARP queries for dummy0 on physical eth0
net.ipv4.conf.all.arp_ignore = 1
net.ipv4.conf.all.arp_announce = 2
EOF

sudo sysctl --system

Step 3: Install and Configure BIRD 2

Install the BIRD 2 package:

sudo apt-get update && sudo apt-get install -y bird2

Replace /etc/bird/bird.conf with the following production configuration. BIRD inspects dummy0 and announces the 192.0.2.53/32 prefix to the ToR switch:

# /etc/bird/bird.conf
log syslog all;
router id 10.10.10.11;

define ANYCAST_IP = 192.0.2.53/32;

protocol device {
    scan time 5;
}

# Read routes and IPs configured on dummy0 via netlink
protocol direct direct_anycast {
    interface "dummy0";
}

# Filter: Only advertise the Anycast /32 prefix
filter anycast_export {
    if net = ANYCAST_IP then accept;
    reject;
}

# Peering session to Top-of-Rack Switch 1
protocol bgp tor_switch1 {
    local 10.10.10.11 as 65001;
    neighbor 10.10.10.1 as 65000;
    
    ipv4 {
        import none;
        export filter anycast_export;
    };
    
    # Hold time 6s, keepalive 2s for fast 6-second failure detection
    hold time 6;
    keepalive time 2;
}

Test the configuration syntax and restart the service:

sudo bird -p
sudo systemctl restart bird
sudo systemctl enable bird

Step 4: Automate Route Withdrawal with Health Checks

Announcing a BGP route for a dead service creates a catastrophic black hole. A local health check must remove the IP address from dummy0 when checks fail, triggering BIRD to withdraw the prefix.

Create an executable script at /usr/local/bin/anycast-healthcheck.sh:

#!/usr/bin/env bash
set -euo pipefail

ANYCAST_IP="192.0.2.53/32"
INTERFACE="dummy0"
CHECK_HOST="127.0.0.1"

# 1. Verify DNS answers locally within 500ms
check_dns() {
    dig +time=1 +tries=1 @"$CHECK_HOST" health.check.local A > /dev/null 2>&1
}

# 2. Verify Nginx / Envoy healthz endpoint returns HTTP 200 within 1s
check_http() {
    curl -s -f -o /dev/null --connect-timeout 1 "http://$CHECK_HOST/healthz"
}

if check_dns && check_http; then
    # Service healthy: ensure IP is bound to dummy0
    if ! ip addr show dev "$INTERFACE" | grep -q "$ANYCAST_IP"; then
        ip addr add "$ANYCAST_IP" dev "$INTERFACE"
        logger -t anycast-healthcheck "Services healthy. Added $ANYCAST_IP to $INTERFACE."
    fi
else
    # Service degraded: remove IP to trigger BGP route withdrawal
    if ip addr show dev "$INTERFACE" | grep -q "$ANYCAST_IP"; then
        ip addr del "$ANYCAST_IP" dev "$INTERFACE"
        logger -t anycast-healthcheck "Check failed! Removed $ANYCAST_IP to withdraw BGP route."
    fi
fi

Make the script executable:

sudo chmod +x /usr/local/bin/anycast-healthcheck.sh

Execute this check every 2 seconds using a systemd timer or an orchestration loop. For production fleets, consider dedicated tools like anycast-healthchecker or exabgp with built-in health monitoring.

Step 5: Verifying Peering and Simulating Node Failure

Use the BIRD interactive client (birdc) to inspect session state:

# Check BGP session state
sudo birdc show protocols

# Inspect exported routes
sudo birdc show route export tor_switch1

The console output confirms that the session is Established and that 192.0.2.53/32 is being advertised:

Name       Proto      Table      State  Since         Info
device1    Device     master4    up     02:30:12
direct_anycast Direct master4    up     02:30:12
tor_switch1 BGP       master4    up     02:30:15      Established

192.0.2.53/32        unicast [direct_anycast 02:30:12] * (240)

Now test failure behavior. Stop your local DNS service:

sudo systemctl stop named
sudo /usr/local/bin/anycast-healthcheck.sh

As soon as the script drops 192.0.2.53/32 from dummy0, the Linux kernel emits a netlink address deletion event. BIRD catches this event immediately and fires a BGP WITHDRAW frame to the ToR switch. Within 50 milliseconds, the switch stops sending packets to Node 1 and redistributes traffic to Node 2. No DNS TTL delays. No manual intervention.

Share:
Tags: