Solving the Latency Puzzle with Anycast
High-traffic services like DNS or globally distributed apps eventually hit a hard physical limit: the speed of light. No matter how much you optimize your database, a user in Singapore will experience at least 200–250ms of latency when connecting to a server in New York. Standard Unicast routing sends traffic to one specific destination. Load balancers help, but they are often confined to a single data center or region.
Anycast rewrites the rules of the road. By announcing the same IP address from multiple geographic locations via BGP (Border Gateway Protocol), the internet’s core infrastructure automatically routes users to the “closest” healthy node.
I’ve deployed this architecture in production to handle regional traffic spikes that previously overwhelmed our single-region entry points. When a node fails, BGP withdraws the route in seconds, and traffic shifts seamlessly to the next best site. It is self-healing by design.
We will set up an Anycast IP on Linux using FRRouting (FRR). In this scenario, two servers will share one IP to provide a high-availability service that reacts faster than any traditional DNS-based failover.
The Architecture of Proximity
Before jumping into the terminal, let’s look at the blueprint. In a standard setup, every server has a unique IP. With Anycast, we assign a shared “Anycast IP” to a dummy interface on every node. Each server then tells the upstream router that it is the legitimate owner of that IP.
Why choose this over a Global Server Load Balancer (GSLB)?
- Sub-second failover: BGP convergence often beats DNS TTL updates, which can take minutes to propagate globally.
- One IP to rule them all: Your clients only need to whitelist or configure a single destination.
- Natural DDoS absorption: Attack traffic is distributed across your entire global footprint instead of concentrating on one pipe.
Anycast is the gold standard for UDP-based services like DNS. It also works for TCP (HTTP/HTTPS), provided your BGP routing is stable. If routes “flap” frequently, TCP sessions will break because packets from the same session might end up on different servers.
Installation: Setting up FRRouting
For BGP on Linux, FRRouting (FRR) is the industry standard. It’s a high-performance fork of Quagga with a CLI that will feel familiar to anyone who has used Cisco or Juniper gear. This example uses Ubuntu 22.04, but the logic applies to any modern distribution.
Install the FRR repository and package:
# Add the FRR GPG key and repository
curl -s https://deb.frrouting.org/frr/keys.asc | sudo apt-key add -
FRRVER="frr-stable"
echo deb https://deb.frrouting.org/frr $(lsb_release -s -c) $FRRVER | sudo tee /etc/apt/sources.list.d/frr.list
# Update and install
sudo apt update
sudo apt install frr frr-pythontoolsFRR runs a modular daemon system. We need to flip the switch for BGP manually.
# Enable BGP daemon
sudo sed -i 's/bgpd=no/bgpd=yes/' /etc/frr/daemons
sudo systemctl restart frrConfiguration: Making BGP Work for You
The configuration spans two layers: the Linux network stack and the FRR routing engine.
1. The Dummy Interface
We need a virtual interface to hold our Anycast IP. We avoid using the physical interface (like eth0) because we want the ability to “hide” the Anycast IP if the local service fails, without dropping the server’s management connection.
# Create a dummy interface named 'anycast0'
sudo ip link add anycast0 type dummy
sudo ip link set anycast0 up
# Assign your Anycast IP (e.g., 192.0.2.100)
sudo ip addr add 192.0.2.100/32 dev anycast02. BGP Peering via vtysh
Now we tell FRR to advertise this IP to the world. We use vtysh, the integrated shell. You’ll need your Autonomous System Number (ASN) and the IP of your upstream peer.
sudo vtysh
configure terminal
# Replace 65001 with your ASN
router bgp 65001
bgp router-id 1.1.1.1
# Your upstream neighbor (e.g., your ISP or ToR switch)
neighbor 203.0.113.1 remote-as 65000
neighbor 203.0.113.1 description Upstream_Transit
address-family ipv4 unicast
# Advertise the specific Anycast IP
network 192.0.2.100/32
neighbor 203.0.113.1 activate
exit-address-family
exit
write memory3. Avoiding the “Black Hole”
Here is a lesson I learned the hard way: if your Nginx or BIND service crashes but the server stays up, BGP will keep telling the world to send traffic there. This creates a black hole. You must link the BGP announcement to service health.
A simple bash script can monitor your application and pull down the anycast0 interface if curl -s localhost fails. Once the interface is down, FRR stops announcing the route, and the internet automatically redirects traffic to your other nodes within seconds.
Verification: Proving it Works
Check your route propagation from inside vtysh:
show ip bgp summary
show ip bgp 192.0.2.100/32The status code *> is what you want to see—it confirms a valid, best-path route. But don’t just trust the local output. To verify true Anycast behavior, use a tool like RIPE Atlas or mtr from different global VPS locations.
# Run this from a remote client
mtr -rw 192.0.2.100If you see the final hop reaching Server A from London and Server B from Tokyo, your setup is live. Monitoring these BGP sessions via Prometheus with the frr_exporter is non-negotiable for production stability. Anycast was once a tool for giants, but with modern Linux, any team with an ASN can build a global, resilient network.
