The Problem: Paying for Bandwidth You Can’t Use
Managing a high-traffic server node with two ISPs often feels like owning a V8 engine but only using four cylinders. By default, Linux relies on a single default gateway. All your outgoing traffic squeezes through one pipe while the second link—which you’re paying for—sits idle. If that primary link goes down, your connection drops entirely until you manually intervene.
Standard failover scripts fix the redundancy problem, but they don’t solve the underutilization issue. Equal-Cost Multi-Path (ECMP) routing changes this. ECMP allows the Linux kernel to spread outgoing traffic across several paths at once. It’s a powerful way to aggregate a 1Gbps fiber line with a 500Mbps backup link to maximize your total available pipe.
I’ve deployed this setup on edge routers and VPN concentrators where every megabit counts. It bridges the gap between basic administration and professional-grade network engineering without requiring expensive hardware load balancers.
How ECMP Works in the Linux Kernel
ECMP functions by adding multiple “next hops” to a single destination in the routing table, typically the 0.0.0.0/0 route. When a packet needs to leave the system, the kernel chooses a path based on a hashing algorithm.
Flow-Based vs. Packet-Based Hashing
A common fear is that ECMP will split packets like a round-robin. If Linux sent every other packet to a different ISP, TCP performance would collapse due to out-of-order delivery. Instead, modern kernels use flow-based hashing. By analyzing the source IP, destination IP, and ports, the kernel ensures that all packets for a specific “flow” stay on the same path. This preserves packet ordering while distributing different user sessions across all available gateways.
Pro Tip: By default, Linux often hashes only based on Layer 3 (IPs). To get better distribution using Layer 4 (ports), enable this sysctl setting:
sudo sysctl -w net.ipv4.fib_multipath_hash_policy=1Hands-on: Implementing ECMP with iproute2
Let’s look at a practical scenario. Suppose your server has two network interfaces:
- eth1: ISP A (Gateway: 192.168.1.1)
- eth2: ISP B (Gateway: 192.168.2.1)
Step 1: Clear the Path
Existing static default routes will conflict with your new multipath setup. Check your current table first:
ip route showIf a default route exists, wipe the slate clean:
sudo ip route del defaultStep 2: Add the Multipath Route
The nexthop keyword is the secret sauce here. Use the following command to split traffic 50/50 between your two providers:
sudo ip route add default \
nexthop via 192.168.1.1 dev eth1 weight 1 \
nexthop via 192.168.2.1 dev eth2 weight 1The weight parameter is highly flexible. If ISP A is a 1Gbps dedicated line and ISP B is a 500Mbps Starlink, set ISP A to weight 2 and ISP B to weight 1. This tells the kernel to send roughly 66% of new flows to the faster link.
Step 3: Verify the Traffic Split
Run ip route show again. You should see a multipath entry listed under the default route. To see the balancing in action, run a traceroute to two different public IPs:
traceroute 8.8.8.8
traceroute 1.1.1.1If the hashing works, you will see the first hop switch between your two gateway IPs.
The “Dead Gateway” Trap
Standard iproute2 commands are static. The kernel knows if a cable is unplugged (link DOWN), but it doesn’t know if the ISP’s upstream network has failed while your local link stays UP. If ISP A goes dark but the interface remains active, the kernel will keep throwing 50% of your traffic into a black hole.
Production environments need a health-check mechanism. You have two main options:
- Monitoring Scripts: A simple Bash script can ping an external target (like 1.1.1.1) through a specific interface. If it fails, the script triggers a route update.
- Dynamic Routing: For mission-critical setups, use
FRR(Free Range Routing). It handles BFD (Bidirectional Forwarding Detection), which can detect a dead link in milliseconds and update the routing table automatically.
Making it Permanent
Manual ip route commands vanish after a reboot. For Ubuntu or Debian systems using Netplan, you can define these routes in your /etc/netplan/*.yaml file. However, Netplan’s multipath syntax is often finicky. Many administrators prefer using a post-up script or a simple systemd service to re-apply the nexthop command once the network stack is initialized.
Key Considerations for Success
- Source IP Selection: If the server itself starts a connection, it needs to know which source IP to use. Ensure your NAT (Masquerade) rules in
iptablesornftablesare configured to handle both outgoing interfaces. - MTU Matching: Keep your MTU consistent. If your fiber link uses 1500 bytes but your VPN-based backup uses 1400, you will encounter broken page loads and stalled SSH sessions.
- Conntrack: Ensure the
nf_conntrackkernel module is active. This allows the firewall to remember which path a connection started on so it can handle return traffic correctly.
Conclusion
ECMP is the most cost-effective way to squeeze every bit of performance out of a multi-homed Linux server. It turns a passive backup link into an active asset, increasing your aggregate bandwidth instantly. While you’ll need to add a monitoring layer for true high availability, the core routing logic is built right into the kernel, ready for you to use.
