The 2 AM Networking Crisis: Why Tunnels Matter
It’s 2 AM, and the monitoring alerts are screaming. Your database cluster nodes in the secondary data center just lost synchronization because the provider doesn’t support Layer 2 adjacency across sites. You need these servers to act like they’re on the same local switch, but they’re separated by three routers and a patchy public internet connection. This is where tunneling saves the day.
Stretching a network across existing Layer 3 infrastructure is a common requirement for Kubernetes, OpenStack, and multi-site disaster recovery. I’ve deployed these tunnels in production environments handling over 10Gbps of traffic; the stability is rock-solid if you get the configuration right. Whether you’re linking two remote offices or building a massive virtualized overlay, mastering GRE and VXLAN is a core skill for any senior Linux engineer.
Think of GRE (Generic Routing Encapsulation) as the reliable old-school workhorse for point-to-point Layer 3 links. VXLAN (Virtual Extensible LAN) is its modern cousin. It wraps Layer 2 Ethernet frames inside UDP packets, essentially turning your disparate IP network into one giant virtual switch.
Preparing the Linux Environment
Don’t jump straight into the config. First, make sure your kernel is actually ready to handle encapsulation. While modern distros like Ubuntu 22.04 or RHEL 9 usually come with these modules enabled, you don’t want to find out one is missing during an emergency. You’ll need the iproute2 suite—if you’re still using ifconfig, it’s time to let go.
Verify your modules with these commands:
# Check for GRE support
sudo modprobe ip_gre
lsmod | grep gre
# Check for VXLAN support
sudo modprobe vxlan
lsmod | grep vxlanTo make these persistent after a reboot, add them to /etc/modules-load.d/tunnels.conf. It only takes ten seconds now but saves ten minutes of head-scratching later.
Deploying a GRE Tunnel for Point-to-Point Connectivity
Let’s look at a real-world scenario. Site A is at 1.1.1.1 and Site B is at 2.2.2.2. We need a private “cable” between them where Site A uses 10.0.0.1 and Site B uses 10.0.0.2 on a /30 subnet.
On Site A (1.1.1.1):
# Create the tunnel interface
sudo ip tunnel add gre01 mode gre remote 2.2.2.2 local 1.1.1.1 ttl 255
# Bring the interface up
sudo ip link set gre01 up
# Assign a private IP address
sudo ip addr add 10.0.0.1/30 dev gre01On Site B (2.2.2.2):
# Create the tunnel interface
sudo ip tunnel add gre01 mode gre remote 1.1.1.1 local 2.2.2.2 ttl 255
# Bring the interface up
sudo ip link set gre01 up
# Assign a private IP address
sudo ip addr add 10.0.0.2/30 dev gre01GRE is straightforward, but it has one big limitation: it only handles Layer 3 traffic. It won’t pass broadcast packets or non-IP frames natively. If you’re just running OSPF or BGP between routers, GRE is perfect. But if your application requires servers to “see” each other’s MAC addresses for Layer 2 adjacency, you’ll need VXLAN.
Scaling with VXLAN: Creating an L2 Overlay Network
VXLAN is the industry standard for overlay networks. It uses a 24-bit Virtual Network Identifier (VNI), allowing for 16 million isolated networks. This obliterates the 4,096 limit of traditional VLANs. We’ll bridge two remote servers so they appear on the same 192.168.100.x subnet.
On Host A (1.1.1.1):
# Create the VXLAN interface
# VNI (id) is 42, using standard port 4789
sudo ip link add vxlan42 type vxlan id 42 remote 2.2.2.2 local 1.1.1.1 dstport 4789
# Bring it up
sudo ip link set vxlan42 up
# Assign an IP on the virtual L2 network
sudo ip addr add 192.168.100.1/24 dev vxlan42On Host B (2.2.2.2):
sudo ip link add vxlan42 type vxlan id 42 remote 1.1.1.1 local 2.2.2.2 dstport 4789
sudo ip link set vxlan42 up
sudo ip addr add 192.168.100.2/24 dev vxlan42At this stage, Host A can ping 192.168.100.2. If you run arp -a, the remote host’s MAC address will appear. You have successfully stretched a Layer 2 network across a Layer 3 boundary.
The Silent Performance Killer: MTU Fragmentation
Here is the most common mistake: ignoring the MTU (Maximum Transmission Unit). A standard Ethernet frame is 1500 bytes. When you wrap that frame in a 50-byte VXLAN header, it grows to 1550 bytes. If your physical network expects 1500, that packet gets fragmented or simply dropped.
This is why a ping might work, but your SSH session hangs as soon as you run a large command. To keep things snappy, always lower the MTU on your tunnel interfaces:
# For GRE
sudo ip link set dev gre01 mtu 1476
# For VXLAN
sudo ip link set dev vxlan42 mtu 1450Lowering the MTU prevents the performance nightmare of fragmentation and ensures large data transfers don’t fail silently.
Verification & Monitoring
Verification isn’t just about a successful ping. You need to ensure the encapsulation is actually working without errors.
Start by checking the interface statistics:
ip -s link show vxlan42Watch for dropped packets. If traffic isn’t flowing, use tcpdump to inspect the physical wire. You should see UDP packets on port 4789 for VXLAN or protocol 47 for GRE.
# Monitor VXLAN traffic on the physical eth0 interface
sudo tcpdump -i eth0 port 4789 -nIf you see packets leaving Host A but never reaching Host B, check your firewall. Ensure 4789/UDP is open for VXLAN and the gre protocol is permitted in your security groups. Manual configuration is your foundation. Sure, in a massive 500-node cluster, you’d use Ansible or a BGP EVPN control plane. But knowing how to “wire” these interfaces by hand is what saves the day when the automation fails at 3 AM.
