The Silent Connection Killer: ICMP Black Holes
I once spent an entire shift troubleshooting a branch office that could access the company’s internal chat but couldn’t open the document portal. The screen just sat there, spinning. It wasn’t a firewall block or a DNS failure. It was a classic ICMP Black Hole caused by a breakdown in Path MTU Discovery (PMTUD).
When we wrap traffic inside a tunnel—like GRE, IPsec, or VXLAN—we add extra headers to every packet. A standard Ethernet frame has an MTU (Maximum Transmission Unit) of 1500 bytes.
If an IPsec tunnel adds 60 bytes of overhead, the original packet cannot exceed 1440 bytes. When a router receives a packet too large for the tunnel and the “Don’t Fragment” (DF) bit is set, it must drop it. Normally, the router sends back an ICMP “Destination Unreachable – Fragmentation Needed” message to tell the sender to shrink its packets.
An ICMP Black Hole happens when a security policy or a misconfigured firewall drops that specific ICMP message. The sending server never hears the bad news. It keeps retransmitting the oversized packet until the connection eventually times out. This explains why 64-byte pings work perfectly, while a 1400-byte web response fails every time.
Setting Up Your Diagnostic Toolkit
To fix the issue, you need tools that reveal what is happening at the packet level. Most Linux distributions include these, but you should ensure they are updated on your diagnostic machine.
On Ubuntu or Debian systems, install iputils-ping, tcpdump, and mtr. These are essential for tracing where packets vanish.
# Update and install networking tools
sudo apt update
sudo apt install iputils-ping tcpdump mtr-tiny -y
For RHEL, CentOS, or Fedora users, run:
sudo dnf install iputils tcpdump mtr -y
I have used these tools in production environments to solve connectivity issues for hundreds of remote sites. Having them ready allows you to stop guessing and start measuring.
Configuration: Fixing the MTU and MSS Mismatch
You can solve this in two ways. You can manually lower the MTU on the interface, or you can use TCP MSS Clamping, which is the more robust choice for modern networks.
Method 1: Manual MTU Adjustment
If you manage the end-user devices or the specific tunnel interface, you can lower the MTU directly. For a standard GRE tunnel, setting the MTU to 1476 (1500 bytes minus the 24-byte GRE header) usually resolves the conflict.
# Lower MTU on a specific interface
sudo ip link set dev eth0 mtu 1400
Manual changes are difficult to scale. If you have 500 laptops, you don’t want to touch each one. This is why most engineers prefer MSS Clamping.
Method 2: TCP MSS Clamping (The Professional Fix)
The TCP Maximum Segment Size (MSS) determines the largest amount of data a device accepts in a single segment. During the initial SYN/ACK handshake, both sides agree on this value. By “clamping” the MSS at the router, we force both endpoints to use smaller packets from the very first byte.
On a Linux gateway using iptables, apply this rule to automatically adjust the MSS for all traffic passing through the tunnel:
# Clamp MSS to PMTU automatically
sudo iptables -t mangle -A FORWARD -p tcp --tcp-flags SYN,RST SYN \
-j TCPMSS --clamp-mss-to-pmtu
For IPsec tunnels, which have unpredictable overhead, I recommend a hard limit of 1360 bytes. This provides a safety margin for various encryption headers.
# Set a specific MSS value for IPsec safety
sudo iptables -t mangle -A FORWARD -p tcp --tcp-flags SYN,RST SYN \
-j TCPMSS --set-mss 1360
If you are using nftables, use this syntax:
# nftables equivalent for MSS clamping
nft add rule ip filter forward tcp flags syn tcp option maxseg size set 1360
Verification: Proving the Fix Works
After applying the configuration, verify that large packets actually pass through the path. The most effective method is a “Sweep Ping” with the Don’t Fragment bit enabled.
Testing with Ping
Run a ping test that forbids fragmentation. Start with a size of 1472 bytes (which becomes 1500 with headers) and work your way down.
# -M do: set the Don't Fragment bit
# -s: packet data size
ping -M do -s 1472 1.1.1.1
If you see the error “Frag needed and DF set,” your MTU is still too high. Reduce the -s value until the pings succeed. If the request simply times out, the ICMP error message is still being blocked somewhere upstream.
Monitoring with tcpdump
Use tcpdump to watch the TCP handshake in real-time. You want to confirm that your gateway is successfully rewriting the MSS value in the SYN packets.
# Capture SYN packets to check MSS values
sudo tcpdump -i any -n "tcp[tcpflags] & (tcp-syn) != 0"
Check the options [mss XXXX] field in the output. If your rule is working, you will see 1360 (or your chosen value) instead of the default 1460.
Final Thoughts
To ensure long-term stability, check your firewall’s ICMP policy. You should always allow ICMP Type 3, Code 4. Without it, PMTUD is broken by design. By combining MSS clamping with a conservative limit of 1350-1380 bytes, you can eliminate the mysterious connection drops that often plague VPN and SD-WAN environments.
