The Hidden Lag: Understanding TCP Retransmissions
Your Linux server looks perfect on paper. CPU usage is idling at 5%, and you have gigabytes of free RAM, yet your application feels sluggish. Requests that usually resolve in 20ms are suddenly spiking to 200ms or even 2 seconds. When the database and code aren’t the problem, the bottleneck is likely hiding in the network stack as TCP Retransmissions.
TCP is designed to be reliable. When a sender transmits a packet, it waits for an acknowledgment (ACK). If that ACK doesn’t arrive before the Retransmission Timeout (RTO) hits, the sender assumes the packet vanished and tries again. While this prevents data loss, it kills performance. On a 10Gbps link, even a 1% retransmission rate can slash your effective throughput by 50% or more.
I have seen high-traffic clusters crawl because of a single faulty network switch port or a mismatched MTU. Often, teams waste days rewriting database queries when the fix actually required a simple one-line kernel tweak.
Setting Up Your Diagnostic Toolkit
To find out where packets are disappearing, you need a few core utilities. Most of these are already in your system, but a few might need a quick install.
1. iproute2 (ss and nstat)
The ss and nstat tools come bundled in the iproute2 package. This is standard on Ubuntu, CentOS, Debian, and Fedora. If you can run ip addr, you already have these.
2. Wireshark and TShark
Wireshark is great for desktop analysis, but tshark is the version you’ll use on remote servers via CLI. Install it using your package manager:
# Ubuntu / Debian
sudo apt update && sudo apt install tshark -y
# RHEL / CentOS / AlmaLinux
sudo yum install wireshark-cli -ySpotting the Problem with nstat and ss
Don’t start capturing massive packet files immediately. First, confirm the scale of the problem by checking the kernel’s internal counters.
Checking Global Stats with nstat
The nstat command pulls metrics directly from the kernel. To see how many segments have been retransmitted since the last boot, run:
nstat -az TcpRetransSegsIn a healthy network, this number should be low—ideally less than 0.05% of total segments. To see the current rate of retransmissions while you run a benchmark, use this command to refresh every second:
nstat -n 1 1Inspecting Individual Sockets with ss
The ss (socket statistics) tool is the modern replacement for netstat. It is significantly faster and reveals the internal state of a TCP connection. Use the -i flag to see these details.
ss -tiLook for the retrans field in the output. Here is an example of a problematic connection:
ESTAB 0 0 192.168.1.10:443 1.2.3.4:5678
cubic rto:204 rtt:0.187/0.037 mss:1448 cwnd:10 bytes_retrans:4500 retrans:0/1In retrans:0/1, the first digit shows current unacknowledged retransmissions. The second digit is the total count for that session. If you see bytes_retrans climbing into the megabytes, you’ve found your bottleneck.
Root Cause Analysis with TShark
Once you know retransmissions are happening, you need to see the “shape” of the failure. Is the server failing to send, or is the client failing to acknowledge?
Capturing a Trace
Record traffic on your active interface (like eth0) and save it to a file. Keep the capture short to avoid filling your disk.
sudo tshark -i eth0 -f "tcp" -w trace.pcapFiltering for Errors
Analyze the file using a filter that highlights only the problematic packets:
tshark -r trace.pcap -Y "tcp.analysis.retransmission"If you see a pattern of “Previous segment not captured” followed by a retransmission, an upstream device like a load balancer or a firewall is likely dropping packets before they even reach your server.
Proven Solutions for Common Bottlenecks
After pinpointing the issue, apply these fixes based on what you found.
1. Fix the “Black Hole” MTU
If small pings work but large file transfers stall at 99%, you likely have an MTU mismatch. This happens when your server tries to send 1500-byte packets, but a VPN or tunnel along the path can only handle 1400 bytes. The packet is dropped silently.
Test it: Try sending a large packet that cannot be fragmented.
ping -M do -s 1472 8.8.8.8If this fails but a standard ping works, lower your MTU to 1400:
sudo ip link set dev eth0 mtu 14002. Clear Kernel Buffer Bottlenecks
When nstat shows TcpExtTCPBacklogDrop, it means your application is too slow to pull data out of the kernel’s queue. The buffer fills up, and the kernel starts binning new packets.
Increase the backlog limits in /etc/sysctl.conf to give your app more breathing room:
net.core.netdev_max_backlog = 5000
net.ipv4.tcp_max_syn_backlog = 4096Run sudo sysctl -p to apply the settings.
3. Switch to BBR Congestion Control
The default cubic algorithm handles packet loss poorly on long-distance or busy links. Google’s BBR (Bottleneck Bandwidth and Round-trip propagation time) is much smarter. It ignores minor packet loss and focuses on actual throughput.
Enable it with these commands:
sudo sysctl -w net.core.default_qdisc=fq
sudo sysctl -w net.ipv4.tcp_congestion_control=bbrSummary Checklist
Follow this workflow the next time the network feels slow:
- Run nstat to see if global retransmission counters are ticking up.
- Use ss -ti to find which specific IP addresses are struggling.
- Capture a tshark trace to see if packets are arriving out of order.
- Check MTU sizes if the connection hangs during large data transfers.
- Enable BBR if you are dealing with high-latency, cross-region traffic.
