The 10Gbps Bottleneck Reality
Ever plugged a high-end SFP+ module into a server and expected a glorious 10Gbps stream, only to see it crawl at 4.2Gbps? I’ve been there. Watching a single CPU core peg at 100% while the rest of the system sits idle is a frustrating wake-up call. High-speed networking on Linux isn’t a “plug-and-play” experience; it requires the OS to be as fast as the glass fibers carrying the data.
Standard Linux distributions prioritize compatibility. They ship with settings optimized for 1Gbps office LANs or home connections. When you scale to 10Gbps, 25Gbps, or 100Gbps, these defaults become a massive liability. Processing 1,500-byte packets at 10Gbps speeds forces the CPU to handle over 800,000 interrupts every second. This creates a “death by a thousand cuts” scenario where the processor spends more time managing headers than moving actual data.
If you’re building Ceph storage clusters, high-traffic NVMe-over-Fabrics nodes, or virtualization hosts, these tweaks aren’t optional. They are the difference between getting the 9.41Gbps theoretical maximum or wasting thousands of dollars on hardware you aren’t fully using.
Why Default Settings Fail at High Speeds
Before changing configurations, you need to understand the three primary limiters in the Linux networking stack.
1. The MTU and Jumbo Frames
The Maximum Transmission Unit (MTU) dictates the largest packet size allowed on your wire. The standard 1500-byte MTU is a relic of 1980s Ethernet. By increasing this to 9000 bytes (Jumbo Frames), you allow the system to pack six times more data into a single packet. This simple shift slashes the number of CPU interrupts required to move the same amount of data.
2. TCP Window Scaling
TCP uses a “window” to decide how much data can stay in flight before the sender needs an acknowledgment (ACK). On high-latency or high-bandwidth links, a small window causes the sender to pause and wait. This creates the “long fat pipe” problem where bandwidth is available, but the protocol refuses to use it.
3. Ring Buffers and Interrupts
Your Network Interface Card (NIC) has internal queues called Ring Buffers. If these buffers are too small, the NIC will drop packets before the CPU even has a chance to see them. Furthermore, how the NIC notifies the CPU—known as Interrupt Coalescing—can be tuned to prioritize raw throughput over micro-second latency.
Step 1: Implementing Jumbo Frames (MTU 9000)
This change offers the highest performance gains, but there is a catch. Every single device in the data path must support Jumbo Frames. This includes the source, the destination, and every switch in between. If a single legacy switch in the middle is capped at 1500, your packets will be dropped or fragmented, destroying your performance.
To test Jumbo Frames on the fly for an interface like eth0:
sudo ip link set dev eth0 mtu 9000Verify the change immediately with:
ip link show eth0For permanent changes on Ubuntu or Debian, update your Netplan YAML file in /etc/netplan/:
network:
version: 2
ethernets:
eth0:
mtu: 9000
addresses:
- 10.0.0.10/24Run sudo netplan apply to commit the new MTU.
Step 2: Tuning the Kernel Network Stack (sysctl)
The Linux kernel needs larger memory buffers to handle the sheer volume of data in a 10Gbps stream. We need to expand these so the system doesn’t choke during bursts.
Open /etc/sysctl.conf and add these parameters. These values, specifically the 32MB maximums, are battle-tested for stabilizing 10G links in production environments:
# Increase maximum buffer sizes to 32MB
net.core.rmem_max = 33554432
net.core.wmem_max = 33554432
# TCP buffer sizes: [min, default, max] in bytes
net.ipv4.tcp_rmem = 4096 87380 33554432
net.ipv4.tcp_wmem = 4096 65536 33554432
# Increase the length of the processor input queue
net.core.netdev_max_backlog = 10000
# Ensure TCP Window Scaling (RFC 1323) is active
net.ipv4.tcp_window_scaling = 1
# Raise the limit for concurrent connections
net.core.somaxconn = 4096Activate the changes without rebooting:
sudo sysctl -pStep 3: Optimizing NIC Hardware with ethtool
Hardware-level bottlenecks often hide in the NIC’s own configuration. Use the ethtool utility to see if your hardware is holding back the software.
Increasing Ring Buffers
Check your current hardware capacity:
ethtool -g eth0If the “Current hardware settings” are significantly lower than the “Pre-set maximums,” crank them up. For many Intel and Mellanox cards, this means moving from 512 to 4096:
sudo ethtool -G eth0 rx 4096 tx 4096Interrupt Coalescing
If your goal is maximum throughput for large file transfers, tell the NIC to wait 30 microseconds before interrupting the CPU. This allows it to batch packets together efficiently:
sudo ethtool -C eth0 rx-usecs 30Step 4: Verification and Testing
Never assume the config worked just because the command didn’t return an error. Verify Jumbo Frames are working end-to-end using ping with a “do not fragment” (DF) flag:
ping -M do -s 8972 10.0.0.11Note: We use 8972 because the payload must account for the 28-byte ICMP and IP headers (9000 – 28 = 8972). If the ping returns, your path is clean.
Finally, run a real-world throughput test using iperf3. With an MTU of 9000 and the kernel tuned, you should see results hitting the 9.4Gbps mark with significantly lower CPU usage than your baseline test.
Final Thoughts
Optimizing 10Gbps networking isn’t about a single magic setting. It’s about ensuring the entire path—from the kernel memory to the physical switch—is wide enough for the data. If you can’t control your switches, skip the Jumbo Frames and focus strictly on sysctl tuning and NIC ring buffers. When you finally hit that 9.4Gbps sustained transfer rate, the effort pays for itself in system stability and performance.
