Fixing the Single-Core Bottleneck: Scaling Linux Networking with RSS, RPS, and RFS

The Single-Core Bottleneck in High-Traffic Servers

You have a server with 64 CPU cores and a 10Gbps uplink, yet network performance is hitting a wall. If you run top during a traffic spike, you might notice a strange sight: CPU0 is pinned at 100% in the “si” (software interrupt) column, while the other 63 cores are essentially idling. This is the classic single-core bottleneck.

By default, many Network Interface Cards (NICs) handle all incoming packets on a single CPU interrupt. When traffic hits 1 million packets per second (pps) or higher, a single core simply cannot keep up with the interrupt processing. The kernel begins dropping packets, leading to 200ms+ latency spikes and connection timeouts. Solving this doesn’t require new hardware; it requires telling the Linux kernel how to distribute that load.

Understanding the Solutions: RSS, RPS, and RFS

Linux offers three primary tools to spread network processing across your CPU topology. The right choice depends on whether your hardware handles the heavy lifting or if the kernel needs to intervene via software.

1. RSS (Receive Side Scaling)

RSS is the most efficient method because it happens at the hardware level. The NIC uses multiple hardware receive queues. When a packet arrives, the NIC calculates a hash based on the IP addresses and ports (the 4-tuple). It then assigns that packet to a specific queue tied to a specific CPU core. This happens before the packet even reaches the Linux kernel.

2. RPS (Receive Packet Steering)

RPS is the software-based alternative to RSS. It is a lifesaver for virtualized environments, such as AWS EC2 t3 instances or older hardware that only provides a single RX queue. Once a single core receives the packet, RPS calculates a hash and hands the packet off to other CPUs for further processing. While this consumes a small amount of extra CPU cycles for the handoff, it prevents a single core from becoming a total system choke point.

3. RFS (Receive Flow Steering)

RFS improves upon RPS by making it application-aware. Instead of just spreading packets randomly, RFS tracks which core your application (like Nginx or Redis) is actually using. It then steers packets to that specific core. This increases L1/L2 cache hits significantly, reducing the latency caused by moving data between different CPU caches.

Comparing the Approaches

The Optimization Hierarchy

Scaling a server’s network stack usually follows a logical progression. Start with hardware queues, then fill the gaps with software steering.

1. Maximize RSS: If your NIC supports 8 queues but you have 16 cores, configure all 8 queues first.
2. Layer RPS: Use RPS to bridge the gap between your hardware queues and your total core count. For example, use RPS to spread those 8 hardware queues across all 16 available cores.
3. Fine-tune with RFS: Enable RFS if you are running database workloads or high-concurrency web servers where every microsecond of cache latency matters.

Implementation Guide

Step 1: Configure Hardware Queues (RSS)

Verify your hardware capabilities using ethtool. You want to see how many “Combined” channels your driver supports.

# Check current and max queue settings
sudo ethtool -l eth0

If the current “Combined” value is lower than the maximum, increase it to match your CPU core count (or the maximum the hardware allows):

# Increase queues to 8
sudo ethtool -L eth0 combined 8

Step 2: Configure Software Steering (RPS)

RPS uses a bitmask to define which CPUs can process packets. This mask is a hexadecimal representation of your CPU cores. For instance, a mask of f (binary 1111) uses cores 0-3, while ff uses cores 0-7.

# Enable RPS on queue 0 for the first 8 cores (hex ff)
echo "ff" | sudo tee /sys/class/net/eth0/queues/rx-0/rps_cpus

If your NIC has multiple receive queues (rx-0, rx-1, etc.), you must apply this mask to every queue directory to ensure uniform distribution.

Step 3: Enable Application Steering (RFS)

RFS requires a two-tier configuration. First, set the global limit for the total number of flows the kernel should track. For a busy web server, 32,768 is a solid starting point.

sudo sysctl -w net.core.rps_sock_flow_entries=32768

Next, set the flow count for each individual hardware queue. This is usually calculated as total_entries / number_of_queues. For a single-queue setup, match the global value:

# Set flow count for the first queue
echo "32768" | sudo tee /sys/class/net/eth0/queues/rx-0/rps_flow_cnt

Verifying the Results

Don’t assume the settings worked just because the commands returned no errors. You need to observe the actual CPU behavior under load. Use mpstat to watch software interrupt distribution in real-time.

# Monitor all CPUs every 1 second
mpstat -P ALL 1

Look at the %soft column. Before these changes, CPU0 likely sat at 90-100% while others stayed at 0%. After a successful configuration, you should see %soft values distributed across all cores in your mask, such as 10-15% per core.

Final Thoughts

Efficient packet distribution is about more than just throughput; it’s about system predictability. When one core is maxed out by interrupts, the kernel often struggles to schedule other critical tasks, leading to jitter. By implementing RSS, RPS, and RFS, you ensure your network stack scales linearly with your hardware, allowing your server to handle heavy traffic without breaking a sweat.