Linux Networking at Scale: Mastering IRQ Affinity, RSS, and RPS

Table of Contents

The Invisible Ceiling: Why Your Server Drops Packets at Low Load

We were pushing roughly 800,000 packets per second (PPS) through an API gateway when the timeouts hit. On paper, the server looked healthy. Bandwidth sat at a modest 300Mbps on a 10Gbps link, and the total CPU load across the 16-core machine hovered around 15%. Yet, netstat -s reported thousands of dropped packets every second.

A quick look at top revealed the bottleneck. While 15 cores were essentially idling, Core 0 was pinned at 100% in %si (software interrupt) mode. The server wasn’t struggling with data volume; it was choking on the interrupt overhead. Every tiny packet triggered an event that only one core was trying to handle. If you’re scaling infrastructure beyond a few basic VMs, learning to distribute this load is a requirement, not an option.

The Single-Core Trap: Understanding Interrupts

When a Network Interface Card (NIC) receives a packet, it signals the CPU via an Interrupt Request (IRQ). By default, many Linux distributions route all IRQs from a specific NIC to CPU 0. This creates a massive performance hurdle.

Imagine receiving 1.5 million small packets per second—typical for high-frequency microservices or a UDP-based game server. That single core must stop its current task 1.5 million times a second to manage the NIC. Even with a 64-core EPYC or Xeon processor, your networking throughput is limited by the speed of that one single core.

Hardware vs. Software Scaling

We solve this by spreading the workload. There are three main tools in the Linux arsenal:

RSS (Receive Side Scaling): A hardware feature where the NIC distributes packets into multiple queues. Each queue has its own IRQ that different CPUs can handle.
RPS (Receive Packet Steering): The software version of RSS. The kernel grabs packets from a single hardware queue and hands them off to other CPUs for protocol processing.
IRQ Affinity: A configuration that binds specific IRQs to specific CPU cores, preventing the kernel from bouncing tasks between cores and destroying cache performance.

Step 1: Diagnosing the Bottleneck

First, verify if your interrupts are lopsided. Run this command to watch the distribution in real-time:

watch -n 1 "cat /proc/interrupts | grep eth0"

(Replace eth0 with your actual interface, like ens3 or p4p1).

If the numbers in one CPU column are skyrocketing while others stay flat, you have an affinity issue. You should also check mpstat from the sysstat package:

mpstat -P ALL 1

Focus on the %soft column. If one core shows 90-100% while others show 0%, that core is drowning in network interrupts.

Step 2: Enabling RSS (Hardware Queues)

Most modern 10GbE and 25GbE NICs support multiple queues. Check your hardware limits with ethtool:

sudo ethtool -l eth0

If “Combined” shows 1, but “Maximum” is higher, you aren’t using your hardware’s full potential. Increase the active queues to match your core count (or a maximum of 8 for most workloads):

sudo ethtool -L eth0 combined 8

Increasing queues allows the NIC to talk to multiple CPUs simultaneously.

Step 3: Pinning IRQ Affinity

The irqbalance service usually manages this, but it often makes poor decisions on high-traffic servers. It frequently moves interrupts between cores, which flushes CPU caches and spikes latency. For predictable performance, disable it.

sudo systemctl stop irqbalance
sudo systemctl disable irqbalance

Now, find the IRQ numbers for your NIC queues:

grep eth0 /proc/interrupts | awk '{print $1}' | sed 's/://'

To map IRQ 45 to CPU 0, you write a hexadecimal bitmask to the smp_affinity file. In this mask, 1 is CPU 0, 2 is CPU 1, 4 is CPU 2, and 8 is CPU 3.

# Bind IRQ 45 to CPU 0
echo 1 | sudo tee /proc/irq/45/smp_affinity

# Bind IRQ 46 to CPU 1
echo 2 | sudo tee /proc/irq/46/smp_affinity

Step 4: Using RPS for Virtualized Environments

If you’re on a cloud provider like AWS or DigitalOcean, your virtual NIC might only support a single queue. This is where RPS saves the day. It handles the heavy lifting of the TCP/IP stack across multiple cores in software.

To enable RPS, write a CPU mask to the rps_cpus file. For an 8-core system, a mask of ff (binary 11111111) lets the kernel use every core.

# Enable RPS for eth0 on all 8 cores
echo "ff" | sudo tee /sys/class/net/eth0/queues/rx-0/rps_cpus

Step 5: Fine-Tuning with RFS

RFS (Receive Flow Steering) takes optimization further. It tracks which CPU is actually running the application (like Nginx or HAProxy) and directs the packet processing to that same core. This minimizes expensive cross-core data transfers.

Set the global flow table limit first:

sudo sysctl -w net.core.rps_sock_flow_entries=32768

Then, set the limit for each specific queue:

echo 4096 | sudo tee /sys/class/net/eth0/queues/rx-0/rps_flow_cnt

The Production Strategy

In high-load environments, I’ve found that a hybrid approach works best:

Maximize Hardware: Set ethtool queues to match your physical cores (up to 8 or 16).
Manual Pinning: Bind each NIC queue to its own physical core. Avoid CPU 0 if possible, as it usually handles system timers and disk I/O.
Software Fallback: Use RPS and RFS only if your hardware queues are fewer than your CPU cores.

After applying these tweaks to our API gateway, we saw the %si load drop from 100% on one core to a balanced 12% across all cores. Latency plummeted by 40ms during peak hours, and the packet loss vanished. Tuning the network stack isn’t about magic; it’s about ensuring the data has a clear, multi-lane highway to the CPU.