Why the Kernel Network Stack Becomes the Bottleneck
Every time a packet arrives at a Linux system, it travels through the kernel’s networking stack — interrupt handling, socket buffers, netfilter hooks, routing table lookups. At 1 Gbps this is perfectly fine. Push it to 10 Gbps or 100 Gbps and you start seeing CPU saturation, packet drops, and latency spikes that no amount of kernel tuning can fix.
I’ve benchmarked four approaches for a software router project: pure kernel forwarding, Open vSwitch, DPDK with custom code, and FD.io VPP. The throughput range runs from 1–3 Mpps per core at the bottom to 40–80 Mpps at the top — a 40x gap that determines whether you need one server or twenty.
Four Approaches to High-Speed Packet Forwarding
Linux Kernel Forwarding (iptables + ip route)
The default. Works everywhere, zero extra setup. For a home lab or a lightly-loaded gateway, it’s perfectly fine. But the kernel processes packets one at a time through multiple layers — netfilter, conntrack, routing — and each packet incurs interrupt overhead. On a modern server you’re looking at roughly 1–3 Mpps per core before things get shaky.
Open vSwitch (OVS)
OVS adds OpenFlow programmability with a kernel datapath for fast forwarding and a userspace daemon for complex rules. It fits well in an SDN environment where you need OpenFlow-driven traffic steering. The fast path is still bounded by per-packet kernel overhead, and the userspace path is slow. On a modern server with a 10GbE NIC, expect 3–5 Mpps before you hit the ceiling.
DPDK with Custom Application
DPDK bypasses the kernel entirely by polling NICs directly from userspace. You can write a forwarding loop hitting 20–40 Mpps per core on a 10GbE NIC. The catch: DPDK is a library, not a framework. It handles the NIC, but routing logic, ARP, VLAN handling — all of that is on you. Building a full router from DPDK primitives is months of work.
FD.io VPP (Vector Packet Processing)
VPP is what you get when Cisco engineers open-source their fast-path forwarding code. It runs on DPDK and processes packets in vectors — batches of up to 256 at a time rather than one by one. The built-in feature set is extensive: L2 bridging, L3 routing, NAT, VLAN, MPLS, GRE, VXLAN, and more. I’ve consistently seen 40–80 Mpps on a single 10GbE port with two CPU cores using commodity x86 hardware.
VPP Pros and Cons: What You’re Actually Getting
What Works Well
- Vector processing model — batching 256 packets per cycle amortizes function call overhead, cache misses, and branch mispredictions. This is the core reason VPP is fast.
- Graph-node architecture — packet processing is a DAG of nodes (ip4-lookup, ip4-rewrite, etc.). Each node is a tight C loop over a packet vector. Easy to profile, easy to extend.
- Production-grade feature set — L2/L3/MPLS, NAT44/NAT64, ACL, VXLAN/GRE tunnels, segment routing — all built in and fast without adding third-party plugins.
- Programmatic control — full control via
vppctlCLI or the binary API with Python bindings for automation.
Where It Gets Annoying
- Steep learning curve — VPP has its own mental model. Concepts like “interface punt”, “feature arc”, and “FIB” behave differently from Linux networking.
- DPDK binding requirement — NICs must be unbound from the kernel driver and handed to DPDK. Your NIC disappears from the OS during VPP operation, which complicates management.
- Not designed for shared hosts — if you need VPP and normal Linux networking on the same machine, you need careful CPU pinning and separate interface planning.
- Documentation gaps — official docs cover the basics well, but advanced topics like custom node development mean reading source code or lurking in the community Slack.
Recommended Setup
Here’s the hardware I’d put together for a dedicated software router:
- CPU: Any modern Intel/AMD with 4+ cores. Reserve 1–2 cores for the VPP main thread, 1 core per NIC queue for forwarding workers.
- NICs: Intel X520 (ixgbe), X710 (i40e), or Mellanox ConnectX-4/5. DPDK driver support is the deciding factor — check the DPDK compatibility list before buying.
- Memory: 4 GB minimum. VPP allocates hugepages for DPDK buffers.
- OS: Ubuntu 22.04 LTS or Rocky Linux 9. VPP packages exist for both via the fd.io package repository.
Implementation Guide: From Zero to Forwarding Packets
Step 1: Enable Hugepages
# Add to /etc/default/grub:
GRUB_CMDLINE_LINUX="default_hugepagesz=2M hugepagesz=2M hugepages=1024"
sudo update-grub && sudo reboot
# Verify after reboot
grep HugePages /proc/meminfo
Step 2: Install VPP
# Ubuntu 22.04
curl -s https://packagecloud.io/install/repositories/fdio/release/script.deb.sh | sudo bash
sudo apt-get install vpp vpp-plugin-core vpp-plugin-dpdk vpp-dev
# Verify
vppctl show version
Step 3: Bind NICs to DPDK
# Find PCI addresses
dpdk-devbind.py --status
# Example output:
# 0000:01:00.0 'Ethernet Controller X710' if=ens3 drv=i40e unused=vfio-pci
# 0000:01:00.1 'Ethernet Controller X710' if=ens4 drv=i40e unused=vfio-pci
# Bind to vfio-pci
sudo modprobe vfio-pci
sudo dpdk-devbind.py -b vfio-pci 0000:01:00.0 0000:01:00.1
Step 4: Configure VPP Startup
unix {
nodaemon
log /var/log/vpp/vpp.log
cli-listen /run/vpp/cli.sock
exec /etc/vpp/init.conf
}
cpu {
main-core 0
corelist-workers 2-3
}
dpdk {
dev 0000:01:00.0 {
name eth0
num-rx-queues 2
}
dev 0000:01:00.1 {
name eth1
num-rx-queues 2
}
}
buffers {
buffers-per-numa 128000
}
Step 5: Configure L3 Routing
Drop your interface and routing commands into /etc/vpp/init.conf — VPP runs these automatically on startup:
set interface ip address eth0 192.168.1.1/24
set interface ip address eth1 10.0.0.1/24
set interface state eth0 up
set interface state eth1 up
ip route add 10.10.0.0/16 via 10.0.0.254
# Check the routing table
sudo vppctl show ip fib
# Real-time per-node packet rates
sudo vppctl show runtime
# Interface statistics
sudo vppctl show interface statistics
Step 6: Automate with the Python API
from vpp_papi import VPPApiClient
from ipaddress import IPv4Address
vpp = VPPApiClient()
vpp.connect("my-controller")
# List all interfaces
for iface in vpp.api.sw_interface_dump():
print(f"{iface.interface_name}: {iface.flags}")
# Add a route programmatically
vpp.api.ip_route_add_del(
is_add=1,
route={
'prefix': {
'address': {'un': {'ip4': bytes(IPv4Address('10.20.0.0'))}, 'af': 0},
'len': 16
},
'n_paths': 1,
'paths': [{
'nh': {'address': {'un': {'ip4': bytes(IPv4Address('10.0.0.254'))}, 'af': 0}},
'sw_if_index': 2,
'proto': 0
}]
}
)
vpp.disconnect()
Production Tips That Actually Matter
My current setup pairs a 4-core Xeon with dual X710 NICs, sustaining 12–15 Gbps of mixed traffic without breaking a sweat. Getting there meant learning a few things the hard way:
- Isolate CPUs at boot — add
isolcpus=2-3to your GRUB line matching the VPP worker cores. Without this, the OS scheduler interrupts forwarding threads and you see latency spikes under load. - NUMA alignment is non-negotiable — if your server has two NUMA nodes, VPP worker cores and the NIC must be on the same NUMA node. Cross-NUMA memory access cuts throughput by 30–40%.
- Match rx-queues to workers — set
num-rx-queuesequal to the number of worker threads. If they don’t match, some queues go unserviced and you get uneven load distribution. - Use
show runtimeto find bottlenecks — this command shows cycles-per-packet per graph node. Ifip4-rewriteis consuming most cycles, your ARP cache is likely cold and triggering slow-path resolution. - Benchmark with T-Rex, not iperf — iperf3 measures TCP throughput, not raw forwarding rate. T-Rex generates line-rate traffic and gives you accurate Mpps measurements with latency histograms.
VPP is not for every use case. Routing a few hundred Mbps? The kernel stack handles that fine. But when you need multi-10G or 100G forwarding on commodity hardware without paying for dedicated ASICs, nothing else comes close. VPP gives you a forwarding plane that rivals dedicated hardware in raw speed — programmable, extensible, and fully open-source. The first week is steep. After that, it’s just another tool in your stack.
