Posted inNetworking

High-Speed Packet Processing with DPDK on Linux: Installation and Building Your First Network App

Networking tutorial - IT technology blog
Networking tutorial - IT technology blog

Quick Start: Get DPDK Running in 5 Minutes

I remember the first time a network team lead dropped a packet capture showing 40% packet loss at 10Gbps on a standard Linux box. The kernel network stack was choking. That’s when DPDK entered my toolkit — and honestly, it’s one of those things I wish I’d learned two years earlier.

DPDK (Data Plane Development Kit) lets userspace applications talk directly to your NIC, completely bypassing the kernel network stack. No interrupt handling, no context switches, no copying packets between kernel and user space. Just raw, polling-mode speed.

Let’s get a working environment up fast, then dig into what’s actually happening underneath.

Step 1: Install DPDK

On Ubuntu 22.04 / Debian-based systems:

# Install DPDK development libraries and tools
apt update
apt install -y dpdk dpdk-dev libdpdk-dev dpdk-kmods-dkms python3-pyelftools

# Verify installation
dpdk-devbind.py --version

For RedHat/Rocky Linux:

dnf install -y dpdk dpdk-devel rdma-core-devel

Step 2: Configure Hugepages

DPDK relies on hugepages for memory efficiency — standard 4KB pages create massive TLB pressure at high packet rates. You want 2MB pages (or 1GB on supported hardware).

# Allocate 1024 x 2MB hugepages (= 2GB total)
echo 1024 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages

# Make it persistent across reboots
# NOTE: If GRUB_CMDLINE_LINUX_DEFAULT already exists in your grub file,
# append hugepage params to the existing value — don't add a second definition.
cat >> /etc/default/grub <<'EOF'
GRUB_CMDLINE_LINUX_DEFAULT="hugepages=1024 hugepagesz=2M"
EOF
update-grub

# Mount the hugepage filesystem
mkdir -p /dev/hugepages
mount -t hugetlbfs nodev /dev/hugepages

# Verify
grep HugePages /proc/meminfo

Step 3: Bind Your NIC to a DPDK Driver

This is the step that trips people up. Once you bind a NIC to the DPDK driver (vfio-pci or igb_uio), it disappears from the regular Linux network stack — no more eth1 or similar. Plan accordingly.

# List available NICs and their current drivers
dpdk-devbind.py --status

# Load the vfio-pci driver (recommended, IOMMU-safe)
modprobe vfio-pci

# Enable IOMMU if not already on (add to grub: intel_iommu=on)
# Then bind your NIC (replace 0000:03:00.0 with your PCI address)
dpdk-devbind.py --bind=vfio-pci 0000:03:00.0

# Confirm binding
dpdk-devbind.py --status

Use lspci | grep -i ethernet to find your PCI addresses before running the bind command.

How DPDK Actually Works (Deep Dive)

Skip this section and you’ll spend hours debugging things that aren’t bugs — they’re misconfiguration. Here’s what actually happens when DPDK moves packets at line rate.

Poll Mode Drivers vs. Interrupt-Driven Networking

The standard Linux network stack is interrupt-driven. A packet arrives, the NIC fires an interrupt, the kernel wakes up, copies from NIC memory into kernel space, then copies again into userspace. Two copies per packet. At 10Gbps with 64-byte frames, that’s up to 14.88 million interrupts per second — your CPU spends more time context-switching than processing actual packet data.

DPDK flips this completely. A dedicated CPU core runs a tight polling loop (PMD — Poll Mode Driver), constantly asking the NIC: got any packets? No interrupt, no context switch, no kernel involvement. The core burns 100% CPU, but you get deterministic, line-rate throughput.

# See this in action — a DPDK PMD core will show 100% CPU usage
top -H  # Look for your l2fwd or testpmd process threads

The Environment Abstraction Layer (EAL)

EAL is the runtime layer every DPDK app depends on — a minimal OS-within-an-OS that handles platform-specific work so your application code doesn’t have to. It manages:

  • CPU core allocation and pinning (NUMA-aware)
  • Memory pool management (mempools backed by hugepages)
  • PCI device initialization and driver binding
  • Cross-platform abstraction (Linux, FreeBSD)

The first thing any DPDK app does — before opening ports, creating mempools, or spawning threads — is initialize EAL:

int main(int argc, char *argv[]) {
    // EAL init — must come first
    int ret = rte_eal_init(argc, argv);
    if (ret < 0)
        rte_exit(EXIT_FAILURE, "EAL init failed\n");

    // Your application logic follows
    // ...
    return 0;
}

Memory Architecture: mbufs and mempools

Packets in DPDK are represented as rte_mbuf structures, allocated from pre-allocated memory pools (rte_mempool). Pre-allocation eliminates malloc/free from the hot path entirely. At 14Mpps, even a single malloc per packet introduces microsecond stalls that compound into visible latency spikes under load.

// Create a mempool for packet buffers
struct rte_mempool *mbuf_pool = rte_pktmbuf_pool_create(
    "MBUF_POOL",      // name
    8192,             // number of mbufs
    256,              // per-core cache size
    0,                // application private size
    RTE_MBUF_DEFAULT_BUF_SIZE,
    rte_socket_id()   // NUMA socket
);
if (!mbuf_pool)
    rte_exit(EXIT_FAILURE, "Cannot create mbuf pool\n");

Building and Running Your First DPDK Application

The fastest way to see DPDK in action is with the included l2fwd (Layer 2 forwarder) example. It forwards packets between port pairs — simple, but it demonstrates the full PMD pipeline.

Build the Example Applications

# Find DPDK examples directory
ls /usr/share/dpdk/examples/

# Build l2fwd
cd /usr/share/dpdk/examples/l2fwd
make RTE_SDK=/usr/share/dpdk RTE_TARGET=x86_64-native-linuxapp-gcc

# Or with meson/ninja (DPDK >= 20.11)
meson build
ninja -C build

Run l2fwd

# EAL args come before --, application args come after
# -l 0,1: use cores 0 and 1
# -n 4: 4 memory channels
# -- -p 0x3: enable ports 0 and 1
sudo ./build/l2fwd \
    -l 0,1 \
    -n 4 \
    -- \
    -p 0x3 \
    --portmask=0x3

Per-port statistics update every second in the terminal. With a traffic generator hitting the bound NIC, expect forwarding rates between 5–14Mpps depending on packet size, with packet loss near zero once the setup is dialed in.

Testing with testpmd (No Custom Code Required)

Don’t write a single line of application code until testpmd confirms your environment works. It validates hugepages, driver binding, and NIC compatibility in one shot — if it can’t receive packets, no amount of custom code will fix the underlying problem.

# Run in rxonly mode to measure raw receive throughput
sudo dpdk-testpmd \
    -l 0-3 \
    -n 4 \
    -- \
    --nb-cores=2 \
    --nb-ports=1 \
    --rxq=2 \
    --txq=2 \
    --forward-mode=rxonly

# Inside testpmd prompt:
testpmd> show port stats all
testpmd> start

Practical Tips From Real Deployments

Getting DPDK running is one afternoon’s work. Getting it to actually perform is where most engineers lose a week. The typical pattern: you hit 50–60% of theoretical throughput, assume the NIC is the bottleneck, and start chasing hardware specs. Usually it’s one of three things — PMD cores interrupted by the OS scheduler, packet buffers crossing NUMA boundaries, or too few RX queues for your core count.

Isolate CPUs from the Linux Scheduler

Your PMD cores must not be interrupted by OS scheduling. Add isolcpus to your kernel boot parameters:

# In /etc/default/grub, add to GRUB_CMDLINE_LINUX_DEFAULT:
# isolcpus=2,3,4,5 nohz_full=2,3,4,5 rcu_nocbs=2,3,4,5

# After update-grub and reboot, verify:
cat /sys/devices/system/cpu/isolated

Cores 0-1 stay available for the OS. Cores 2+ are reserved for DPDK polling threads. Without this, OS scheduler interruptions cause latency spikes and throughput drops.

NUMA Awareness Matters at Scale

# Check your NUMA topology
numactl --hardware

# Always allocate mempool on the same NUMA node as the NIC
# rte_socket_id() returns the socket of the current thread
# Match PMD cores to the NIC's NUMA socket

# Check which socket your NIC is on:
cat /sys/bus/pci/devices/0000:03:00.0/numa_node

Crossing NUMA boundaries for packet buffers can cut throughput by 30-40% on dual-socket servers. This burned us badly on a 25GbE deployment — the NIC was on socket 0, but we’d pinned the PMD threads to socket 1 cores. Throughput was half of theoretical maximum until we caught it.

Tune Your RSS Queue Count

RSS (Receive Side Scaling) distributes incoming flows across multiple RX queues. More queues = more PMD threads = better multi-core utilization:

# In testpmd, check queue stats per-queue
testpmd> show port 0 rxq 0 desc used count
testpmd> show port stats 0

# For production: match RX queue count to isolated core count
# --rxq=4 --nb-cores=4 is a common starting point for 10GbE

Debugging Packet Drops

# Check NIC-level drop counters (before DPDK even sees packets)
dpdk-devbind.py --status

# Inside testpmd:
testpmd> show port xstats 0
# Look for: rx_missed_errors, rx_no_mbuf_errors
# rx_missed_errors = NIC dropped (mbuf pool exhausted or ring full)
# rx_no_mbuf_errors = application too slow consuming from the ring

If you see rx_no_mbuf_errors, your mempool is too small or your processing logic is too slow. Increase the pool size first, then profile your packet processing function.

Start with testpmd Before Writing Any Code

Every time I set up DPDK on a new server, testpmd is the first thing I run. It catches hugepage misconfig, wrong driver bindings, and IOMMU issues in seconds — before I’ve invested hours in custom code. If it doesn’t show packets flowing, there’s a hardware or OS configuration problem to fix first.

DPDK has a real learning curve — the first afternoon feels like fighting the hardware. But when hugepages are tuned, CPUs are isolated, and your first benchmark comes back at 14Mpps on a 10GbE NIC with 800ns median latency, the investment pays off fast. The kernel network stack becomes something you read about in “why we rewrote our network layer” postmortems.

Share:
Tags: