Why SR-IOV Changed How I Think About VM Networking
Running a bare-metal hypervisor with 20+ KVM guests, I kept hitting the same wall: the virtual bridge (virbr0) and virtio networking were eating CPU cycles and introducing latency that showed up clearly in iperf3 benchmarks. A 10GbE NIC was delivering maybe 4–5 Gbps to a VM through the software stack. That’s painful when the whole point of using physical hardware is performance.
SR-IOV — Single Root I/O Virtualization — is the hardware answer to that problem. The NIC itself exposes multiple lightweight Virtual Functions (VFs) that map almost directly into the guest OS, cutting the host kernel out of the data path. I’ve run this in production across multiple clusters, and it holds up. Throughput consistently hits 9.4–9.8 Gbps on a 10GbE card — no tuning tricks, no DPDK, just the VF passed straight to the guest.
This guide covers the full path: verifying hardware support, enabling IOMMU, creating VFs, and assigning them to KVM guests. Tested on Ubuntu 22.04 and RHEL 9, with Intel X550 and Mellanox ConnectX-4 cards.
Prerequisites and Hardware Verification
SR-IOV requires support at three layers: the NIC itself, the CPU/chipset (IOMMU), and the BIOS. Check them in this order before touching any config file.
Check NIC SR-IOV Support
# Find your NIC's PCI address
lspci | grep -i ethernet
# Check SR-IOV capability (look for "Single Root I/O Virtualization")
lspci -v -s 0000:03:00.0 | grep -i sriov
# Or check sysfs directly
cat /sys/bus/pci/devices/0000:03:00.0/sriov_totalvfsA number from sriov_totalvfs — like 63 for an Intel X550 or 8 for some Mellanox cards — means the NIC supports SR-IOV. No file? The card doesn’t.
Check IOMMU Support
# Intel CPU
grep -o 'vmx' /proc/cpuinfo | head -1
# AMD CPU
grep -o 'svm' /proc/cpuinfo | head -1
# Check if IOMMU is already active
dmesg | grep -e DMAR -e IOMMU | head -20Lines like DMAR: IOMMU enabled or AMD-Vi: Interrupt remapping enabled mean you’re set. Nothing there? IOMMU is either off in BIOS or not yet passed to the kernel — we’ll fix that next.
Installation: Enabling IOMMU and Loading Drivers
Enable IOMMU in Kernel Boot Parameters
Edit your GRUB config to pass IOMMU flags at boot. The exact parameter depends on your CPU vendor:
# Edit GRUB config
sudo nano /etc/default/grub
# For Intel — add to GRUB_CMDLINE_LINUX_DEFAULT:
# intel_iommu=on iommu=pt
# For AMD — add:
# amd_iommu=on iommu=pt
# Example line after edit:
GRUB_CMDLINE_LINUX_DEFAULT="quiet splash intel_iommu=on iommu=pt"That iommu=pt flag enables passthrough mode — it skips IOMMU translation overhead for devices that stay on the host. Without it, every DMA transaction goes through address remapping even for the host’s own NIC and storage controllers.
# Apply and reboot
sudo update-grub # Debian/Ubuntu
# sudo grub2-mkconfig -o /boot/grub2/grub.cfg # RHEL/CentOS
sudo rebootVerify IOMMU After Reboot
dmesg | grep -e DMAR -e IOMMU
# Expected: "DMAR: IOMMU enabled" or "AMD-Vi: AMD IOMMUv2 loaded and initialized"Install Required Packages
# Ubuntu/Debian
sudo apt install -y qemu-kvm libvirt-daemon-system virt-manager
sudo apt install -y linux-modules-extra-$(uname -r) # extra driver modules
# RHEL/CentOS
sudo dnf install -y qemu-kvm libvirt virt-install
sudo dnf install -y kernel-modules-extraConfiguration: Creating Virtual Functions and Assigning to KVM
Create Virtual Functions on the Physical NIC
Find the physical function (PF) interface name first, then create the VFs:
# Get interface name from PCI address
ls /sys/bus/pci/devices/0000:03:00.0/net/
# Output: ens3f0
# Create 4 Virtual Functions (adjust count based on your needs)
echo 4 | sudo tee /sys/bus/pci/devices/0000:03:00.0/sriov_numvfs
# Verify VFs were created
lspci | grep "Virtual Function"
ip link show ens3f0After the echo command, ip link show should list vf 0 through vf 3 under the physical interface, each with its own MAC address.
Make VF Count Persistent Across Reboots
That sysfs write doesn’t survive a reboot. Use a systemd service to restore it automatically:
# Create a systemd service for persistence
sudo tee /etc/systemd/system/sriov-setup.service <<'EOF'
[Unit]
Description=Enable SR-IOV Virtual Functions
After=network.target
[Service]
Type=oneshot
ExecStart=/bin/bash -c 'echo 4 > /sys/bus/pci/devices/0000:03:00.0/sriov_numvfs'
RemainAfterExit=yes
[Install]
WantedBy=multi-user.target
EOF
sudo systemctl enable --now sriov-setup.serviceFind VF PCI Addresses for Passthrough
# List VF PCI addresses
lspci | grep "Virtual Function"
# Output example:
# 03:10.0 Ethernet controller: Intel Corporation Ethernet Virtual Function 700 Series
# 03:10.2 Ethernet controller: Intel Corporation Ethernet Virtual Function 700 Series
# Check IOMMU group for a VF
for d in /sys/kernel/iommu_groups/*/devices/*; do
echo "Group $(basename $(dirname $(dirname $d))): $(basename $d) $(lspci -nns $(basename $d) 2>/dev/null | cut -d' ' -f3-)"
done | grep "03:10"Each VF should land in its own IOMMU group. Shared groups cause passthrough conflicts — a hardware routing issue that varies by board and can’t be fixed in software alone.
Assign a VF to a KVM Guest via virt-manager / virsh
Two options: XML editing or the virsh command. XML is more explicit and easier to audit:
# Get the VF device XML snippet to add
virsh nodedev-dumpxml pci_0000_03_10_0
# Edit the VM definition and add under <devices>:
sudo virsh edit vm-nameAdd this block inside <devices>:
<hostdev mode='subsystem' type='pci' managed='yes'>
<source>
<address domain='0x0000' bus='0x03' slot='0x10' function='0x0'/>
</source>
<address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
</hostdev>The managed='yes' attribute tells libvirt to automatically detach the VF from the host driver and re-attach it when the VM starts/stops — no manual driver binding needed.
Optional: Set a Specific MAC Address on the VF
# Set MAC on VF 0 of physical interface ens3f0
sudo ip link set ens3f0 vf 0 mac 52:54:00:ab:cd:01
# Verify
ip link show ens3f0Useful when your DHCP server ties leases to MACs — without this, the VF gets a random MAC on each boot.
Verification and Monitoring
Confirm the VF is Visible Inside the Guest
Boot the VM and verify from inside the guest:
# Inside the KVM guest
lspci | grep Ethernet
# Should show Intel or Mellanox device, NOT virtio-net
ip link show
ethtool -i eth0 | grep driver
# driver: iavf (Intel Adaptive Virtual Function driver)
# or: mlx5_core (Mellanox)Throughput Benchmark with iperf3
# On the receiver (another host or VM)
iperf3 -s
# On the guest with SR-IOV VF
iperf3 -c <receiver_ip> -t 30 -P 4
# Compare: same test on a virtio-net interface
# SR-IOV result: ~9.4 Gbps on 10GbE, CPU usage ~8%
# virtio-net result: ~4.5 Gbps on 10GbE, CPU usage ~35%Monitor VF Statistics from the Host
# Watch per-VF stats from host
watch -n 1 'cat /sys/class/net/ens3f0/device/sriov/0/stats 2>/dev/null || \
ip -s link show ens3f0'
# Check VF link state and trust mode
ip link show ens3f0
# Enable VF trust mode (needed for promiscuous in some use cases)
sudo ip link set ens3f0 vf 0 trust on
# Check dmesg for VF errors
dmesg | grep -i 'vf\|iavf\|mlx5' | tail -20Troubleshooting Common Issues
- VF not appearing after echo to sriov_numvfs: Check
dmesg | grep sriov— often a BIOS setting (SR-IOV Enable, ARI Forwarding) needs to be turned on first. - libvirt fails to start VM with VF: Run
virsh nodedev-detach pci_0000_03_10_0manually first, then try starting the VM. Sometimes the auto-detach races with driver binding. - IOMMU group contains multiple devices: This is an ACS (Access Control Services) issue. Some kernels have an ACS override patch; alternatively, pass the entire IOMMU group to the guest.
- Guest loses VF on host reboot: Confirm the systemd service from earlier is enabled and check
systemctl status sriov-setup.serviceafter reboot.
SR-IOV takes more setup than virtio — there’s no pretending otherwise. But once the VFs are configured and the systemd service is in place, you stop thinking about it. The guest sees a real NIC.
CPU overhead on the host drops from ~35% to under 10%. Latency goes sub-millisecond. For workloads pushing serious traffic — database replication, NFS storage backends, high-frequency log pipelines — that overhead gap adds up fast. An hour of BIOS spelunking and udev setup is a one-time cost; the throughput delta runs every day after that.
