The Mystery of the Invisible Hardware
You’ve just installed a Mellanox 10GbE network card or a high-end NVMe controller, but ip link or lsblk shows absolutely nothing. The hardware is physically seated and the LEDs are blinking, yet the operating system acts like it’s invisible. This isn’t usually a hardware failure. Instead, it’s the classic sign of a missing or dormant kernel module.
Linux uses a modular design. Think of kernel modules (.ko files) as plugins that add features on the fly without requiring a full system reboot. When these modules fail to load—or when a driver breaks after a routine apt upgrade—your system’s performance and stability are immediately at risk.
Inside the Modular Kernel
Why not just build everything into the kernel? If every driver for every piece of hardware made since 1991 were compiled directly into the vmlinuz image, the kernel would be massive. It would hog hundreds of megabytes of RAM just to support hardware you don’t even own. Instead, the kernel stays lean, loading only the specific code needed for your environment.
Auditing Your System with lsmod
Before you start changing configurations, you need to see what’s currently running. The lsmod command pulls data from /proc/modules to show you the active stack. In a typical Ubuntu server environment, you might see 80 to 120 modules loaded at any given time.
lsmod | head -n 10The output provides three columns: Module, Size, and Used by. Pay close attention to the “Used by” count. If you try to unload a module that has a non-zero count, the kernel will block the request to prevent an immediate system crash or kernel panic.
Deep Diving with modinfo
If you encounter an ambiguous module name like overlay or iwlwifi, don’t guess its purpose. Use modinfo to inspect it:
modinfo iwlwifiThis command lists the developer, license, and—most importantly—the available parameters. For instance, if your Intel Wi-Fi is dropping connections, modinfo might reveal a power_save parameter that you can toggle to stabilize the link.
Active Management with modprobe
Historically, administrators used insmod and rmmod. These tools are “dumb” because they don’t understand dependencies. If Module A needs Module B to function, insmod will simply error out. modprobe is the modern standard because it handles the entire dependency chain automatically.
Loading and Unloading on the Fly
To inject a module and its required dependencies into the running kernel, use:
sudo modprobe [module_name]To safely remove one:
sudo modprobe -r [module_name]Efficiency matters in production. While tuning a high-traffic web gateway on an Ubuntu 22.04 instance with 4GB of RAM, I stripped out unused modules like floppy, bluetooth, and joydev. This reduced the kernel’s memory footprint by roughly 120MB and slightly decreased interrupt latency.
Refreshing Dependencies
modprobe relies on modules.dep. If you manually compile a driver or move .ko files, modprobe won’t find them until you update the map. Run sudo depmod -a to force the kernel to re-index its available modules and fix pathing issues.
Persistence and Blacklisting
Manual modprobe commands are temporary and disappear after a reboot. To make a driver load every time the system starts, add its name to /etc/modules or create a dedicated configuration file in /etc/modules-load.d/.
# Force load the dummy network interface for testing
echo "dummy" | sudo tee /etc/modules-load.d/dummy.confSidelining Problematic Drivers
Conflicts are common, especially with graphics. The open-source nouveau driver often fights with proprietary NVIDIA drivers. To stop a module from loading entirely, you must blacklist it in /etc/modprobe.d/.
# Create a blacklist file
sudo nano /etc/modprobe.d/blacklist-nouveau.conf
# Add these lines:
blacklist nouveau
options nouveau modeset=0After modifying these files, you must run sudo update-initramfs -u. This ensures the blacklist is baked into the early boot environment before the kernel even mounts your root filesystem.
Automating Rebuilds with DKMS
Few things kill uptime like a kernel security patch that breaks your third-party drivers. If you use out-of-tree drivers—like ZFS, VirtualBox, or specialized RAID drivers—they are tied to a specific kernel version. When the kernel version changes, the driver stops working.
Dynamic Kernel Module Support (DKMS) is the solution. It detects when a new kernel is installed and automatically recompiles your drivers against the new headers. It turns a manual 20-minute compilation task into a background process that happens during apt upgrade.
Check your current DKMS status with:
dkms statusIf a module is stuck in the “added” state, you can force the build manually:
sudo dkms install -m [module_name] -v [version]Troubleshooting the Kernel Ring Buffer
When modprobe fails with a vague “Operation not permitted,” the real answer is hidden in the kernel ring buffer. Use dmesg to see exactly why the kernel is complaining.
sudo dmesg | grep -i [module_name] | tail -n 20Look for these three red flags:
- Unknown symbol: Your module is likely compiled for a different kernel version (check your DKMS status).
- Signature failure: UEFI Secure Boot is active, and you haven’t signed your custom module with a MOK (Machine Owner Key).
- Missing firmware: The module loaded, but it’s searching for a binary blob in
/lib/firmwarethat isn’t there.
Mastering lsmod, modprobe, and dkms transforms the Linux kernel from a mysterious “black box” into a manageable, transparent component of your infrastructure.
