The 9-Minute Maintenance Window
I once had to patch a critical kernel vulnerability on a high-traffic database server. I issued the reboot command and waited. Then I waited some more. Seven minutes passed while the server performed its internal memory checks. Another two minutes vanished during the RAID controller initialization. By the time the OS actually started loading, our monitoring system had already triggered a P1 incident. On a production rig, that 9-minute gap isn’t just an inconvenience—it’s a threat to your SLA.
After managing dozens of enterprise-grade servers, I realized that hardware initialization is the enemy of uptime. This led me to a tool that fundamentally changed my maintenance workflow: kexec.
The Bottleneck: Why Hardware Resets Waste Time
A standard reboot triggers a “cold boot.” The CPU resets, the BIOS/UEFI executes the Power-On Self-Test (POST), and every peripheral—from 10GbE NICs to NVMe controllers—must re-initialize. On a Dell PowerEdge or HP ProLiant with 512GB of RAM, the hardware phase alone can take 5 to 8 minutes.
Linux offers a shortcut. Instead of handing control back to the hardware, we can instruct the running kernel to load a new kernel directly into memory. This bypasses the motherboard’s firmware entirely. This process is handled by the kexec system call.
Quick Start: Your First kexec Reboot
To use this, you need the kexec-tools package. This utility bridges the gap between your shell and the kernel’s internal kexec functions.
1. Install the Tools
For Ubuntu or Debian-based systems:
sudo apt update && sudo apt install kexec-toolsFor RHEL, AlmaLinux, or Rocky Linux:
sudo dnf install kexec-tools2. Locate Your Target Kernel
You must point kexec to the specific kernel and initrd (initial RAM disk) you want to boot. These files live in the /boot directory. Usually, you’ll want the highest version number available.
ls /boot/vmlinuz* /boot/initrd.img*3. Stage the New Kernel
This step loads the kernel into a reserved area of memory without running it yet. Replace the placeholders with your actual file paths:
sudo kexec -l /boot/vmlinuz-$(uname -r) \
--initrd=/boot/initrd.img-$(uname -r) \
--reuse-cmdline-l: Loads the kernel into memory.--initrd: Points to the RAM disk image.--reuse-cmdline: Copies the current boot parameters (like root UUID and console settings) to the new kernel.
4. Trigger the Jump
When you’re ready to switch, run:
sudo kexec -eYour SSH session will hang instantly. Within 20 to 30 seconds, the server will be back online, having skipped the entire BIOS sequence.
How kexec Works Under the Hood
The kexec_load system call is the engine here. When you stage a kernel, the OS allocates non-contiguous memory pages to store the new image. To avoid overwriting the memory currently used by the active kernel, kexec uses a “control page” containing architecture-specific machine code.
Once you execute the jump (-e), the system follows a strict sequence:
- The current kernel shuts down subsystems like networking and storage as cleanly as possible.
- The CPU disables all interrupts.
- The control code moves the new kernel image into its final, contiguous memory destination.
- The CPU jumps directly to the entry point of the new kernel.
Because the hardware never loses power, the CPU never cycles back to the reset vector. You effectively swap the “brain” of the server while the body stays awake.
Advanced Usage: Integration with systemd
Manually loading kernels works for a one-off test, but production environments require automation. Most modern distributions use systemd to manage this process safely.
Checking the Load Status
You can verify if a kernel is staged and ready by checking a virtual file in the sysfs filesystem:
cat /sys/kernel/kexec_loadedA 1 means you are ready to jump; a 0 means no kernel is staged.
The Safer systemd Approach
Using kexec -e is blunt. It doesn’t always give services enough time to stop. Instead, use the systemd wrapper, which ensures that databases flush buffers and filesystems unmount properly before the kernel swap:
sudo systemctl kexecI recommend this for database nodes. It provides the speed of kexec with the data integrity of a standard shutdown.
Real-World Caveats and Safety
kexec is powerful, but it isn’t a universal replacement for reboots. Here is what I’ve learned from deploying this in the field.
Handling “Dirty” Hardware
Since the motherboard doesn’t reset, some hardware devices might stay in an active state. If a driver for the new kernel doesn’t properly re-initialize a specific NIC or Fiber Channel card, that device might hang. Always test kexec on a staging server that mirrors your production hardware. If a device fails to wake up, you may need to perform a traditional cold boot.
Secure Boot Restrictions
If UEFI Secure Boot is active, the standard kexec -l command often fails with a “Permission Denied” error. This is a security feature to prevent unsigned kernels from being side-loaded. To bypass this, use the -s flag (kexec -s -l ...). This forces the kernel to use its own signature verification mechanism rather than the user-space utility.
When to Use a Cold Boot
Kexec only updates the software layer. If you are updating BIOS/UEFI firmware, changing RAID levels, or swapping physical RAM, you must perform a standard reboot. Kexec cannot apply hardware-level configuration changes.
Final Thoughts
By incorporating systemctl kexec into your patching scripts, you can reduce a 10-minute downtime window to less than 60 seconds. It turns a stressful maintenance event into a minor blip. Use it for routine kernel updates to keep your services responsive and your users happy.
