Rebooting a production server at 2 AM because of a critical kernel CVE is not fun — especially when you manage a fleet of 10+ VPS instances spread across multiple datacenters. After going through that cycle too many times, I invested time into setting up kernel live patching properly, and it changed how I handle security updates entirely.
Kernel live patching lets you apply security fixes directly to a running kernel — no downtime, no maintenance window, no frantic Slack messages. Here’s what I’ve learned from actually running this in production.
Quick Start: Get Kernel Live Patching Running in 5 Minutes
The fastest path depends on your distro. For Ubuntu systems enrolled with Canonical Livepatch, you can be up and running almost immediately.
Ubuntu — Canonical Livepatch
Ubuntu offers Livepatch as part of Ubuntu Pro (free for up to 5 machines with a personal account).
# Get your free token at ubuntu.com/security/livepatch
pro attach YOUR_TOKEN_HERE
# Enable livepatch
pro enable livepatch
# Check status
canonical-livepatch status --verboseThat’s literally it for Ubuntu. Within minutes, your machine is receiving live kernel patches automatically.
RHEL / CentOS / AlmaLinux — kpatch
Red Hat’s kpatch gives you more manual control, which I actually prefer on critical servers.
# Install kpatch
dnf install kpatch kpatch-dnf
# Enable the kpatch service
systemctl enable --now kpatch
# Check if kernel supports live patching
grep CONFIG_LIVEPATCH /boot/config-$(uname -r)The output should show CONFIG_LIVEPATCH=y. If it doesn’t, your kernel wasn’t compiled with live patching support — time to check for a different kernel package.
Verify the Kernel Supports klp
# Check kernel live patch subsystem
ls /sys/kernel/livepatch/
# If directory exists and is accessible, you're good
# Empty directory means no patches applied yet — that's normalDeep Dive: How Kernel Live Patching Actually Works
Understanding the mechanism matters when something goes wrong at 3 AM. The kernel live patching framework (klp) works by inserting a new kernel module that redirects function calls from the vulnerable function to the patched version.
The process looks like this:
- A patch module is compiled containing the fixed version of the vulnerable function
- The module gets loaded into the running kernel
- The
klpsubsystem uses kernel function tracing (ftrace) to intercept calls to the original function - All subsequent calls are redirected to the patched version
- Existing tasks complete with the old function; new calls use the patched one
The transition state is the tricky part. You can monitor it:
# List active live patches and their state
cat /sys/kernel/livepatch/*/enabled
# Check transition status (1 = transitioning, 0 = complete)
cat /sys/kernel/livepatch/*/transition
# See which tasks are blocking transition
cat /sys/kernel/livepatch/*/signalIf transition stays at 1 for a long time, some kernel threads haven’t reached a safe point yet. This is usually fine — just wait. Long-running kernel threads (like kworker or storage I/O threads) need to complete their current operation before the patch fully activates.
Applying Patches Manually with kpatch
For RHEL-based systems without an active subscription, you can build and apply patches manually — though this requires kernel development headers.
# Install build dependencies
dnf install kernel-devel kernel-headers elfutils-libelf-devel
# Apply a specific kpatch module (from your vendor or built manually)
kpatch load /path/to/kpatch-module.ko
# List currently loaded patches
kpatch list
# Remove a specific patch (if needed)
kpatch unload kpatch-module-nameAdvanced Usage: Automation and Fleet Management
Running this manually on 10+ servers quickly becomes its own problem. Here’s how I’ve automated the monitoring side.
Monitoring Patch Status Across Multiple Servers
A simple shell script that I run via cron to get a daily summary:
#!/bin/bash
# livepatch-check.sh — run on each server or via SSH loop
SERVERS=("server1.example.com" "server2.example.com" "server3.example.com")
for host in "${SERVERS[@]}"; do
echo "=== $host ==="
ssh -o ConnectTimeout=5 "$host" '
echo "Kernel: $(uname -r)"
if command -v canonical-livepatch &>/dev/null; then
canonical-livepatch status 2>/dev/null | grep -E "(running|last-check|fixed)"
elif command -v kpatch &>/dev/null; then
kpatch list 2>/dev/null || echo "No patches loaded"
else
echo "No live patching configured"
fi
echo "Livepatch transition: $(cat /sys/kernel/livepatch/*/transition 2>/dev/null || echo N/A)"
' 2>/dev/null || echo "SSH failed"
echo
doneForcing Patch Transition (Use With Caution)
Occasionally a patch stays in transition because of a stuck kernel thread. You can try sending a signal to force threads to a safe point:
# Signal stuck tasks to reach a safe patching point
# Only do this if transition has been stuck for hours
echo 1 > /sys/kernel/livepatch/PATCH_NAME/signal
# Check if it resolved
cat /sys/kernel/livepatch/PATCH_NAME/transitionI use this sparingly. If a thread is genuinely busy doing something important (database I/O, heavy computation), forcing the signal can interrupt it. Wait at least 30 minutes before considering this.
Setting Up Automatic Patch Application on Ubuntu
# Check livepatch daemon is running and auto-updating
systemctl status snap.canonical-livepatch.canonical-livepatchd
# The daemon checks for new patches every 60 minutes by default
# View the timer
systemctl list-timers | grep livepatchSUSE Linux Enterprise — kGraft / klp
SUSE uses klp (kernel live patching) integrated directly into their support tooling:
# On SLES with active SCC subscription
zypper install kernel-livepatch-tools
# Apply via zypper (patches show up as regular updates)
zypper patch --category security --advisory SUSE-SU-XXXX:YYYY-1
# Check status
klp statusPractical Tips From 3 Years of Production Experience
After managing 10+ Linux VPS instances over 3 years, I learned to always test thoroughly before applying to production. These are the lessons that actually stuck.
Tip 1: Never Skip Staging
Live patching makes it tempting to push security fixes straight to production — it’s just a module load, right? Wrong mindset. I had a patch conflict once with a custom kernel module we were running for a network driver. The symptoms only appeared under load.
My current workflow: staging server gets the patch, runs for 48 hours under normal load monitoring, then production gets it. Live patching buys you time, not the right to skip testing.
Tip 2: Track Which CVEs Are Covered
Live patches don’t cover every kernel CVE — only the high-severity ones that vendors prioritize. Always cross-reference:
# Ubuntu — check which CVEs are covered by current livepatch
canonical-livepatch status --verbose | grep -i cve
# Check if a specific CVE is addressed
canonical-livepatch status --verbose | grep CVE-2024-XXXXXFor CVEs not covered by live patching, you still need a reboot. Live patching reduces how often that happens, not to zero.
Tip 3: Plan Your Eventual Reboot
Live patches accumulate in memory. At some point — usually during a maintenance window for other reasons — you should reboot to get a clean kernel with all patches baked in. I schedule this quarterly even when nothing is broken.
# Check how long since last reboot
uptime -s
# See how many live patches are currently active
ls /sys/kernel/livepatch/ | wc -lTip 4: Monitor Kernel Version Drift
If you’re running different kernel versions across your fleet, live patch availability varies per version. A monitoring snippet I add to my weekly check:
#!/bin/bash
# Check kernel versions across fleet
for host in "${SERVERS[@]}"; do
echo -n "$host: "
ssh "$host" 'uname -r' 2>/dev/null
done | sort -t: -k2 | uniq -f1 -cSeeing 4 servers on one kernel version and 1 on an older one means that older server might not get live patches — or worse, might miss them entirely because the vendor doesn’t support that version anymore.
Tip 5: Understand the Limits
Kernel live patching handles security vulnerabilities in existing kernel functions. It cannot:
- Add new system calls or kernel interfaces
- Patch drivers that aren’t part of the core kernel
- Fix issues requiring kernel data structure changes
- Replace patches that themselves have been live-patched (you need a reboot to reset)
When a CVE requires structural changes to kernel data types, live patching simply isn’t possible — vendors will tell you a reboot is required, and they’re right.
Tip 6: Set Up Alerting
Add a check to your monitoring stack (Prometheus, Nagios, whatever you use) that alerts if:
- A live patch has been in
transitionstate for more than 2 hours - The livepatch daemon hasn’t checked for updates in 24+ hours
- A new patch is available but not yet applied
Silent failures are the worst kind. I had a livepatch daemon crash quietly once and went two weeks thinking I was protected when I wasn’t.
Choosing the Right Tool for Your Setup
Quick reference for picking your approach:
- Ubuntu (personal/small business): Canonical Livepatch via Ubuntu Pro — free for ≤5 machines, zero configuration
- Ubuntu (enterprise): Ubuntu Pro with paid tier — covers unlimited machines
- RHEL with active subscription: Red Hat’s kpatch service handles everything automatically
- CentOS/AlmaLinux without subscription: Manual kpatch with community patch modules — more work, but doable
- SUSE Enterprise: klp via SCC subscription
- Mixed fleet or commercial option: TuxCare KernelCare — vendor-neutral, supports almost everything
Kernel live patching isn’t a silver bullet, but it’s one of the highest-ROI security tools I’ve added to my workflow. The setup time is under an hour for most configurations, and the payoff — skipping emergency reboots for critical CVEs — is immediate. Start with one non-critical server, verify the patch status tooling works, then roll it out from there.
