When a Kernel or Package Upgrade Goes Wrong
Few things derail a workday faster than a kernel update that bricks the bootloader or a shared library upgrade that crashes critical daemons. Standard disaster recovery tools—rsync mirrors, disk clone images, or tar archives—take too long to pull and restore. A 100 GB system restore over a gigabit LAN can burn through 20 to 30 minutes of downtime before you even start troubleshooting.
You need a reliable, filesystem-level mechanism. It should freeze known-good states instantly and let you rewind breaking changes in seconds without eating up extra disk space.
How Btrfs and Snapper Handle State Protection
Btrfs (B-tree File System) is built directly into the mainline Linux kernel. It uses a Copy-on-Write (CoW) design that handles storage differently than classic filesystems like ext4.
1. Subvolumes vs. Standard Directories
A Btrfs subvolume looks like a normal directory, but the kernel treats it as an independent filesystem root. Each subvolume has its own inode tree. You can mount them with distinct mount options, set per-subvolume storage quotas, or snapshot them individually.
2. Copy-on-Write (CoW) Snapshots
Creating a Btrfs snapshot takes under 5 milliseconds. Because of CoW architecture, a fresh snapshot does not duplicate file blocks. It simply duplicates the root metadata tree pointing to existing data blocks. When your system writes a modified 4 KB file block, Btrfs writes the new block to free space and updates the live pointers. The snapshot retains pointers to the original block. A newly created snapshot consumes almost zero additional storage.
3. Automated Snapshot Management with Snapper
Manual snapshot scripts run via cron quickly become unmanageable. Snapper—originally built by openSUSE—runs as a system daemon. It handles timeline snapshots, pre- and post-upgrade hooks for package managers, and automatic pruning policies so your drives do not run out of space.
Step-by-Step Setup: Subvolumes, Snapper, and Rollbacks
Step 1: Partitioning and Subvolume Layout
Format your target disk partition (such as /dev/nvme0n1p2) with Btrfs:
# Format the partition
sudo mkfs.btrfs -L sys_storage -f /dev/nvme0n1p2
# Mount the top-level root (subvolume id=5)
sudo mount /dev/nvme0n1p2 /mnt
# Create clean subvolumes for system and user data
sudo btrfs subvolume create /mnt/@
sudo btrfs subvolume create /mnt/@home
sudo btrfs subvolume create /mnt/@var_log
sudo btrfs subvolume create /mnt/@snapshots
# Unmount the temporary mount point
sudo umount /mntMount these subvolumes in /etc/fstab with performance-oriented options:
# Entries in /etc/fstab
LABEL=sys_storage / btrfs subvol=@,defaults,noatime,compress=zstd:1 0 0
LABEL=sys_storage /home btrfs subvol=@home,defaults,noatime,compress=zstd:1 0 0
LABEL=sys_storage /var/log btrfs subvol=@var_log,defaults,noatime 0 0
LABEL=sys_storage /.snapshots btrfs subvol=@snapshots,defaults,noatime,compress=zstd:1 0 0Step 2: Installing and Configuring Snapper
Install Snapper packages across your distribution:
# On Ubuntu/Debian systems
sudo apt update && sudo apt install -y snapper
# On Rocky Linux, AlmaLinux, or Fedora
sudo dnf install -y snapperGenerate a baseline configuration profile for the root filesystem:
# Initialize root profile
sudo snapper -c root create-config /
# Check active configurations
sudo snapper list-configsOpen /etc/snapper/configs/root and fine-tune your retention policies to keep disk usage predictable:
# Retention limits in /etc/snapper/configs/root
TIMELINE_MIN_AGE="1800"
TIMELINE_LIMIT_HOURLY="12"
TIMELINE_LIMIT_DAILY="7"
TIMELINE_LIMIT_WEEKLY="4"
TIMELINE_LIMIT_MONTHLY="2"
TIMELINE_LIMIT_YEARLY="0"Enable Snapper’s systemd automation timers:
sudo systemctl enable --now snapper-timeline.timer
sudo systemctl enable --now snapper-cleanup.timerStep 3: Setting Up Pre/Post Package Manager Hooks
Snapper can automatically create snapshot pairs before and after package changes. When using DNF on Fedora/RHEL, snapper registers its plugin automatically. On Debian and Ubuntu, configure a hook script or install libpam-snapper:
# Verify existing snapshot history
sudo snapper -c root listCompare system changes before and after an update. If snapshot 18 was created before an upgrade and 19 after, you can inspect modified files or view unified diffs directly:
# Show modified, added, or deleted files between snapshots 18 and 19
sudo snapper -c root status 18..19
# View the exact text changes in a modified configuration file
sudo snapper -c root diff 18..19 /etc/nginx/sites-available/defaultStep 4: Performing a System Rollback
If an upgrade breaks a service like OpenSSL or causes a kernel panic, revert to the last working snapshot.
Identify your target snapshot ID from the list:
sudo snapper -c root listOn openSUSE layouts or systems mounting via default subvolume IDs, run:
# Rollback root to healthy snapshot #18
sudo snapper -c root rollback 18
sudo rebootSnapper creates a backup of the broken state, generates a writable snapshot from snapshot 18, and designates it as the new default root subvolume.
Disaster Recovery via Live USB
If you use explicit subvol=@ mounts in /etc/fstab and cannot reach a shell, boot from any Linux live environment to swap the subvolume manually:
# Mount the top-level filesystem
sudo mount -o subvolid=5 /dev/nvme0n1p2 /mnt
# Archive the broken root subvolume
sudo mv /mnt/@ /mnt/@_corrupted_$(date +%F)
# Create a new writable subvolume from snapshot #18
sudo btrfs subvolume snapshot /mnt/@snapshots/18/snapshot /mnt/@
# Clean up and restart
sudo umount /mnt
sudo rebootProduction Best Practices
- Exclude write-heavy paths: Always isolate high-write directories—such as
/var/lib/docker,/var/lib/containers, and/var/lib/mysql—into separate subvolumes. Disable CoW on databases (chattr +C /path) to avoid severe B-tree fragmentation. - Monitor metadata and space margins: Btrfs requires free space to allocate metadata for new writes. An out-of-space error on Btrfs can make snapshot deletion difficult. Keep your volume utilization below 80%.
- Schedule periodic scrubs: Run
btrfs scrub start /once a month. Scrubs scan data and parity blocks, verifying SHA256/CRC32c checksums to catch and repair silent bit rot before it corrupts production data.
