Problem Statement: The 2:14 AM Storage Post-Mortem
The pager went off at 2:14 AM. An automated update script had gone rogue across my primary hypervisor, mangling dataset permissions and corrupting a 4 GB SQLite database backing Vaultwarden and Home Assistant. Staring at the terminal, I had two options: spend four painful hours piecing together files from yesterday’s rsync dump, or roll back to a clean block-level snapshot taken 14 minutes earlier.
I chose the snapshot. Thirty seconds later, all services were healthy.
Too many HomeLabbers still rely on fragile shell scripts that wrap zfs snapshot in cron. These homebrew setups break in predictable ways. They leak thousands of orphan snapshots that consume pool metadata. They fail silently when you rename a dataset. Worst of all, they rarely enforce granular retention tiers (hourly, daily, weekly, monthly).
Replicating those snapshots with rsync isn’t much better. Rsync walks the entire directory tree and reads files byte-by-byte through the VFS layer. That thrashes disk I/O, chokes on 500 GB virtual disk images, and fails on open database lockfiles. You need atomic, block-level delta syncs instead.
Core Concepts: Why Sanoid and Syncoid?
ZFS handles Copy-on-Write (CoW) snapshots and serialized data streams natively using zfs send and zfs receive. What it lacks is an orchestrator to schedule snapshots, prune old data, and handle network transfers. That is where Jim Salter’s Sanoid suite fits in.
Sanoid: Policy-Driven Snapshot Lifecycle
Sanoid automates creation and retention without custom bash scripts. You define retention policies in a single config file (/etc/sanoid/sanoid.conf). A systemd timer triggers Sanoid every minute or hour. It creates timestamped snapshots (like autosnap_2026-10-06_02:00:00_hourly) and automatically purges expired ones according to your retention window (for instance, keeping 24 hourly, 7 daily, 4 weekly, and 3 monthly snapshots).
Syncoid: Resilient Stream Replication
Syncoid handles network transport. It wraps zfs send and zfs receive in intelligent discovery logic over SSH.
When pointed at a remote host, Syncoid locates the most recent shared snapshot between the source and target datasets. It then calculates the delta, compresses it (using LZ4 or zstd), buffers the stream with mbuffer to smooth out network spikes, and applies it to the destination pool. Even across an erratic 1 Gbps Wi-Fi or WAN link, Syncoid moves 50 GB of modified database blocks without taking your services offline.
Hands-on Practice: Building the Automated Pipeline
1. Installing Dependencies
Run these commands on both your primary node (source) and your backup storage server (target). On Debian, Ubuntu, or Proxmox VE hosts:
sudo apt update
sudo apt install -y sanoid lzop mbuffer pv openssh-client openssh-server2. Configuring the Sanoid Snapshot Policy
On your primary server, create the configuration directory and file at /etc/sanoid/sanoid.conf:
sudo mkdir -p /etc/sanoid
sudo nano /etc/sanoid/sanoid.confDefine your snapshot retention profiles and assign them to your datasets:
# Template for critical production data
[template_production]
frequently = 0
hourly = 24
daily = 7
weekly = 4
monthly = 3
yearly = 0
autosnap = yes
autoprune = yes
# Template for short-lived temp storage
[template_ephemeral]
frequently = 0
hourly = 6
daily = 1
weekly = 0
monthly = 0
yearly = 0
autosnap = yes
autoprune = yes
# Dataset assignments
[tank/appdata]
use_template = template_production
recursive = yes
[tank/vms]
use_template = template_production
recursive = yes
[tank/scratch]
use_template = template_ephemeral
recursive = no3. Testing and Automating Sanoid via Systemd
Always validate your config with a dry run before enabling automated pruning:
# Validate configuration syntax and check planned actions
sudo sanoid --cron --debug --dry-run
# Enable and start the native systemd timer
sudo systemctl enable --now sanoid.timer
# Confirm the timer is active and scheduled
sudo systemctl list-timers | grep sanoid4. Configuring Dedicated SSH Authentication for Syncoid
Syncoid requires SSH access to stream snapshots. Generate a dedicated Ed25519 keypair on the primary node without a passphrase:
# Generate an SSH keypair on the primary host
sudo ssh-keygen -t ed25519 -f /root/.ssh/id_syncoid -N "" -C "syncoid-backup"
# Copy the public key to the backup target (e.g., 192.168.10.50)
sudo ssh-copy-id -i /root/.ssh/id_syncoid.pub [email protected]
# Verify key-based authentication
sudo ssh -i /root/.ssh/id_syncoid [email protected] "zfs list"5. Executing an Initial Synchronization
Run your baseline replication manually. Syncoid creates the destination dataset automatically and streams the full initial dataset:
sudo syncoid --sshkey=/root/.ssh/id_syncoid \
--no-sync-snap \
--compress=lz4 \
--recursive \
tank/appdata [email protected]:backup-pool/appdataThe --no-sync-snap flag instructs Syncoid to reuse Sanoid’s existing automated snapshots rather than generating ephemeral sync snapshots.
6. Automating Syncoid Replication
Schedule replication to fire shortly after Sanoid completes its hourly snapshot run. Create /etc/cron.d/syncoid-replication on the primary server:
sudo nano /etc/cron.d/syncoid-replicationAdd the cron definition (offset to 15 minutes past the hour):
# Replicate tank/appdata to backup node at minute 15 of every hour
15 * * * * root /usr/sbin/syncoid --sshkey=/root/.ssh/id_syncoid --quiet --no-sync-snap --compress=lz4 --recursive tank/appdata [email protected]:backup-pool/appdata >> /var/log/syncoid.log 2>&17. The Recovery Drill: Restoring Data
A backup is only as good as your last restore test. Here are the two ways to recover when disaster strikes:
# Option A: Instant local rollback (takes < 2 seconds)
sudo zfs rollback tank/appdata@autosnap_2026-10-06_02:00:00_hourly
# Option B: Pull data back from the backup target if local pool is lost
sudo syncoid --sshkey=/root/.ssh/id_syncoid \
[email protected]:backup-pool/appdata tank/appdata_recoveredConclusion: Turning Storage Outages into Routine Fixes
Storage failures and bad update scripts are inevitable. By pairing Sanoid’s scheduled retention policies with Syncoid’s fast delta replication, you remove human error from the equation. Your datasets stay protected locally with granular rollbacks and offsite with bandwidth-efficient block streams—giving your HomeLab true production-grade resilience.
