Configuring Watchdog Daemon on Linux: Auto-Detect System Freezes and Reboot for High Availability

Quick Start (5 Min)

When a remote Linux server completely locks up, you usually have to log into an IPMI console, access a cloud hypervisor, or physically hit the reset button. The Linux watchdog daemon bypasses that manual hassle. It continuously monitors operating system health and commands an immediate reboot if the kernel hangs or critical resources run dry.

You can set up a basic software watchdog on Debian or Ubuntu in under five minutes using the softdog kernel module and the standard watchdog package.

First, load the kernel module and configure it to persist across reboots:

sudo modprobe softdog
echo "softdog" | sudo tee -a /etc/modules-load.d/watchdog.conf
ls -l /dev/watchdog

Confirm that the character device /dev/watchdog exists. Next, install the user-space daemon:

sudo apt update
sudo apt install -y watchdog

Open /etc/watchdog.conf in your editor of choice:

sudo nano /etc/watchdog.conf

Uncomment or append these basic parameters to establish heartbeat intervals and CPU load thresholds:

watchdog-device = /dev/watchdog
watchdog-timeout = 60
interval = 10
max-load-1 = 24

Save the file. Then enable and start the service:

sudo systemctl enable --now watchdog
sudo systemctl status watchdog

Your watchdog is now armed. If the kernel freezes or load averages spike past your threshold (such as 24 on a 16-core server), the watchdog timer expires and forces an immediate reboot after 60 seconds.

Deep Dive: How the Linux Watchdog Daemon Works

Understanding the interplay between the kernel-level timer and the user-space daemon prevents accidental reboots and configuration mistakes.

The Heartbeat Mechanism

Everything centers on a hardware or kernel countdown timer. When initialized, the timer counts down toward zero from a defined window, such as 60 seconds. During normal operation, the user-space daemon writes periodic keepalive bytes to /dev/watchdog. System administrators refer to this refresh as “kicking the dog” or sending a heartbeat.

What happens if the operating system stalls? If an out-of-memory lockup halts the scheduler or a driver causes a kernel panic, the daemon misses its check-in. The timer hits zero, and the underlying watchdog hardware forces an immediate system reset.

Hardware vs. Software Watchdogs

Choose your watchdog driver based on your hardware architecture and reliability requirements:

• Software Watchdog (softdog): A pure kernel timer module. It works inside virtual environments like KVM, Proxmox, and AWS EC2 instances without special hardware pass-through. However, because it relies on kernel timer interrupts, a catastrophic CPU lockup or hypervisor halt will prevent softdog from triggering a reset.
• Hardware Watchdog (IPMI / iTCO_wdt / bcm2835_wdt): Physical servers (such as Dell PowerEdge with iDRAC or HPE ProLiant with iLO) and single-board computers like the Raspberry Pi 4 include dedicated timer microcontrollers on the motherboard. Because this clock operates independently of the host CPU and OS, it guarantees a hardware reset even if the kernel halts completely.

Run this command to check whether your server motherboard exposes a hardware watchdog:

dmesg | grep -i watchdog

If you see drivers like iTCO_wdt (Intel chipsets) or ipmi_watchdog, point /etc/watchdog.conf directly to that device node instead of loading softdog.

The “Magic Close” Feature

What prevents an unexpected reboot when you stop the service for maintenance? The daemon implements a safety protocol known as “Magic Close.” When stopped gracefully via systemctl stop watchdog, the daemon writes the ASCII character V to /dev/watchdog right before closing the file handle. The kernel driver recognizes this deliberate shutdown code and safely disarms the countdown timer.

Advanced Usage: Custom Health Checks and Network Ping

Beyond checking whether the CPU scheduler responds, the watchdog daemon can monitor free memory, network gateways, critical daemon PIDs, and custom application endpoints.

Configuring Resource and Network Thresholds

Here is an battle-tested /etc/watchdog.conf configuration for an internet-facing node:

# Watchdog device and heartbeat pacing
watchdog-device = /dev/watchdog
interval = 5
watchdog-timeout = 30

# Reboot if free RAM falls below 128MB
min-memory = 128M

# Trigger a reboot if 1-minute load exceeds 30
max-load-1 = 30

# Ping the local gateway to ensure network viability
ping = 192.168.1.1
interface = eth0

# Verify the web server process PID remains alive
pidfile = /var/run/nginx.pid

If your local gateway stops answering ICMP packets for too long, or a memory leak consumes available RAM down to less than 128MB, the watchdog flags the condition and reboots the host.

Writing Custom Test Scripts

Predefined directives do not cover every edge case. Fortunately, you can drop custom shell scripts into /etc/watchdog.d/. The daemon executes every script in this directory on every heartbeat cycle. Any exit status between 1 and 244 signals a check failure.

For instance, you can verify that an internal microservice on port 8080 responds with an HTTP 200 within three seconds:

sudo mkdir -p /etc/watchdog.d
sudo nano /etc/watchdog.d/check_api.sh

Paste the following script:

#!/usr/bin/env bash

# Check internal API health with a 3-second timeout
HTTP_STATUS=$(curl -s -o /dev/null -w "%{http_code}" --max-time 3 http://127.0.0.1:8080/health)

if [ "$HTTP_STATUS" -eq 200 ]; then
    exit 0
fi

exit 1

Make the script executable:

sudo chmod +x /etc/watchdog.d/check_api.sh

Next, enable test directory polling inside /etc/watchdog.conf:

test-directory = /etc/watchdog.d
test-timeout = 10

Reload the service to activate the health script:

sudo systemctl restart watchdog

Practical Tips for Production Systems

Aggressive watchdog settings can easily trap your servers in an endless reboot cycle. Apply these practical guidelines to keep your recovery automation safe.

Tune Intervals and Keep Scripts Lean

Keep every health-check script brief. If a probe stalls waiting for an upstream DNS query or database lock, it consumes execution time from the watchdog loop itself. Avoid complex database queries or nested subshells inside /etc/watchdog.d/.

Always set watchdog-timeout to at least two or three times your interval. If your heartbeat interval is 10 seconds, set the watchdog timeout to 30 or 60 seconds. This buffer gives the kernel room to process heavy temporary I/O spikes without misinterpreting a brief delay as a terminal freeze.

Simulating a System Lockup for Verification

Always test your configuration in staging before deploying it to production fleets. You can safely simulate an unrecoverable kernel panic using Linux Magic SysRq keys. First, ensure SysRq triggers are enabled in the kernel:

sudo sysctl -w kernel.sysrq=1

Attach to the server’s serial or out-of-band console. Then issue an intentional kernel crash:

echo c | sudo tee /proc/sysrq-trigger

The SSH session will freeze immediately. Watch your out-of-band management console: as soon as the configured watchdog-timeout window elapses, the hardware timer will fire, cycle system power, and return the host to a clean login prompt.

Preventing Boot Loops and Filesystem Corruption

Remember that a watchdog-triggered reset is an abrupt hardware reset, not a graceful shutdown -r now. Databases like PostgreSQL, Redis, or MySQL can take several minutes to run crash recovery on startup. If your watchdog monitors MySQL immediately on boot with a low timeout, it might trigger another reboot while the database is still replaying its WAL logs.

To avoid boot loops, use systemd dependencies (such as After=network.target) and configure reasonable start timeouts on critical application units. That buffer guarantees that services finish crash recovery before the watchdog holds them accountable.