Six months ago, one of my VPS instances running a Node.js application froze solid at 3 AM. SSH connections timed out. The monitoring dashboard showed a flatline. The only fix was a hard reboot. The culprit? The system had run out of memory, and the Linux OOM killer — the kernel’s built-in last resort — had stalled trying to decide what to kill. By the time it finally acted, the machine was already unresponsive.
Running 10+ Linux VPS instances over three years teaches you that this scenario is more common than anyone admits. I started testing systemd-oomd as a fix, and after six months running it in production on a 2 vCPU / 2 GB RAM DigitalOcean droplet, I can give you a concrete picture of what it actually does.
The Real Problem: Why Your Server Hangs Instead of Recovering
When a Linux server runs out of RAM, the theory is simple: the kernel kills something, memory frees up, life goes on. The reality plays out differently:
- Applications exhaust available RAM
- The system starts heavy swap usage — if swap even exists
- Disk I/O spikes because everything is swapping in and out
- The system becomes so busy swapping it can’t respond to input
- Eventually the OOM killer fires — but by then, the machine is already frozen
That gap between “memory is critically low” and “the OOM killer finally acts” is where servers die. On an SSD-backed VPS I’ve watched this stretch from 30 seconds to over two minutes, depending on swap size and disk throughput. During that window, SSH won’t respond, cron jobs stall, and web traffic flatlines. You end up rebooting anyway.
Root Cause Analysis: The OOM Killer’s Achilles Heel
Reactive by design — that’s the kernel OOM killer in a phrase. It only triggers after memory pressure reaches a true crisis: specifically, after the kernel fails to allocate memory for a new request. By that point, the system is so bogged down in swap I/O that it can barely execute the OOM killer code itself.
Targeting is the other problem. The OOM killer scores each process using a heuristic based on memory usage, runtime, and other factors. It’s not predictable. On one of my instances, it killed a PostgreSQL process while a runaway cron job consuming 600 MB survived — the exact opposite of what I wanted.
Check what pressure your system is currently under:
cat /proc/meminfo | grep -E "MemAvailable|SwapFree|MemTotal"
vmstat -s | head -20
dmesg | grep -i "oom\|kill"Already seeing OOM kills in dmesg? The kernel is reacting in full crisis mode. That’s the gap systemd-oomd is built to close.
Solutions Compared: Three Ways to Handle Memory Pressure
Option 1: Tune the Kernel OOM Killer Directly
You can adjust oom_score_adj per-process to influence what gets killed first:
# Make a process less likely to be killed (-1000 = immune)
echo -500 > /proc/$(pidof your-critical-service)/oom_score_adj
# Make a process more likely to be killed (1000 = first target)
echo 1000 > /proc/$(pidof disposable-worker)/oom_score_adjBetter targeting, yes. But timing doesn’t change. The OOM killer still fires too late, after the system is already struggling to stay responsive.
Option 2: earlyoom
earlyoom is a userspace daemon that monitors memory usage and kills processes before the kernel’s OOM killer would. Simple, effective, and battle-tested for years:
sudo apt install earlyoom
sudo systemctl enable --now earlyoomWorks well for single-machine setups. Two downsides worth knowing: it’s a third-party dependency, and it doesn’t understand systemd’s cgroup hierarchy. It kills individual processes, not entire service units. If your app forks worker processes, killing one worker doesn’t stop the memory bleed — the rest keep running.
Option 3: systemd-oomd
This is what I’ve settled on after six months. systemd-oomd ships with systemd (available since version 247), integrates natively with cgroups v2, and kills entire service units rather than individual processes. That service-level kill is the key difference. When a service misbehaves, you want the whole thing gone — not one worker while four others keep eating memory.
It also monitors memory pressure using PSI (Pressure Stall Information), which measures how much time the system spends stalled waiting for memory. PSI gives earlier warning than raw memory counts. oomd can act while the system is still responsive enough to actually do something useful.
Configuring systemd-oomd: A Production Setup
Check Prerequisites
systemd-oomd requires cgroups v2 and PSI support. Verify both before doing anything else:
# Confirm cgroups v2 is mounted
mount | grep cgroup2
# Expected output: cgroup2 on /sys/fs/cgroup type cgroup2 ...
# Confirm PSI is available
cat /proc/pressure/memory
# If this file exists, PSI is enabledcgroups v2 is the default on Fedora 31+, Ubuntu 21.10+, and Debian 11+. On older systems, add systemd.unified_cgroup_hierarchy=1 to your kernel command line in /etc/default/grub, then run update-grub and reboot.
Install and Enable
# Ubuntu 22.04+ / Debian 12+
sudo apt install systemd-oomd
# Fedora / RHEL 9+
sudo dnf install systemd-oomd
# Enable and start
sudo systemctl enable --now systemd-oomd
# Verify it's running
systemctl status systemd-oomdConfigure /etc/systemd/oomd.conf
This is the configuration I’ve been running in production. Conservative thresholds — I’d rather oomd act slightly early than sit idle while the system freezes:
[OOM]
# Trigger monitoring when swap is 80% used (20% remaining)
SwapUsedLimit=80%
# Kill a cgroup when memory pressure exceeds 60% for 30 seconds
DefaultMemoryPressureLimit=60%
# 30s window avoids false positives from brief spikes
DefaultMemoryPressureDurationSec=30sAfter editing, reload:
sudo systemctl daemon-reload
sudo systemctl restart systemd-oomdMark Services as OOM Candidates
systemd-oomd only acts on cgroups that explicitly opt in. For each service you’re willing to sacrifice under memory pressure, create a systemd override:
sudo systemctl edit your-background-service.serviceAdd this to the override file:
[Service]
# Let oomd kill this service if swap is critically high
ManagedOOMSwap=kill
# Let oomd kill this service under sustained memory pressure
ManagedOOMMemoryPressure=kill
# Override the global pressure threshold for this service (optional)
# Lower = more sensitive; useful for expendable background jobs
ManagedOOMMemoryPressureLimit=40%For user sessions and interactive workloads, configure the user slice to be killable as a group:
sudo systemctl edit user.slice[Slice]
ManagedOOMSwap=kill
ManagedOOMMemoryPressure=killProtect Critical Services
For services you never want oomd touching — your database, the SSH daemon, your monitoring agent — explicitly set them to auto. That leaves any kill decision entirely to the kernel’s OOM killer as a last resort:
sudo systemctl edit postgresql.service[Service]
ManagedOOMSwap=auto
ManagedOOMMemoryPressure=autoVerify What oomd Is Watching
# Show all cgroups currently monitored by oomd
oomctl
# Follow oomd activity in real time
journalctl -u systemd-oomd -f
# Check current PSI values directly
cat /proc/pressure/memory
# Output: some avg10=0.03 avg60=0.15 avg300=0.08 total=12345678
# avg10 above 60% sustained for 30s triggers a kill with the config aboveoomctl shows which cgroups are candidates and their current pressure readings. If a service you configured doesn’t appear, check two things: confirm cgroups v2 is active for that unit, and verify the override applied correctly with systemctl cat your-service.
Testing Before You Trust It
Two weeks on a staging VPS before any production deployment — that’s the rule I follow. I ran stress-ng against a clone of the production setup to simulate sustained memory pressure before touching live systems. You can do the same:
# Install stress testing tool
sudo apt install stress-ng
# Simulate heavy memory pressure for 60 seconds
stress-ng --vm 2 --vm-bytes 80% --timeout 60sRun journalctl -u systemd-oomd -f in a separate terminal while this fires. If oomd is configured correctly and your target services are opted in, you’ll see it identify candidates and kill them — and your SSH session stays alive throughout. That last part is what matters. It’s the difference between a system that absorbs OOM events and one that just dies more noisily.
Six Months In: The Honest Assessment
Two OOM events hit my production instances since deploying oomd. Both times, it killed the background job queue within seconds of pressure crossing the threshold. Memory freed up, the primary application kept serving traffic, and neither incident required a reboot. Pre-oomd, both would have been 3 AM hard-reboot calls.
PSI-based monitoring catches pressure early — usually while the system still has enough headroom to execute kills cleanly. My 30-second window filters out brief spikes without letting sustained pressure accumulate unnoticed. In six months, oomd has never triggered on a spike that wasn’t a real problem.
Most of the work happens upfront: mapping which services are expendable and which need protection. Worth doing regardless of oomd — it forces you to think explicitly about service priorities on each host. After that, oomd disappears into the background. You stop worrying about 3 AM freeze calls. That’s the outcome worth optimizing for.
