We’ve all been there — a service crashes at 2 AM, the process vanishes, and the only clue is “Segmentation fault” buried in syslog. Debugging it the next morning feels like a forensic investigation with half the evidence missing. Getting core dumps right makes that investigation actually possible.
The tricky part is that Linux gives you several ways to handle core dumps — and they behave very differently once you’re in production. What works in dev often silently fails the moment systemd enters the picture. Let me share what actually works, based on real server experience.
Approach Comparison: Three Ways to Handle Core Dumps on Linux
1. Traditional ulimit + kernel.core_pattern
The classic approach predates systemd by decades. You set the core file size limit and tell the kernel where to write dumps:
# Check current core dump settings
ulimit -c
# Enable unlimited core dump size (current session only)
ulimit -c unlimited
# Set core dump location and filename pattern
echo '/tmp/cores/core.%e.%p.%t' | sudo tee /proc/sys/kernel/core_pattern
# Make it persistent across reboots
echo 'kernel.core_pattern = /tmp/cores/core.%e.%p.%t' | sudo tee -a /etc/sysctl.conf
sudo sysctl -p
# Ensure the directory exists and is writable
sudo mkdir -p /tmp/cores
sudo chmod 1777 /tmp/coresFor services managed by systemd, the per-session ulimit doesn’t carry over. You also need this inside the unit file:
# /etc/systemd/system/myapp.service.d/override.conf
[Service]
LimitCORE=infinity2. systemd-coredump (The Modern Approach)
When systemd is your init system — which it is on any modern Ubuntu, Debian, RHEL, or Fedora — systemd-coredump hooks directly into the kernel’s dump mechanism. It captures dumps automatically, stores them compressed in /var/lib/systemd/coredump/, and indexes them via the journal. Query them instantly with coredumpctl.
# Verify the kernel is routing dumps through systemd
cat /proc/sys/kernel/core_pattern
# Expected output:
# |/lib/systemd/systemd-coredump %P %u %g %s %t %c %h3. ABRT (Automatic Bug Reporting Tool)
ABRT is a Red Hat invention that ships by default on Fedora and RHEL. It watches for crashes, captures them, and optionally sends bug reports upstream to Bugzilla. It runs multiple daemons and has a GUI component — useful for desktop workstations, but heavy for a production server. On Ubuntu-based systems or minimal installs, the overhead rarely justifies it.
Pros and Cons of Each Approach
Traditional ulimit
- Pros: Works everywhere with zero extra dependencies; full control over file location; no background daemons
- Cons: Core files can fill up disk fast without active cleanup; no automatic compression or metadata; easy to miss per-service limits in systemd units; finding the right core file after the fact requires manual hunting
systemd-coredump
- Pros: Automatic zstd compression (expect 3–5× ratio on heap dumps); rich metadata stored in the journal (PID, signal, executable path, timestamp, cgroup);
coredumpctlmakes searching and extracting trivial; integrates withjournalctlfor cross-referencing logs - Cons: Requires systemd (a non-issue on modern distros); dumps accumulate in
/var/lib/systemd/coredump/— you need adequate disk there; slightly more overhead at crash time due to compression
ABRT
- Pros: Automated upstream reporting; good GUI integration for workstation use
- Cons: Heavy resource footprint; sends data externally by default (privacy concern); unavailable on Debian/Ubuntu without significant manual setup; overkill for server post-mortem debugging
Recommended Setup: Why systemd-coredump Wins
For any production Linux server running systemd, systemd-coredump is the right choice. On my Ubuntu 22.04 box with 4GB RAM, the difference became obvious after the first serious incident. Previously, tracking down a core file meant running find / -name 'core.*' and hoping nothing had already clobbered it. Now coredumpctl list shows every crash — binary path, signal, timestamp — in under a second.
The zstd compression is a real bonus. A 400MB heap dump compresses to around 120MB, so repeated crashes during a bad deploy don’t eat your disk alive. Ten lines of config. Then coredumpctl plus GDB handle everything from quick triage to unwinding a 50-frame call stack.
Implementation Guide
Step 1: Verify systemd-coredump is Active
On Ubuntu 20.04+ and most modern systemd-based distros, systemd-coredump is installed by default. Double-check:
# Ubuntu/Debian
sudo apt install systemd-coredump
# RHEL/Fedora
sudo dnf install systemd
# Confirm kernel is routing dumps through systemd
cat /proc/sys/kernel/core_pattern
# Should show: |/lib/systemd/systemd-coredump %P %u %g %s %t %c %hStep 2: Tune coredump.conf
Edit /etc/systemd/coredump.conf to match your server’s memory profile:
[Coredump]
# Store as compressed files on disk (not embedded in journal)
Storage=external
# Use zstd compression
Compress=yes
# Max size per core dump — tune to your largest process RSS
ProcessSizeMax=2G
# Total disk budget for all stored cores
ExternalSizeMax=10G
MaxUse=5G
KeepFree=1G# Apply the config
sudo systemctl daemon-reloadOn a 4GB RAM server, ProcessSizeMax=2G captures the full heap of any runaway process. It also leaves enough headroom that the dump itself won’t trigger an OOM kill.
Step 3: Trigger a Test Crash
Write a small C program with a deliberate null pointer dereference:
// crash_test.c
#include <stdio.h>
#include <stdlib.h>
void broken_function() {
int *ptr = NULL;
*ptr = 42; // Segfault here
}
int main() {
printf("About to crash...\n");
broken_function();
return 0;
}# Compile WITH debug symbols — critical for readable GDB output
gcc -g -o crash_test crash_test.c
# Run it
./crash_test
# Output: About to crash...
# Segmentation fault (core dumped)Step 4: Find and Extract the Core
# List all captured core dumps
coredumpctl list
# Sample output:
# TIME PID UID GID SIG COREFILE EXE
# Tue 2026-06-03 14:22:11 JST 4821 1000 1000 SIGSEGV present /home/user/crash_test
# Detailed metadata for the latest crash
coredumpctl info
# Extract core file for GDB analysis
coredumpctl dump -o ./core_crash_testStep 5: Analyze with GDB
Load GDB with the binary and the extracted core:
gdb ./crash_test ./core_crash_test
# Or let coredumpctl handle it directly:
coredumpctl gdbInside GDB, these are the commands that do the actual work:
# Full call stack at crash time
(gdb) bt
(gdb) bt full # includes local variables per frame
# Navigate to a specific stack frame
(gdb) frame 1
# Inspect variables in current frame
(gdb) info locals
# Print a specific variable
(gdb) print ptr
# Show CPU registers at crash point
(gdb) info registers
# Show source code around the crash
(gdb) list
# Disassemble (useful for stripped binaries)
(gdb) disassembleFor the test program above, bt points straight to broken_function with the exact line number and a null pointer in ptr. That’s the full picture — binary, crash location, variable state — reconstructed hours after the fact.
Debugging Production Binaries Without Debug Symbols
Real production binaries are usually stripped. Install the debuginfo package if available:
# Ubuntu/Debian — enable debug symbol repos
sudo apt install ubuntu-dbgsym-keyring
echo "deb http://ddebs.ubuntu.com $(lsb_release -cs) main restricted universe multiverse" | \
sudo tee /etc/apt/sources.list.d/ddebs.list
sudo apt update
# Install debug symbols for a specific package (e.g., nginx)
sudo apt install nginx-dbgsymFor your own compiled apps, keep a separate debug symbol file alongside your stripped binary:
# Build with symbols, then strip for deployment while keeping the debug info
objcopy --only-keep-debug myapp myapp.debug
strip --strip-debug myapp
# In GDB, load the symbols manually:
(gdb) symbol-file /path/to/myapp.debugBuilding This Into Your Incident Runbook
Once systemd-coredump is configured, these steps take 2 minutes and should be the first thing you do after any crash incident:
- Run
coredumpctl list --since "2 hours ago"to spot recent crashes - Cross-reference timestamps with
journalctl -u yourservice --since "..." - Archive the core before it gets rotated:
coredumpctl dump PID -o /var/incident-cores/$(date +%Y%m%d_%H%M%S).core - For your own compiled services, always store the unstripped binary or
.debugfile in a versioned location tied to the build
A crash without a core dump is a mystery. A crash with one is just a debugging session. Set this up once — next time the 2 AM alert fires, you’ll have the full picture waiting.
