Real-World Problem: The Midnight Disk Space Outage
It is 2:00 AM on a Sunday when pager alerts start firing: your main production API has stopped responding. You SSH into the rescue console, but bash throws an immediate error: No space left on device. PostgreSQL halts write transactions, Redis crashes, and system monitoring goes dark. A quick check with df -h reveals the culprit: a single application log at /var/log/app/output.log grew to 85GB over 72 hours, consuming 100% of the root partition.
Unmanaged log growth quietly takes down production clusters every day. Web servers, background workers, and container runtimes generate millions of log lines per hour. Without an automated rotation policy, these files will fill your drives and exhaust filesystem inodes.
Root Cause Analysis: How Unchecked Logs Exhaust Storage and Inodes
Linux processes write to files using active file descriptors. When a daemon writes continuously, storage degrades in a few distinct ways:
- Block Storage Exhaustion: Verbose debug traces and brute-force web attacks can dump 500MB of plain text in under an hour. Raw text wastes massive amounts of block storage compared to compressed gzip files, which shrink log sizes by 85% to 90%.
- File Descriptor Locking: Running
rm /var/log/app/output.logon an active log file does not free disk space. The kernel keeps the unlinked inode alive as long as the application process holds an open file handle. Storage stays locked at 100% until you restart the daemon or truncate the handle manually through/proc. - Permission and Inode Drift: Ad-hoc rotation scripts that move files often create replacements with
root:rootownership instead of application users likewww-data. The application crashes immediately on the next write attempt due to permission denial.
Solutions Compared: Manual Truncation, Custom Shell Scripts, vs. Logrotate
Engineers generally handle log growth in one of three ways:
| Method | Pros | Cons |
|---|---|---|
Manual Truncation (> file.log) |
Zero downtime; frees disk blocks instantly without killing processes. | Completely manual. Erases all historical data and does nothing to prevent the next outage. |
| Custom Cron Shell Scripts | Flexible pipeline using find -mtime, gzip, and custom S3 upload hooks. |
Brittle. Fails to handle atomic log file swapping, signal delivery, or state recovery when cron jobs overlap. |
| Native Logrotate Utility | Pre-installed across Debian, Ubuntu, RHEL, and Rocky Linux. Handles permissions, signal reloads, age-based pruning, and compression out of the box. | Requires correct directive syntax and an understanding of how your target service handles file descriptors. |
Ad-hoc shell scripts tend to work fine until an edge case breaks them. The native logrotate tool provides a battle-tested, declarative system that eliminates custom maintenance overhead.
The Best Approach: Implementing Automated Log Rotation with Logrotate
Most Linux distributions execute logrotate once per day via a systemd timer (logrotate.timer) or a cron script in /etc/cron.daily/logrotate. Logrotate loads baseline defaults from /etc/logrotate.conf and reads modular rules from individual files inside /etc/logrotate.d/.
Step 1: Anatomy of a Production-Ready Logrotate Configuration
Keep your configurations modular. Instead of touching global defaults, create a dedicated file for your service inside /etc/logrotate.d/:
sudo nano /etc/logrotate.d/my-appAdd the following configuration block:
/var/log/my-app/*.log {
daily
missingok
rotate 14
compress
delaycompress
notifempty
create 0640 myappuser myappgroup
sharedscripts
postrotate
/usr/bin/systemctl reload my-app.service > /dev/null 2>&1 || true
endscript
}Here is what these directives actually do behind the scenes:
daily: Checks log status every 24 hours. You can switch toweekly,monthly, or enforce file size thresholds likesize 250M.missingok: Silences errors if log files have not been generated yet.rotate 14: Keeps 14 rotated archives before deleting old logs. This maintains a clean two-week historical audit trail.compress&delaycompress: Compresses archives with gzip.delaycompresswaits one cycle before compressing the most recent archive (app.log.1), ensuring running processes have finished flushing pending write buffers.notifempty: Skips rotation entirely if a file is 0 bytes, avoiding clutter from empty archive files.create 0640 myappuser myappgroup: Generates a fresh log file with strict permission bits and assigned application ownership right after rotation.
Step 2: Managing Permissions and Size Triggers
Security-hardened environments often restrict directory access to dedicated service accounts. If logrotate runs as root against a directory owned by a non-privileged user, it can refuse to operate. Add the su directive to enforce the exact operational context:
/var/log/my-app/*.log {
su myappuser myappgroup
daily
rotate 7
maxsize 500M
compress
create 0644 myappuser myappgroup
}The maxsize 500M directive acts as a safety valve. If an application suddenly loops and writes 2GB of errors before the daily rotation schedule runs, logrotate forces an early rotation as soon as the file hits 500MB.
Step 3: Preventing Service Disruptions with copytruncate vs. Signals
When logrotate renames an active log file, the running daemon keeps writing to that old inode until it receives instructions to reopen the file path. You have two options depending on how your service handles logging:
Option A: Application Supports Log Reopening via Signals (Recommended)
Production software like Nginx, Apache, and HAProxy can reload log file handles on the fly without dropping active connections. Send a POSIX signal like USR1 inside a postrotate block:
/var/log/nginx/*.log {
daily
rotate 30
compress
delaycompress
sharedscripts
postrotate
[ -f /run/nginx.pid ] && kill -USR1 $(cat /run/nginx.pid)
endscript
}Option B: Application Does Not Support Signals (Node.js, Python, or Go Binaries)
Many custom backend services do not implement signal handlers for file rotation. Use copytruncate in these cases. Logrotate copies the live content to a secondary file and then truncates the original file in place, keeping the open file descriptor intact:
/var/log/legacy-app/*.log {
daily
rotate 7
copytruncate
compress
missingok
}Keep in mind that copytruncate has a tiny race condition: any logs written during the split-second between copy and truncation can be lost. For high-volume transaction APIs, configure proper signal handling or route logs to journald instead.
Step 4: Testing and Validating with Dry Runs
Never wait for a midnight cron job to catch a broken configuration. Validate syntax and test your rules immediately with dry-run flags:
# Perform a dry run to inspect planned actions without modifying files
sudo logrotate -d /etc/logrotate.d/my-app
# Force an immediate rotation to test script execution and file permissions
sudo logrotate -vf /etc/logrotate.d/my-appCheck the target directory to verify the resulting permissions, file ownership, and compression state:
ls -la /var/log/my-app/A properly configured logrotate routine keeps disk usage predictable, preserves critical forensic history, and protects your production servers from sudden midnight outages.
