Beyond Basic Logs: Why History and Auth.log Aren’t Enough
If you’ve ever tried to piece together a security incident using only the history command, you know how frustrating it is. Basic shell history is fragile. A smart user can clear it with a single command, and it never shows you the actual output of what they ran. Even /var/log/auth.log only tells you that a command was executed, not what happened inside the session.
Many sysadmins try to use the script command to record activity. However, that requires the user to voluntarily start the recording. Sudo I/O Logging fixes this by moving the recording mechanism into the sudo utility itself. It acts like a DVR for your terminal, capturing every keystroke, every backspace, and every line of output with precise timing.
Let’s look at how the three most common tracking methods compare:
- Shell History: Easily deleted (
history -c), records no output, and offers zero security value. - Auditd: Great for tracking file access and system calls. However, trying to reconstruct a visual session from raw audit logs is like trying to watch a movie by reading the raw metadata of the video file.
- Sudo I/O Logging: Records the full TTY interaction. It is nearly impossible for a non-root user to bypass and provides a perfect visual replay of the session.
The Trade-offs of Session Recording
Implementing session recording isn’t a “set it and forget it” task. I have deployed this in environments ranging from five-node startups to thousand-server data centers, and the impact depends heavily on your workload.
The Advantages
- Absolute Accountability: You don’t just see that a user ran
rm -rf /; you see the exact sequence of commands and mistakes that led to that moment. - Faster Troubleshooting: When a configuration change breaks a production service, you can watch the session to see the exact syntax used, including error messages the admin might have missed.
- Compliance Readiness: This setup satisfies strict requirements for monitoring privileged access under frameworks like PCI-DSS, SOC2, and HIPAA.
The Disadvantages
- Storage Growth: Because you are recording all output, logs can balloon. Running
caton a 500MB log file will create a 500MB session recording. - Sensitive Data Exposure: Everything is captured. If a user types a password into a prompt that fails to mask input, that password is now stored in plain text within your audit logs.
- Performance Overhead: Writing every byte of I/O to disk adds a slight delay. In my testing on an Ubuntu 22.04 instance with 4GB RAM, the CPU overhead remained under 1%, which is significantly lighter than third-party agents.
Recommended Setup Architecture
While you can stream these logs to a central server, we will start with a solid local configuration. We want to ensure that every sudo command is automatically recorded to a protected directory. To keep things organized, we will categorize logs by username and session ID.
The configuration relies on three core parameters:
iolog_dir: The base path where recordings are stored.iolog_file: The naming convention for each session.LOG_OUTPUTandLOG_INPUT: The streams that define what we capture.
Implementation Guide: Setting Up Sudo I/O Logging
This guide assumes you are using a modern distribution like Ubuntu, Debian, or RHEL. You need Sudo version 1.8.7 or higher, which has been standard for nearly a decade.
Step 1: Create the Secure Log Directory
We need a dedicated space for recordings. This directory must be owned strictly by root to prevent users from tampering with their own session history.
sudo mkdir -p /var/log/sudo-io
sudo chmod 0700 /var/log/sudo-ioStep 2: Configure Sudoers Safely
Never edit /etc/sudoers directly if you can avoid it. Instead, create a modular configuration file in /etc/sudoers.d/. This keeps your main config clean and prevents package updates from breaking your settings.
sudo visudo -f /etc/sudoers.d/audit-logsPaste the following configuration into the file:
# Organize logs by user and host
Defaults iolog_dir="/var/log/sudo-io/%{user}/%H"
Defaults iolog_file="%D-%T"
# Capture both what the user types and what they see
Defaults log_output
Defaults log_input
# Inform users they are being monitored
Defaults mail_badpass
Defaults lecture="always"
Defaults lecture_file="/etc/sudo_lecture"What these variables do:
%{user}: Creates a unique subfolder for every user.%H: Includes the hostname, which is vital if you later aggregate logs from multiple servers.%D-%T: Timestamps the session for easy searching.log_output: Records the screen output (the “movie”).log_input: Records every keystroke, including backspaces and hidden characters.
Step 3: Create a Legal Warning (Optional)
In many regions, you are legally required to notify users that their activity is being recorded. Create the lecture file mentioned in the config:
sudo nano /etc/sudo_lectureAdd a clear notice:
ATTENTION: To ensure security and compliance, all activity performed under sudo is recorded.Step 4: Verify the Recording
Switch to a standard user account and run a few commands to generate data.
sudo apt update
sudo tail /var/log/syslog
exitNow, inspect the log directory as root. You will see a new folder structure containing binary files like ttyin and ttyout. Do not try to read these with cat; they require a specific player.
Replaying the Sessions
To watch a session exactly as it happened, use the sudoreplay tool. To see a list of all recorded sessions on the system, run:
sudo sudoreplay -lTo play back a specific session, provide the path found in the list:
sudo sudoreplay /var/log/sudo-io/myuser/myhost/2023-10-27-14:30:05Pro Tip: If a user was idling for long periods, don’t sit through the silence. Use the -s flag to speed things up (e.g., -s 5 for 5x speed) or press the > key during playback to skip forward.
Maintenance and Security
If left unmanaged, /var will eventually fill up. A single admin running journalctl -f for an hour can generate several megabytes of logs.
1. Automate Log Cleanup
Sudo I/O logs do not rotate automatically. I recommend a simple cron job to prune recordings older than 90 days. For a team of five active admins, 90 days of logs typically occupy between 1GB and 5GB of space.
# Run daily at 2:00 AM to delete old logs
0 2 * * * find /var/log/sudo-io -type d -mtime +90 -exec rm -rf {} +2. Hardening Log Integrity
An attacker with root access will immediately try to wipe these logs to hide their tracks. While you cannot fully stop a root user on a local machine, you should stream these logs to a remote, write-only server using rsyslog. This ensures that even if the local server is compromised, the evidence remains safe on a separate machine.
3. Selective Logging
If storage is at a premium, you don’t have to log everything globally. You can apply log_output to specific command aliases in your sudoers file. However, global logging is almost always better for security if you have the disk space to spare.
Sudo I/O logging is a high-impact, low-effort addition to any Linux security stack. Whether you are proving that an unauthorized change occurred or simply retracing your own steps after a long night of troubleshooting, the level of detail it provides is a lifesaver for any sysadmin.
