Configure Linux Auditd: Monitor Sensitive Files and Detect Unauthorized Commands

Real-World Problem: The 2 AM Production Tampering Alert

At 2:14 AM on a Tuesday, my pager vibrated across the nightstand. A production payment node processing 450 transactions per second threw a high-severity alert. A web daemon had spawned an unauthorized interactive bash shell, followed by a checksum mismatch on /etc/pam.d/common-auth. Within eight minutes, checkout requests began dumping 401 Unauthorized errors.

I remoted in over our WireGuard bastion, squinting at terminal windows. Traditional triage hit a dead end immediately. Running history returned empty lines because the intruder opened the session with a leading whitespace trick and unset HISTFILE. Checking timestamps with ls -la showed the file changed at 02:11 UTC. But it couldn’t tell me who modified it, which parent process ran the write, or what account escalated permissions.

Our centralized /var/log/syslog held 14 GB of noise from standard Nginx access traces and microservice health checks. The attacker had altered credentials right under our noses, and basic Linux logging left us completely blind.

Root Cause Analysis: Why Standard Linux Logs Fail Forensics

The failure stems from architectural design: standard logs sit in userland. Application logs, shell histories, and syslog facilities rely on cooperative reporting. If an unprivileged daemon spawns a reverse shell or avoids calling syslog(), the execution never registers in userland logs.

Userland tools also cannot monitor system-call boundaries. When an attacker invokes execve, drops a shell via Python, or opens /etc/shadow with openat, tools like ps or lsof only capture point-in-time snapshots. They miss short-lived spikes. Furthermore, an attacker who obtains root via a misconfigured sudoer entry can clear /var/log/auth.log in three seconds flat.

Reliable accountability requires kernel-level hooks. We need hooks that capture exact syscalls, track the original login identifier (auid), and write tamper-resistant logs before rogue processes clean up after themselves.

Solutions Compared: Userland Tools vs. File Integrity Checkers vs. Kernel Auditing

Engineering teams typically weigh three options when monitoring production systems:

• Process Accounting (psacct / acct): Records commands along with CPU and memory consumption. However, it ignores file reads, write actions, and socket connections. An attacker easily bypasses it by renaming binaries before running them.
• File Integrity Monitoring (AIDE / Tripwire): Runs periodic database comparisons against cryptographic hashes. While it flags that someone touched /etc/pam.d/common-auth, it runs as a cron job—often hours later. It cannot name the active process descriptor or reveal which user issued the write command.
• Linux Audit Daemon (Auditd): Sits directly in the Linux kernel’s audit pipeline. Every system call passes through audit filters before execution completes. It preserves the original login UID (auid)—surviving commands like sudo -i or su—and captures exact binary paths, working directories, and parent PIDs in real time.

Best Approach: Production-Grade Linux Auditd Configuration

Below is the hardened Auditd deployment we rolled out across 120 production nodes to close forensic blind spots.

1. Installing and Enabling Auditd

Auditd is included in core package mirrors across major Linux distributions. On Debian and Ubuntu systems:

sudo apt update && sudo apt install -y auditd audispd-plugins
sudo systemctl enable auditd
sudo systemctl start auditd

On RHEL, Rocky Linux, or AlmaLinux:

sudo dnf install -y audit audit-libs
sudo systemctl enable auditd
sudo systemctl start auditd

2. Hardening Auditd Service Rules

System administrators place custom rule files inside /etc/audit/rules.d/. Never edit /etc/audit/audit.rules directly; boot routines routinely regenerate that file using augenrules. Create a custom configuration file at /etc/audit/rules.d/99-security-hardening.rules:

sudo nano /etc/audit/rules.d/99-security-hardening.rules

Append these directives:

# Flush any legacy rules in memory
-D

# Set kernel audit buffer size (values between 8192 and 32768 handle high-traffic nodes)
-b 8192

# Buffer failure behavior: 1 = syslog warning, 2 = kernel panic
-f 1

# Watch changes to identity files and auth databases
-w /etc/passwd -p wa -k identity_changes
-w /etc/shadow -p wa -k identity_changes
-w /etc/group -p wa -k identity_changes
-w /etc/gshadow -p wa -k identity_changes
-w /etc/security/ -p wa -k pam_security
-w /etc/pam.d/ -p wa -k pam_security
-w /etc/sudoers -p wa -k sudoers_tampering
-w /etc/sudoers.d/ -p wa -k sudoers_tampering

# Watch SSH server configurations and authorized key locations
-w /etc/ssh/sshd_config -p wa -k ssh_config_tampering
-w /root/.ssh/ -p wa -k root_ssh_keys

# Track non-daemon binary executions by interactive users (UID >= 1000)
-a always,exit -F arch=b64 -S execve -F auid>=1000 -F auid!=4294967295 -k command_execution
-a always,exit -F arch=b32 -S execve -F auid>=1000 -F auid!=4294967295 -k command_execution

# Track invocation of privilege escalation tools
-w /bin/su -p x -k priv_escalation
-w /usr/bin/sudo -p x -k priv_escalation
-w /usr/bin/passwd -p x -k password_mod

# Lock audit configuration into kernel memory until reboot
-e 2

Here is what these flags control. The -w flag monitors a specific path, while -p wa monitors write operations and file attribute changes. The -k flag tags matched events with a searchable string. For process monitoring, -S execve logs command invocations from accounts with a UID of 1000 or higher, skipping unset kernel sessions (UID 4294967295). Finally, -e 2 locks the configuration in memory. Once set, not even the root user can clear rules or silence the daemon without restarting the host.

Compile and load your new rule file:

sudo augenrules --load

3. Operational Security & Access Controls

Kernel auditing tells you what happened, but strong credential policies stop attacks before they begin. Attackers who harvest or brute-force administrative credentials bypass standard access walls entirely.

When provisioning jump boxes or production bastion accounts, generate long, random secrets with the password generator at toolcraft.app/en/tools/security/password-generator — its client-side Web Cryptography execution guarantees zero transmission over the wire. Strong passphrases paired with immutable kernel rules remove trivial initial-access paths.

4. Investigating Incidents with ausearch and aureport

Avoid searching through raw lines in /var/log/audit/audit.log using grep. The log entries are dense and encode parameters as hexadecimal strings. Instead, query your tags with ausearch and aureport.

Query every event registered under the pam_security key:

sudo ausearch -k pam_security -i

Adding -i converts raw system call IDs and numeric UIDs into plain text names. A breach event displays unambiguous metadata:

type=PROCTITLE msg=audit(10/22/2026 02:14:18.102:491) : proctitle=nano /etc/pam.d/common-auth
type=SYSCALL msg=audit(10/22/2026 02:14:18.102:491) : arch=x86_64 syscall=openat success=yes exit=3 a0=0x7ffd9b8 ... exe=/usr/bin/nano key="pam_security" auid=deployer uid=root

Take a close look at the IDs. While the current UID shows uid=root, the immutable audit ID shows auid=deployer. Even though the attacker escalated to full root access, the kernel remembers that this session originated from the deployer account. That single line establishes the attack vector.

To view an aggregated summary of failed logins and executed commands over the last day, run:

sudo aureport --start yesterday --executable --summary
sudo aureport --auth --failed

Configuring Auditd takes roughly ten minutes per host. That brief investment turns chaotic midnight incidents into structured, evidence-backed postmortems.