Posted inLinux

Mastering Linux Security Auditing with Auditd: Rules, File Monitoring, and Log Analysis

Linux tutorial - IT technology blog
Linux tutorial - IT technology blog

Real-World Problem: The Mystery of Modified Production Files

A production web server suddenly behaves erratically. An unauthorized public key appears in /root/.ssh/authorized_keys, or an unknown process reads a production .env file holding database secrets. You inspect /var/log/syslog and .bash_history. Nothing useful shows up. The attacker wiped their shell history with unset HISTFILE, and syslog only recorded routine cron jobs and service restarts.

Standard Linux system logs report that a service restarted or when a session opened. They fail to answer critical forensic questions:

  • Who ran the command? Syslog cannot reliably distinguish a real login account from an escalated sudo session.
  • What exact file path or system call was touched?
  • Which process ID or executable initiated the change?
  • When did an unauthorized process read a sensitive file without modifying its contents?

Without kernel-level telemetry, security incident investigations stall. You are left blind to privilege escalation and insider threats.

Root Cause Analysis: Why Standard Logging Falls Short

Application-level logging relies on software voluntarily dispatching messages to daemons like rsyslog or systemd-journald. If an intruder edits /etc/passwd using vim or executes a dropped binary from /dev/shm, userland loggers stay completely silent. Direct file operations never communicate with syslog.

Attackers routinely forge file timestamps using touch -r or clear environment variables to cover their tracks. Reliable monitoring must run below userland inside the Linux kernel itself. Every file read, process execution, and network binding flows through system calls—such as openat(), execve(), and unlinkat(). The Linux Audit Framework (auditd) hooks directly into these syscalls, capturing immutable event records before user space can alter them.

Solutions Compared: Choosing the Right Auditing Mechanism

Linux administrators generally evaluate three main approaches for tracking system activity:

1. Process Accounting (psacct / acct)

  • Pros: Minimal setup, extremely lightweight on RAM (< 10 MB), tracks basic command terminations.
  • Cons: Trivial for root users to disable. It records executed command names but ignores file paths, file reads, network calls, and modified file attributes.

2. eBPF Security Engines (Falco / Tetragon)

  • Pros: Programmable, near-zero syscall overhead, real-time alerting, purpose-built for Kubernetes and container workloads.
  • Cons: Requires Linux kernel 5.8+ for full feature sets, consumes more memory (often 200–500 MB per node), and carries a steeper learning curve for custom policy development.

3. Linux Audit Daemon (auditd)

  • Pros: Built directly into the kernel, pre-installed on enterprise distributions (RHEL, Debian, Ubuntu, Rocky Linux), satisfies strict compliance mandates (PCI-DSS 10.2, CIS Benchmark, SOC 2, HIPAA), and includes dedicated forensic query utilities (ausearch, aureport).
  • Cons: High I/O overhead and rapid disk consumption if rules are overly broad or uncurated.

Best Approach: Implementing and Configuring Auditd Step-by-Step

On busy production nodes handling 500+ requests per second, poorly scoped audit rules can generate gigabytes of logs per hour. Follow this structured process to deploy, configure, and verify auditd safely without overwhelming storage.

Step 1: Install and Enable the Audit Daemon

Install the necessary packages on Ubuntu or Debian:

sudo apt update
sudo apt install auditd audispd-plugins -y

On RHEL, Rocky Linux, or AlmaLinux:

sudo dnf install audit -y

Start and enable the service on boot:

sudo systemctl enable --now auditd
sudo systemctl status auditd

Step 2: Define Persistent Audit Rules

Direct commands via auditctl apply immediately but disappear after a reboot. For persistence, write modular rules in /etc/audit/rules.d/. The augenrules script compiles these into /etc/audit/audit.rules.

Rules fall into two primary categories:

  1. File Watches: Track access to files or directories for read (r), write (w), execute (x), or attribute changes (a).
  2. Syscall Rules: Intercept kernel-level functions like execve, settimeofday, or ptrace.

Create a dedicated configuration file at /etc/audit/rules.d/security.rules:

sudo nano /etc/audit/rules.d/security.rules

Add these production-grade rules:

## Delete existing rules and allocate kernel buffer (8,192 messages)
-D
-b 8192
-f 1

## 1. Monitor user and group database modifications
-w /etc/passwd -p wa -k identity_changes
-w /etc/shadow -p wa -k identity_changes
-w /etc/group -p wa -k identity_changes
-w /etc/gshadow -p wa -k identity_changes

## 2. Track SSH key additions and daemon configuration changes
-w /root/.ssh/ -p wa -k ssh_key_tamper
-w /etc/ssh/sshd_config -p wa -k sshd_config_change

## 3. Monitor scheduled jobs and privilege escalation rules
-w /etc/crontab -p wa -k cron_tamper
-w /etc/cron.d/ -p wa -k cron_tamper
-w /etc/sudoers -p wa -k sudo_privilege_escalation
-w /etc/sudoers.d/ -p wa -k sudo_privilege_escalation

## 4. Detect system clock tampering (often used to forge log order)
-a always,exit -F arch=b64 -S adjtimex,settimeofday,clock_settime -k system_time_change

## 5. Capture privilege escalation: commands run by standard users with root EUID
-a always,exit -F arch=b64 -S execve -C uid!=euid -F euid=0 -k elevated_priv_exec

Rule parameter breakdown:

  • -b 8192: Sets the kernel audit buffer to 8,192 messages. This prevents dropped events during bursty write spikes.
  • -w <path>: Defines the target file or directory watch.
  • -p wa: Fires whenever a process writes to or changes the attributes (permissions, ownership) of the file.
  • -k <key_name>: Tags records with a custom filter key for fast indexing.
  • -a always,exit: Records the audit event as soon as the specified syscall completes.
  • -F arch=b64: Limits filtering to 64-bit architecture system calls.

Step 3: Load and Verify the Rules

Compile and load your new rules without restarting the daemon:

sudo augenrules --load

List active kernel rules to verify they loaded properly:

sudo auditctl -l

Step 4: Query and Filter Logs with ausearch and aureport

Raw entries live in /var/log/audit/audit.log. These records contain raw hex paths and numeric IDs. Use ausearch to extract and translate them into human-readable details.

1. Filter by rule key with translated IDs:

sudo ausearch -k identity_changes -i

The -i (interpret) flag automatically converts numeric UIDs, GIDs, and hex-encoded strings into names like ubuntu, root, and /etc/passwd.

2. Search for events within a precise time window:

sudo ausearch -k ssh_key_tamper -ts today -te now -i

3. Filter actions initiated by a specific user (UID 1000):

sudo ausearch -ua 1000 -i

4. Generate fast audit summaries with aureport:

# Summarize all file modifications across the system
sudo aureport -f -i --summary

# List all failed authentication and sudo attempts
sudo aureport -au -i --failed

# Count all executed commands grouped by executable
sudo aureport -x -i --summary

Step 5: Anatomy of an Audit Log Record

When someone modifies a monitored file, auditd writes a multi-line entry. Running ausearch -k identity_changes -i produces output like this:

type=SYSCALL msg=audit(10/03/2026 10:15:22.412:128) : arch=x86_64 syscall=openat success=yes exit=3 a0=0xffffff9c a1=0x55dc87a21390 a2=0x241 a3=0x1b6 items=2 ppid=1420 pid=2105 auid=ubuntu uid=root euid=root exe=/usr/bin/nano key="identity_changes"
type=PATH msg=audit(10/03/2026 10:15:22.412:128) : item=0 name="/etc/passwd" inode=264104 dev=fc:01 mode=file,644 ouid=root ogid=root rdev=00:00 nametype=NORMAL

Key forensic fields to examine:

  • auid=ubuntu: The Audit User ID. It preserves the original login identity. Even if a user executes sudo su to become root, auid maintains their real identity (ubuntu).
  • uid=root / euid=root: The effective user running the binary at execution time.
  • exe=/usr/bin/nano: The exact binary path invoked.
  • name="/etc/passwd": The specific target file accessed or modified.
  • key="identity_changes": The matched rule label from your configuration.

Summary & Production Hardening Tips

Kernel auditing provides unambiguous proof of what happened on your systems. Keep these operational safeguards in place:

  • Prevent disk exhaustion: Edit /etc/audit/auditd.conf and set max_log_file = 50 (MB), num_logs = 5, and max_log_file_action = ROTATE. This caps audit log storage at 250 MB.
  • Avoid noise: Never set unrestricted write watches on high-churn directories like /tmp, /var/log, or active database data paths.
  • Lock the rule set: Add -e 2 as the final line in /etc/audit/rules.d/security.rules once tested. This makes rules immutable; no user—including root—can modify or stop auditing without a complete server reboot.
Share:
Tags: