The 2 AM Ghost: Why Disk Forensics Fails
I found myself hunched over a terminal at 2 AM, watching an Ubuntu web server act like it was possessed. CPU usage was pegged at 98% across all cores, yet top reported a load average of nearly zero. I scoured /var/log/auth.log and /var/log/syslog, but they were pristine—too pristine.
The attacker was thorough. They had wiped the logs and deployed a rootkit to mask their presence from standard tools like ls and ps. This wasn’t a script kiddie; this was a resident threat living entirely in the system’s volatile memory.
When an intrusion reaches this level of stealth, checking the hard drive is a dead end. Disk forensics shows you the past, but memory forensics reveals the right now. Volatility 3 is the go-to tool for this work. It lets us bypass the compromised operating system to see exactly what the kernel is doing, regardless of what the attacker wants us to see.
Building Your Analysis Lab
We are using Volatility 3 for this walkthrough. While Volatility 2 served the community for a decade, it struggles with modern Linux kernels and relies on outdated Python 2 dependencies. Volatility 3 is significantly faster and handles symbol files with much less friction.
1. Setting Up the Toolkit
Always perform your analysis on a clean machine. Running forensic tools on an infected host risks alerting the malware or corrupting evidence. Start by installing the core dependencies:
sudo apt update
sudo apt install -y python3 python3-pip git
git clone https://github.com/volatilityfoundation/volatility3.git
cd volatility3
pip3 install -r requirements.txt2. Grabbing the RAM Image
You need a snapshot of the RAM before you can dig in. For Linux targets, I recommend AVML (Acquire Volatile Memory Dump) by Microsoft. It is a static binary, so you don’t have to install libraries on the target machine. This keeps your forensic footprint small and prevents the “observer effect” from altering the evidence.
Run these commands on the target machine to generate a mem.dump file:
# Download the standalone binary
wget https://github.com/microsoft/avml/releases/download/v0.14.0/avml
chmod +x avml
# Capture the 16GB or 32GB memory image
sudo ./avml mem.dumpOnce finished, move that mem.dump file to your analysis workstation using SFTP or a secure drive.
The Symbol Hurdle: Making Sense of the Data
Linux kernels are highly diverse, which makes memory analysis tricky. Unlike Windows, where Volatility can fetch symbols from Microsoft’s servers, Linux requires an Intermediate Symbol File (ISF) that matches your specific kernel version (e.g., 5.15.0-71-generic).
Most engineers trip up here. First, check if Volatility can identify the image:
python3 vol.py -f mem.dump banner.BannerIf the banner reveals the kernel version but other plugins fail, you must provide symbols. For standard distributions like Ubuntu, you can use the dwarf2json tool to convert debug symbols into a format Volatility understands. For this guide, we’ll assume you’ve placed the resulting JSON file in volatility3/symbols/linux/.
The Investigation: Following the Breadcrumbs
With the environment ready, the hunt begins. I usually look for three things: the entry point, the persistence mechanism, and the Command and Control (C2) callback.
1. Hunting for Hidden Processes
Attackers often hide processes by unlinking them from the task_struct list in the kernel. While pslist shows what the kernel thinks is running, psscan finds the ghosts.
python3 vol.py -f mem.dump linux.pslist.PsList
python3 vol.py -f mem.dump linux.psscan.PsScanCompare the two outputs. If a process appears in psscan but not in pslist, you’ve found a hidden process. Look for suspicious names like kworker_unbound or processes running as root that have no business being there.
2. Analyzing Network Telemetry
A compromised server almost always talks back to its master. Even if the attacker hides the socket from netstat on the live system, the connection data remains etched in RAM.
python3 vol.py -f mem.dump linux.netstat.NetStatSearch for connections to high-numbered ports like 4444 or 31337. Look for IP addresses outside your known CDN or management ranges. If you find a connection to a suspicious IP, note the Process ID (PID) immediately.
3. Spotting Injected Code
Advanced malware often injects shellcode into legitimate processes like nginx or systemd. The malfind plugin identifies memory regions that are both writable and executable. This is a massive red flag.
python3 vol.py -f mem.dump linux.malfind.MalfindIf malfind flags a region, it provides a hex dump. Look for ELF headers or NOP sleds (a long sequence of 0x90 bytes). These are clear indicators of a payload waiting to execute.
4. Exposing Kernel-Level Hooks
The most dangerous rootkits “hook” the System Call Table. They intercept calls like read to hide files or getdents to hide directories.
python3 vol.py -f mem.dump linux.check_syscall.Check_syscallIf a syscall points to a memory address outside the core kernel (vmlinux), Volatility will highlight it. This is the smoking gun of a kernel-level compromise.
The Forensic Mindset
Successful analysis isn’t just about running commands; it’s about telling a story. I might start with a suspicious network connection that leads me to a specific PID. I then use malfind on that PID to see the injected code. Finally, I use the lsof plugin to see which files that process was touching before the attacker deleted them.
python3 vol.py -f mem.dump linux.lsof.Lsof --pid <SUSPICIOUS_PID>Memory forensics gives you the visibility that a compromised OS tries to deny you. It turns a vague suspicion into hard, technical evidence. This data is what you need to rebuild your infrastructure and ensure the attacker doesn’t get back in.
