Hunting Web Shells on Linux: An Incident Response Field Guide

Table of Contents

The Nightmare of the Silent Backdoor

Waking up to a compromised server is a high-stress experience. Years ago, after my first production server was hit by a brute-force attack at 3 AM, I became obsessed with hardening SSH. However, we often overlook the most obvious entry point: the web application itself. A single unpatched plugin or a weak file upload form can let an attacker drop a web shell in seconds.

A web shell is a malicious script written in languages like PHP, Python, or JSP. It provides a remote administration interface, allowing attackers to execute commands and pivot through your network. Because this traffic flows over standard HTTP/HTTPS ports, it bypasses most basic firewalls. This guide covers how to hunt these scripts, analyze their behavior, and clean the infection properly.

Why Web Shells Are Hard to Catch

Web shells hide in plain sight within the application layer. They don’t require an open listener or a new port. If your server can execute .php files, it can run a shell. Attackers use several tricks to stay invisible:

Obfuscation: They use base64_encode or str_rot13 to hide strings like “system” or “exec”.
Timestomping: Attackers use the touch command to match the shell’s modification date with legitimate files from 2018.
Masquerading: You might find a shell named wp-includes.php or thumb_823.php hidden in an uploads folder with 5,000 other images.

Installation: Equipping Your Toolkit

Manual inspection is vital, but automated scanners find the low-hanging fruit quickly. We will use Linux Malware Detect (LMD), often called Maldet, alongside native CLI tools.

Installing Linux Malware Detect (LMD)

LMD is specifically tuned for the types of shells found in shared hosting and web environments. Here is the setup process for a standard Ubuntu or Debian system:

cd /tmp
wget http://www.rfxn.com/downloads/maldetect-current.tar.gz
tar -xvf maldetect-current.tar.gz
cd maldetect-*
sudo ./install.sh

After the installation finishes, pull the latest signatures immediately. New shell variants emerge daily, and outdated signatures are useless.

sudo maldet -u

Native Linux Utilities

You don’t always need fancy software. I rely on grep, find, and awk for 90% of my investigations. Ensure your coreutils are up to date, as we will rely on precise file metadata to spot anomalies.

Configuration: Setting Up the Hunt

Finding a shell requires looking for the “smell” of malicious code. We combine pattern matching with time-based analysis.

1. Searching for Malicious Patterns

Most shells rely on a specific set of PHP functions to do their dirty work. Run a recursive search through your web root—usually /var/www/html—to flag these functions. This command uses Perl-compatible regular expressions to find common execution strings:

grep -RPn "(passthru|shell_exec|system|base64_decode|eval|assert|fopen|gzuncompress)" /var/www/html/

Expect noise. Modern frameworks like Laravel or WordPress use base64 and fopen for legitimate tasks. Focus your attention on files where these functions appear in long, single lines of garbled text.

2. Finding Recently Modified Files

If your site started acting strangely this morning, check for files modified in the last 24 hours. Attackers often forget to timestomp every file they touch.

find /var/www/html -type f -mtime -1 -ls

If you see a .php file inside a /wp-content/uploads/ directory, it is almost certainly malicious. Legitimate upload folders should only contain media, not executable code.

3. Automating Maldet

Edit /usr/local/maldetect/conf.maldet to automate your defenses. I recommend these settings for a balance between security and system performance:

# Set to 1 to receive alerts
email_alert="1"
email_addr="[email protected]"

# Move hits to quarantine automatically
quarantine_hits="1"

# Try to clean malware injections from files
quarantine_clean="1"

Verification & Monitoring: Analyzing the Breach

Finding a suspicious file is only half the battle. You need to know what the attacker did while they were inside.

Step 1: Isolation

Don’t delete the file yet. You might need it for forensics. Instead, move it to a safe zone and strip its ability to run.

mkdir /root/quarantine
mv /var/www/html/path/to/suspect.php /root/quarantine/
chmod 000 /root/quarantine/suspect.php

Step 2: Decoding the Shell

Open the file in vim or nano. If you see eval(base64_decode('...')), the attacker is hiding their code. You can safely decode it using the PHP command line to see the actual logic:

# Use a sandbox if you can
php -r "echo base64_decode('ENCODED_STRING_HERE');" > decoded_output.txt

Step 3: Checking Active Connections

Web shells often connect back to a Command & Control (C2) server. Use ss to see if your web server processes are talking to unusual IP addresses:

ss -antp | grep -E "(httpd|apache|nginx|php-fpm)"

An outbound connection to a random IP on port 4444 or 8080 is a massive red flag. It usually indicates a reverse shell is currently active.

Step 4: Log Correlation

Find the exact second the shell was created. Then, search your access.log for POST requests at that time. You are looking for the entry point:

grep "shell.php" /var/log/apache2/access.log

A typical log entry might look like this: 1.2.3.4 - - [10/Oct/2023:14:00:01] "POST /plugin/upload.php HTTP/1.1" 200. This tells you exactly which plugin is vulnerable. If you don’t patch that plugin, the attacker will return within minutes.

Step 5: Real-Time Monitoring with Auditd

To stop future attacks, use Auditd to watch your web directory. This logs every file change in real-time.

sudo auditctl -w /var/www/html -p wa -k web_modification

Any time a file is written or changed, it will appear in /var/log/audit/audit.log. This provides an undeniable trail of evidence for future incidents.

Summary Checklist

Schedule maldet to run daily via cron.
Audit your web root for dangerous PHP functions weekly.
Disable execution in upload directories using .htaccess or Nginx config.
Correlate file timestamps with web server logs to find the vulnerability.
Always patch the entry point before bringing the site back online.

Server management is a game of persistence. By combining automated tools with manual log analysis, you can catch web shells before they escalate into a full data breach. Stay alert, and never assume a “clean” scan means you’re safe.