The 2 AM Nightmare: Why Standard Tools Fail
Your phone buzzes at 2 AM. It’s a high-priority alert: the production database is crawling, and API response times are spiking. You SSH into the server and run top. CPU usage looks fine. You check iostat, and while disk utilization is pinned at 90%, you can’t see which process is the culprit. Classic tools like top, ps, and iostat have been our go-to for decades. They’re great for a quick glance, but they only tell you ‘what’ is happening. They rarely explain ‘why’ or ‘who’ is responsible.
These traditional utilities rely on the /proc filesystem. This provides aggregated statistics—essentially snapshots in time. They often miss short-lived processes or the granular latency of individual I/O requests. This is where eBPF (Extended Berkeley Packet Filter) redefines observability. It lets you run sandboxed programs directly in the Linux kernel. You can trace almost any event without touching kernel source code or loading risky modules.
The BPF Compiler Collection (BCC) makes this technology accessible. It provides dozens of pre-built Python scripts that handle the complex compilation logic for you. You don’t need to be a kernel engineer to use them. I’ve used these tools to solve mysteries in minutes that would have otherwise required hours of manual log-combing.
Installation: Getting BCC on Your Machine
Your kernel needs to be relatively modern—version 4.1 or higher—to run BCC. Most environments running Ubuntu 20.04+, Debian 11+, or RHEL 8+ are ready out of the box. While the package names differ slightly between distributions, the underlying tools remain the same.
On Ubuntu/Debian
For Debian-based systems, the package is available in the standard repositories. You must also install the kernel headers that match your running version. This allows BCC to compile BPF programs on the fly.
sudo apt update
sudo apt install bpfcc-tools linux-headers-$(uname -r)On RHEL/CentOS Stream/AlmaLinux
RHEL-based systems include BCC in the AppStream repository. Use the following command to get started:
sudo dnf install bcc-toolsHeads up: RHEL and AlmaLinux users will usually find these scripts tucked away in /usr/share/bcc/tools. You might want to add that directory to your PATH or call the tools using their full file path.
Configuration: Preparing the Kernel Environment
Getting the software onto your disk is just the first step. BCC compiles C code into BPF bytecode at runtime, which requires access to your kernel configuration. If you’re working in a stripped-down cloud image or a restricted container, you might see errors regarding missing files in /lib/modules/$(uname -r)/build.
I tested this on a production Ubuntu 22.04 instance with 4GB of RAM. Ensuring the linux-headers were correctly mapped allowed BCC to initialize in under two seconds. Without them, the tools simply fail with cryptic compilation errors. Speed matters when a database is hanging.
Verify the setup by running opensnoop. This tool tracks every open() system call across the entire OS. If it starts printing file paths, your environment is ready for deep diagnostics:
sudo /usr/sbin/opensnoop-bpfccIf you see a live stream of files being accessed by various PIDs, you’re ready to start hunting for real performance bottlenecks.
Verification & Monitoring: Battle-Tested Tools
The BCC suite includes over a hundred utilities. During a live incident, you don’t have time to read the manual. I keep a mental shortlist of four specific tools that solve about 90% of common performance mysteries.
1. Tracking Short-Lived Processes with execsnoop
Have you ever seen CPU spikes while top shows an idle system? This is often caused by “phantom” processes—scripts or cron jobs that start and exit in milliseconds. Standard tools don’t poll fast enough to catch them.
sudo execsnoop-bpfccThis displays every new process execution, including the parent PID and full command-line arguments. I once found a broken shell script that was recursively calling itself. It was spawning 2,500 processes per second, a frenzy that htop completely missed.
2. Measuring Disk Latency with biolatency
Disk utilization percentages are often misleading. A disk at 100% utilization might still be performing well, while a disk at 10% could be causing massive lag due to high seek times. biolatency cuts through the noise by providing a histogram of actual I/O latency.
sudo biolatency-bpfcc 10 1This command captures data for 10 seconds and prints the distribution. Look for clusters. If you see a high count in the >128ms range, your storage backend is failing you, regardless of what your throughput numbers say.
3. Identifying Network Hogs with tcptop
When the network is saturated, iftop shows you bandwidth per host. However, tcptop shows you throughput per process. This is the fastest way to identify which specific container or service is eating your bandwidth.
sudo tcptop-bpfccIt functions like top, but for TCP connections. It displays PIDs, IP addresses, and RX/TX throughput in Kilobytes. It’s perfect for spotting a rogue backup script or a data leak in real-time.
4. Finding File System Latency with ext4slower
Sometimes the hardware is fine, but the filesystem layer is slow due to lock contention. If you use EXT4, ext4slower tracks common operations like reads and writes. It only reports those that exceed a specific threshold, such as 10ms.
sudo ext4slower-bpfcc 10This is invaluable for database troubleshooting. If a write() call to a transaction log takes 50ms, your database performance will tank. This tool identifies the exact file and PID that suffered the delay.
Interpreting the Results
Using BCC tools is all about finding the outliers. In a healthy system, histograms should be “top-heavy,” meaning most operations happen in microseconds. When you see a “bimodal” distribution—where a second peak appears in the millisecond or second range—you’ve found your performance leak.
Overhead is a major concern in production. Strace can cripple a process, slowing it down by 10x or more. BCC tools use eBPF to keep overhead extremely low, typically less than 1%. This safety allows you to run diagnostics on a live server under heavy load without crashing the application.
The next time you face a sluggish system and uptime shows a high load average with no obvious cause, stop guessing. Run execsnoop to check for process churn and biolatency to inspect the disks. Usually, one of these will lead you to the root cause before your coffee even gets cold.
