Writing Custom Probes with bpftrace on Linux: Real-Time Kernel and App Tracing in Minutes

Linux tutorial - IT technology blog
Linux tutorial - IT technology blog

Six Months Ago, I Was Flying Blind

My Ubuntu 22.04 production server started showing intermittent latency spikes — requests jumping from 20ms to 400ms with no obvious pattern. CPU was fine. Memory was fine. Disk I/O looked normal. The usual suspects were clean.

I spent two weeks throwing conventional tools at the problem: strace, perf stat, top, iostat. Each gave me a slice of the picture, but none connected the dots. The latency spikes were real, they were hurting users, and I couldn’t pin them down.

That’s when a colleague mentioned bpftrace. I’d heard of it but never seriously used it. After a weekend of experimentation, I had a custom probe running that identified the root cause in under ten minutes. What I’d spent two weeks chasing turned out to be a specific kernel function being called thousands of times per second during certain workloads.

That experience changed how I think about observability. Here’s what I learned.

Why Standard Tools Fall Short

Tools like strace and perf aren’t bad — they’re just scoped. strace gives you system calls for a single process. perf gives you CPU counters and stack traces. Neither lets you ask surgical questions like: “Every time vfs_read() is called by my app, show me the filename and the calling PID together.”

The Root Cause Problem

Performance problems rarely live in one place. They emerge at the intersection of your application, the kernel, and the hardware. To trace that intersection, you need to instrument multiple layers simultaneously and correlate the data. That’s exactly what bpftrace is designed to do.

bpftrace is a high-level tracing language built on top of eBPF (Extended Berkeley Packet Filter). It compiles small tracing scripts into bytecode that runs safely inside the kernel — no kernel modules, no reboots, no risk of crashing the system. You write a few lines, run them, get answers, then stop. The overhead when not running is literally zero.

Three Approaches Compared

Before settling on bpftrace, I evaluated three options for custom tracing:

Option 1: strace + awk Pipelines

Works for single-process syscall tracing. Easy to start. Completely falls apart when you need cross-process correlation or kernel-internal function tracing. The overhead under load is also significant — attaching strace to a busy process can slow it down by 10–50x.

Option 2: Custom Kernel Modules

Full power, but writing a kernel module for tracing means C code, kernel headers, careful memory management, and a reboot if you get it wrong. It’s the right tool if you need persistent, production-grade instrumentation baked into the kernel. For debugging a live issue, the iteration time is punishing.

Option 3: bpftrace Scripts

The sweet spot. You get kernel-level visibility with the iteration speed of a scripting language. Scripts are three to twenty lines long. They attach to kernel functions, userspace probes, tracepoints, and hardware events. If your script has a bug, it either fails to compile or the kernel’s BPF verifier rejects it — either way, your system stays running.

On my production Ubuntu 22.04 server with 4GB RAM, I found this approach significantly reduced processing time for diagnosing issues compared to any alternative. The feedback loop from “I have a hypothesis” to “I have data” dropped from hours to minutes.

Getting bpftrace Running

Installation

# Ubuntu 20.04+
sudo apt install -y bpftrace

# Verify installation
bpftrace --version

# Check available probes (outputs thousands of lines)
bpftrace -l 'kprobe:vfs_*' | head -20

You need a kernel 4.9 or newer with BPF support compiled in — which is true of virtually every modern distribution kernel. You also need root privileges or the CAP_BPF capability to run probes.

The Basic Probe Structure

Every bpftrace script follows the same pattern:

probe_type:probe_name /filter/ {
    action;
}

That’s it. The filter is optional. The action runs every time the probe fires.

Writing Your First Custom Probe

Tracing File Reads

Start with something concrete. This probe fires every time any process calls vfs_read():

sudo bpftrace -e '
kprobe:vfs_read {
    printf("%s read file\n", comm);
}'

comm is a built-in variable that holds the process name. This will produce a lot of output — every file read on the system. Add a filter:

sudo bpftrace -e '
kprobe:vfs_read /comm == "nginx"/ {
    printf("nginx read: pid=%d size=%d\n", pid, arg2);
}'

arg2 is the third argument to vfs_read() — the requested byte count. Probe arguments map directly to the C function signature.

Measuring Latency with Entry/Return Pairs

This is where bpftrace gets genuinely useful. To measure how long a kernel function takes:

sudo bpftrace -e '
kprobe:do_sys_openat2 {
    @start[tid] = nsecs;
}

kretprobe:do_sys_openat2 /@start[tid]/ {
    @latency = hist(nsecs - @start[tid]);
    delete(@start[tid]);
}'

When this exits (Ctrl+C), it prints a histogram of open() latencies in nanoseconds across all processes. @start is a map (hash table) keyed by thread ID. nsecs is a timestamp built-in. hist() aggregates values into power-of-two buckets automatically.

The Probe That Found My Bug

Back to my original problem. After some investigation, I suspected excessive page cache pressure. Here’s the probe I wrote:

sudo bpftrace -e '
kprobe:shrink_node {
    @shrink_calls = count();
    @by_process[comm] = count();
}

interval:s:5 {
    printf("Last 5s: %d shrink_node calls\n", @shrink_calls);
    print(@by_process);
    clear(@shrink_calls);
    clear(@by_process);
}'

The interval:s:5 probe fires every 5 seconds regardless of kernel events — useful for periodic reporting. Within thirty seconds, I saw one specific process triggering shrink_node (the memory reclaim function) hundreds of times per second during its batch operations. That was the culprit: a data processing job that allocated and freed large buffers in a tight loop, repeatedly thrashing the page cache.

Practical Probe Patterns

Tracing Specific Userspace Functions

bpftrace isn’t limited to kernel functions. You can probe userspace binaries with USDT (User Statically Defined Tracing) probes or uprobes:

# Trace a specific function in a running binary
# Replace /usr/bin/python3 and function offset as needed
sudo bpftrace -e '
uprobe:/usr/bin/python3:PyEval_EvalFrameEx {
    @python_calls[ustack(5)] = count();
}'

Watching System Call Patterns

# Count syscalls per process, update every 3 seconds
sudo bpftrace -e '
tracepoint:raw_syscalls:sys_enter {
    @syscalls[comm] = count();
}

interval:s:3 {
    print(@syscalls);
    clear(@syscalls);
}'

Network Connection Tracing

# Log every new TCP connection with source and dest
sudo bpftrace -e '
kprobe:tcp_connect {
    $sk = (struct sock *)arg0;
    printf("TCP connect: pid=%d comm=%s\n", pid, comm);
}'

Writing Reusable Probe Scripts

One-liners are great for quick investigations. For repeated use, save probes as .bt files:

cat > /usr/local/bin/file-latency.bt << 'EOF'
#!/usr/bin/env bpftrace

// Track open() latency for a given process
// Usage: bpftrace file-latency.bt

kprobe:do_sys_openat2 /comm == str($1)/ {
    @start[tid] = nsecs;
}

kretprobe:do_sys_openat2 /@start[tid]/ {
    @us = hist((nsecs - @start[tid]) / 1000);
    delete(@start[tid]);
}

END {
    printf("\nLatency histogram (microseconds):\n");
    print(@us);
}
EOF

sudo bpftrace file-latency.bt nginx

Limitations Worth Knowing

bpftrace isn’t the answer to everything. The BPF verifier enforces strict limits: loops must have bounded iterations, stack space is limited to 512 bytes, and certain memory access patterns are rejected. For long-running production instrumentation with complex logic, tools like BCC (BPF Compiler Collection) or even kernel modules may be more appropriate.

Probe availability also depends on your kernel configuration. Some functions are inlined by the compiler and can’t be kprobed. If a probe silently produces no output, check with bpftrace -l 'kprobe:function_name' to verify it’s attachable.

The Practical Takeaway

Six months after that initial diagnosis, bpftrace scripts have become a standard part of my troubleshooting workflow. The patterns I use most are latency histograms on suspicious kernel paths, per-process syscall counts when something looks CPU-hungry, and periodic reporting intervals to catch bursty behavior.

The investment to learn the basics is a few hours. The return is the ability to answer questions about your running system that no other tool can answer without custom kernel code. For anyone running Linux infrastructure at any scale, that’s a trade worth making.

Start with the one-liners. Build up to latency histograms. When you hit a problem you can’t explain with conventional tools, write a probe. The answers are usually there within minutes.

Share: