Eliminating Kernel-Level Blind Spots
I once watched a server weather 15,000 SSH brute-force attempts in a single hour. While a firewall can block those IPs, it only addresses the perimeter. If an attacker exploits a zero-day vulnerability in your web app or escapes a misconfigured container, traditional logs often stay silent until the damage is irreversible. You aren’t just looking for a needle in a haystack; you’re looking for a needle that is actively burning the haystack down.
Legacy tools like auditd or syslog are fundamentally reactive. They record the autopsy of an attack rather than preventing it. In high-density environments like Kubernetes, we need to see what the kernel is doing as it happens. We need the ability to kill a malicious process before it even finishes its first system call. This is why eBPF and Tetragon are becoming the new standard for runtime security.
The Problem: The Lag Between Detection and Action
Most security stacks rely on user-space monitoring. When a process runs a malicious script, the OS generates a log. By the time your SIEM parses that entry and pings an engineer, the attacker has already exfiltrated your .env files or encrypted your database. In the world of automated exploits, a five-minute delay is an eternity.
The Reality of Performance Overhead
Traditional monitoring often forces a trade-off between security and speed:
- CPU Tax: Tools like
auditdcan spike CPU usage by 20% to 30% on high-traffic systems due to constant context switching between the kernel and user-space. - Passive Defense: Most monitors are designed to watch, not intervene. They cannot stop a process in the microsecond it tries to access
/etc/shadow.
Comparing the Options: Traditional vs. eBPF-Powered Security
Tetragon shifts the logic from user-space directly into the kernel. Here is how it compares to older methodologies.
| Feature | Auditd / Syslog | Falco (Standard) | Tetragon (eBPF) |
|---|---|---|---|
| Mechanism | Syscall Auditing | Kernel Module / eBPF | Pure eBPF |
| Performance | Low (Heavy Context Switching) | Medium | High (Kernel-native) |
| Real-time Blocking | No | Limited (via sidecars) | Yes (In-kernel enforcement) |
| Granularity | Process level | System call level | Function level (Deep Kernel) |
Why Tetragon? (And Where It Falls Short)
The Benefits
- Zero-Latency Enforcement: Because the security logic lives in the kernel, Tetragon can stop an action before it completes.
- Rich Context: It doesn’t just show a process ID; it maps events to Kubernetes pods, namespaces, and binary metadata.
- Immediate Mitigation: You can configure a “SIGKILL” to terminate a process the moment it violates a policy.
The Trade-offs
- Modern Kernel Required: You’ll need Linux kernel 5.4 or higher. For advanced features like functional enforcement, 5.10+ is mandatory.
- Learning Curve: Crafting
TracingPoliciesrequires you to understand how Linux handles system calls and kernel functions.
Recommended Setup
For production, I recommend running Tetragon as a Kubernetes DaemonSet. For standalone servers, a systemd service works best. This guide uses a containerized approach—it’s the fastest way to see the tool in action without cluttering your host OS.
Prerequisites
- A Linux machine (Ubuntu 22.04 or similar).
- Docker installed.
- BTF (BPF Type Format) support (standard on modern kernels; check via
ls /sys/kernel/btf).
Step-by-Step Implementation
1. Launch Tetragon
Start by running the Tetragon container. It needs privileged access to hook into the host kernel.
docker run --name tetragon --rm \
--privileged -v /sys/kernel/debug:/sys/kernel/debug \
-v /proc:/proc -v /etc/os-release:/etc/os-release \
quay.io/cilium/tetragon:v1.0.0Open a second terminal to monitor the events as they happen:
docker exec tetragon tetra logs2. Create a File Integrity Policy
Let’s catch anyone trying to read /etc/shadow. This file contains password hashes and is a primary target for attackers. We will use a TracingPolicy to monitor the sys_openat system call.
Save this as monitor-shadow.yaml:
apiVersion: cilium.io/v1alpha1
kind: TracingPolicy
metadata:
name: "monitor-etc-shadow"
spec:
kprobes:
- call: "sys_openat"
syscall: true
args:
- index: 1
type: "string"
selectors:
- matchArgs:
- index: 1
operator: "Equal"
values:
- "/etc/shadow"
actions:
- action: PostOnce you apply this, try running sudo cat /etc/shadow. Tetragon will instantly log the event, including the user and the exact command used.
3. Enable the “Kill Switch”
This is where things get interesting. Instead of just logging an event, let’s stop an attacker from running unauthorized tools like nmap. We can instruct the kernel to send a SIGKILL before the process even starts.
Create block-nmap.yaml:
apiVersion: cilium.io/v1alpha1
kind: TracingPolicy
metadata:
name: "block-nmap-execution"
spec:
kprobes:
- call: "sys_execve"
syscall: true
args:
- index: 0
type: "string"
selectors:
- matchArgs:
- index: 0
operator: "Equal"
values:
- "/usr/bin/nmap"
actions:
- action: SigkillIf an attacker manages to download nmap and tries to run it, the process will terminate immediately. They will see a blunt “Killed” message in their terminal, and your network remains unscanned.
Reviewing the Data
Raw JSON logs are hard to read during an active incident. Use the tetra CLI tool to get a clean, human-readable process tree:
docker exec tetragon tetra getevents -o compactThis provides a clear timeline of PROCESS START, FILE OPEN, and SIGKILL events. It allows you to reconstruct the entire attack lifecycle in seconds.
Final Thoughts
Modern security isn’t just about building higher walls; it’s about total visibility inside those walls. By moving your defense into the kernel with eBPF, you eliminate the latency that attackers rely on. After my experience with that midnight SSH attack, I realized that reactive logging isn’t enough. Having the power to kill a malicious process in real-time provides the kind of security that actually lets you sleep at night.
Start by monitoring your most sensitive directories. Once you’re comfortable with the syntax, move to active blocking for high-risk binaries. It’s a massive shift from simply watching your system fail to actively defending it.
