Comparing Runtime Security Approaches in Kubernetes
Static image scanning in CI/CD catches known CVEs before deployment, but it goes blind the moment a container starts running. If an attacker exploits a zero-day flaw, escapes a container, or drops an interactive shell inside a running pod, image scanners won’t alert you.
Kubernetes audit logs help monitor control-plane actions like pod creation or ConfigMap modifications. What they miss is process-level activity inside those pods. Audit logs won’t show you when a rogue process rewrites /etc/shadow or initiates an outbound connection to an unknown IP. Catching those threats requires tracing system calls directly in the Linux kernel.
Falco operates right at this boundary. It intercepts core system calls—including execve, openat, and connect—across every node in real time. Falco evaluates these raw events against behavioral rules, matches them with live Kubernetes metadata (namespace, pod name, image tag), and generates alerts within milliseconds.
Kernel Modules vs Modern eBPF Drivers
Falco captures system calls through two main engine drivers: traditional loadable kernel modules and modern eBPF probes. Your choice directly affects cluster stability, node maintenance, and cloud portability.
1. Traditional Kernel Module (DKMS)
- Pros: Runs on legacy Linux kernels (pre-4.14); incurs low CPU overhead on older distributions.
- Cons: Demands matching kernel headers on every worker node to compile drivers via DKMS. A single bug in the driver can trigger a full kernel panic and bring down the node. Managed platforms like EKS and GKE Autopilot also disallow custom kernel modules entirely.
2. Modern eBPF Probe with CO-RE (Recommended)
- Pros: Executes sandboxed bytecode verified by the Linux kernel, eliminating kernel panic risks. Thanks to BPF Type Format (BTF) and CO-RE (Compile Once – Run Everywhere), it requires no local compilation or installed kernel headers. It integrates seamlessly with modern cloud-managed nodes.
- Cons: Requires Linux kernel 5.8 or newer with BTF enabled. Under heavy I/O spikes (over 50,000 syscalls/sec), ring buffer memory usage may jump by 50–100MB.
Production Architecture and Resource Profile
Running kernel-level monitoring across high-density clusters requires careful overhead management. On nodes handling 2,000 to 5,000 system calls per second, Falco with the modern eBPF driver generally uses less than 1.5% CPU and between 150MB and 250MB of RAM.
A standard production layout deploys Falco as a DaemonSet across worker nodes. From there, Falcosidekick fans out alerts by priority to Slack, PagerDuty, SIEM platforms, or serverless functions for automatic pod isolation.
[ Linux Kernel (eBPF Probe) ]
│ (Syscalls: execve, openat, connect)
▼
[ Falco DaemonSet Pod ]
- Rules Engine Evaluation (<10ms latency)
- K8s Metadata Enrichment
│ (gRPC / HTTP POST)
▼
[ Falcosidekick ]
├── Slack / PagerDuty (Critical Alerts)
├── SIEM / Elasticsearch (Audit Archive)
└── Webhook / FaaS (Automated Pod Isolation)Step-by-Step Implementation Guide
Step 1: Check Node Kernel and BTF Support
First, verify that your cluster nodes run a compatible Linux kernel (5.8+) with BTF enabled:
# Inspect kernel versions across all nodes
kubectl get nodes -o wide
# Check for BTF support via a node debug session
kubectl debug node/<your-node-name> -it --image=busybox -- ls -lh /sys/kernel/btf/vmlinuxIf /sys/kernel/btf/vmlinux is present (typically ~4–6MB), your nodes support modern eBPF out of the box without on-node header packages or driver compilation.
Step 2: Deploy Falco with Modern eBPF via Helm
Add the official Falco repository and set up a dedicated namespace:
helm repo add falcosecurity https://falcosecurity.github.io/charts
helm repo update
kubectl create namespace falcoCreate a customized values file named falco-values.yaml. This selects the modern eBPF driver and configures Falcosidekick for alert routing:
driver:
kind: modern_ebpf
falco:
json_output: true
log_stderr: true
log_level: notice
priority: notice
falcosidekick:
enabled: true
webui:
enabled: true
config:
slack:
webhookurl: "https://hooks.slack.com/services/YOUR/SLACK/WEBHOOK"
minimumpriority: "warning"
messageformat: "*Falco Alert:* {{ .Rule }} | Pod: {{ .OutputFields.k8s.pod.name }} | Priority: {{ .Priority }}"Deploy the chart to your cluster:
helm install falco falcosecurity/falco \
--namespace falco \
-f falco-values.yamlConfirm that the DaemonSet pods and the Falcosidekick service are healthy:
kubectl get pods -n falco -o wideStep 3: Add Custom Threat Detection Rules
Falco ships with robust default rules, but production clusters need explicit policies against suspicious container behavior. Save the following rules to custom-rules.yaml to flag unauthorized shells and sensitive credential reads:
customRules:
custom-threat-rules.yaml: |-
- rule: Unauthorized Shell Spawned in Production Container
desc: Detect interactive shell execution inside production or payment pods
condition: >
container.id != host and
k8s.ns.name in ("production", "payment") and
evt.type = execve and
proc.name in (bash, sh, zsh, ksh, csh)
output: >
Shell spawned in production pod (user=%user.name pod=%k8s.pod.name
ns=%k8s.ns.name container=%k8s.container.name image=%container.image.repository
cmd=%proc.cmdline)
priority: CRITICAL
tags: [mitre_execution, container, production]
- rule: Sensitive Credentials Read Attempt
desc: Detect unauthorized access to sensitive credential files
condition: >
container.id != host and
fd.name startswith "/etc/" and
fd.name in ("/etc/shadow", "/etc/sudoers", "/etc/pam.conf") and
evt.type in (open, openat, openat2)
output: >
Sensitive credential file accessed (user=%user.name file=%fd.name
pod=%k8s.pod.name ns=%k8s.ns.name cmd=%proc.cmdline)
priority: WARNING
tags: [mitre_credential_access, security]Apply the new rules by updating the Helm deployment:
helm upgrade falco falcosecurity/falco \
--namespace falco \
-f falco-values.yaml \
-f custom-rules.yamlStep 4: Trigger Detections and Verify Alerts
Validate your configuration by spinning up a lightweight test pod in the production namespace:
# Launch a temporary test pod
kubectl run test-app --namespace production --image=alpine:3.19 -- sleep 3600
# Trigger Rule 1: Launch an interactive shell
kubectl exec -it test-app -n production -- /bin/sh
# Trigger Rule 2: Attempt reading /etc/shadow inside the shell
cat /etc/shadow
# Exit the session
exitInspect Falco’s log stream to confirm the triggered detections:
kubectl logs -n falco -l app.kubernetes.io/name=falco -c falco --tail=50 | grep "Shell spawned"Falco outputs a structured JSON event showing the exact command, container image, pod name, and originating user within seconds, giving your incident response team actionable context immediately.
