Comparing Runtime Security Approaches in Kubernetes
At 2:15 AM last Tuesday, our alert channel woke the entire on-call rotation. An attacker had exploited a path traversal flaw in a customer-facing service, opened an interactive bash session inside an unprivileged pod, and initiated an Nmap sweep across the 10.244.0.0/16 pod CIDR.
Trivy had cleared that identical image just two hours earlier with zero critical CVEs. Static scanning tells you what software vulnerabilities exist on disk. Runtime monitoring tells you what an adversary is actively executing in memory.
Engineering teams tackling container runtime defense typically evaluate three distinct architectures:
- LSM and Profile Enforcement (AppArmor / Seccomp): Restricting syscalls and Linux capabilities up front by attaching static profiles directly to the container security context.
- Kubernetes Audit Logging: Streaming kube-apiserver events to OpenSearch or Datadog to capture administrative changes across the control plane.
- Kernel Tracing Engines (Falco / eBPF): Hooking directly into kernel system calls to inspect processes, socket creations, and filesystem writes the instant they happen.
Trade-offs Across Approaches
1. AppArmor and Seccomp
Pros: Enforces rules at zero telemetry cost. Blocks unauthorized syscalls outright rather than simply pinging an alert channel.
Cons: Managing custom profiles for 80+ rapidly evolving microservices drains engineering time. Block one unexpected clone3 or epoll_pwait syscall, and your production pod crashes silently under traffic spikes.
2. Kubernetes Audit Log Ingestion
Pros: Operates entirely out of band. It places zero agent overhead on worker nodes while reliably tracking RBAC escalation and Secret reads.
Cons: Blind to workload internals. When an attacker delivers a staged binary through an existing TCP socket and executes it, the Kubernetes control plane records nothing.
3. Kernel Tracing with Falco (eBPF)
Pros: Delivers granular visibility into raw system events like execve, connect, and openat. Falco enriches every event with Kubernetes metadata including namespace, pod name, and container hash. It catches zero-day attacks that static CI checks cannot anticipate.
Cons: Requires node-level kernel instrumentation and privileged daemon access. Poorly tuned filters can also spike node memory usage.
Recommended Architecture: Modern eBPF DaemonSet with Falcosidekick
Legacy kernel modules frequently caused headaches. A minor kernel patch on Ubuntu or Amazon Linux often broke driver compilation during node auto-scaling events, occasionally triggering kernel panics under heavy I/O. The modern eBPF probe solves this issue. It uses BTF (BPF Type Format) and CO-RE (Compile Once – Run Everywhere) on Linux kernels 5.8 and newer. You get near-instant startup, stable instrumentation, and a predictable footprint under 1.5% CPU and 150MB of RAM per worker node.
Falco acts as the sensor engine on each node. Falcosidekick sits alongside it as a lightweight multiplexer, routing structured security events to Slack, PagerDuty, or security data lakes.
Before installing runtime agents, ensure your base host access is locked down. For emergency root access and jump-host setup, generate strong node credentials with the client-side password generator at toolcraft.app/en/tools/security/password-generator. It computes passwords entirely in the browser without sending entropy across the network. With baseline access secured, Falco catches anything that slips past the perimeter.
Implementation Guide: Installing and Tuning Falco
Step 1: Register the Falco Helm Repository
Fetch the official charts and update local index caches:
helm repo add falcosecurity https://falcosecurity.github.io/charts
helm repo updateStep 2: Configure the Helm Values
Create falco-values.yaml. Configure the modern eBPF probe, enable structured JSON formatting, and point alerts to your team channels via Falcosidekick:
driver:
kind: modern_ebpf
falco:
json_output: true
log_stderr: true
log_level: notice
priority: notice
rules_file:
- /etc/falco/falco_rules.yaml
- /etc/falco/falco_rules.local.yaml
- /etc/falco/rules.d
falcosidekick:
enabled: true
webui:
enabled: true
config:
slack:
webhookurl: "https://hooks.slack.com/services/T00000000/B00000000/XXXXXXXXXXXXXXXXXXXXXXXX"
minimumpriority: "warning"Step 3: Deploy the DaemonSet
Install the chart into an isolated namespace:
kubectl create namespace falco
helm install falco falcosecurity/falco \
--namespace falco \
--values falco-values.yamlConfirm every worker node has scheduled an operational probe pod:
kubectl get pods -n falco -o wideStep 4: Craft Targeted Rules for Production Namespaces
Default community rules flag every terminal shell across the cluster. In practice, background CI runners, sidecar sync scripts, and maintenance jobs will trigger dozens of false positives each day. Target high-value environments instead, such as transaction processing or database tiers.
Define your overrides in custom-rules.yaml:
customRules:
rules-payments.yaml: |-
- rule: Unauthorized Shell in Payment Namespace
desc: Triggers when any shell process starts within the payments workload boundary
condition: >
spawned_process and
container and
k8s.ns.name = "payments" and
shell_procs
output: >
CRITICAL: Shell initiated in restricted workload
(user=%user.name cmd=%proc.cmdline pod=%k8s.pod.name
ns=%k8s.ns.name container=%container.name image=%container.image.repository)
priority: CRITICAL
tags: [container, execution, pci_dss]Push the updated configuration using Helm upgrade:
helm upgrade falco falcosecurity/falco \
--namespace falco \
--reuse-values \
-f custom-rules.yamlStep 5: Validate Threat Detection
Never assume an alert pipeline works until you see it fire. Spawn a lightweight test container inside the target namespace and execute a shell session:
kubectl run validation-target --image=alpine:3.19 --namespace payments -- sleep 600
kubectl exec -it validation-target --namespace payments -- /bin/shTail the active Falco log stream from a second shell window:
kubectl logs -n falco -l app.kubernetes.io/name=falco -c falco --tail=10Falco surfaces the event within 15 milliseconds, formatting the output into clean JSON:
{
"output": "CRITICAL: Shell initiated in restricted workload (user=root cmd=/bin/sh pod=validation-target ns=payments container=validation-target image=alpine:3.19)",
"priority": "Critical",
"rule": "Unauthorized Shell in Payment Namespace",
"time": "2026-10-25T02:18:12.312918451Z"
}Step 6: Eliminate Alert Fatigue with Macro Exceptions
Security tools that generate noise end up muted. If an automated backup CronJob legitimately runs a shell script at 3:00 AM, do not disable the detection rule. Append a macro filter to exempt the specific binary and execution path:
- macro: allowed_backup_executions
condition: >
(proc.cmdline startswith "/opt/scripts/backup.sh" and
k8s.pod.name startswith "postgres-backup-")
- rule: Unauthorized Shell in Payment Namespace
condition: >
spawned_process and
container and
k8s.ns.name = "payments" and
shell_procs and
not allowed_backup_executionsReview alerts daily during your first two weeks. Fine-tuning macro boundaries transforms Falco from a noisy log generator into an authoritative, low-latency intrusion detection barrier.
