Debug Kubernetes Like a Pro: Real-Time Observability with Inspektor Gadget

The Invisible Failures of Kubernetes

We’ve all been there: a microservice throws a 500 Internal Server Error, but the application logs are empty. You check kubectl logs, and it shows nothing. You check Prometheus, and the CPU usage looks fine. Yet, the service is clearly broken. This is the reality of debugging distributed systems—the most critical failures often happen in the gaps between your application code and the underlying kernel.

DevOps engineers frequently run into these three “black box” scenarios:

• A container crashes instantly with an exit code 137 or 139, leaving no time for log aggregation to pick up the stack trace.
• Intermittent 50ms latency spikes that only happen during high-concurrency database writes.
• Silent permission errors where a process tries to write to a read-only volume, but the application swallows the error message.

Why Traditional Tools Fail in Production

Standard Linux tools like strace or tcpdump weren’t built for the cloud-native era. If you try to run strace on a high-traffic production process, you might see the application slow down by 10x or even 50x. This performance hit often makes the bug disappear while you’re watching it—a classic “Heisenbug.” More importantly, these tools see Process IDs (PIDs), not Pod names or Namespaces. Mapping a raw PID back to a specific container in a cluster of 500 nodes is a manual nightmare.

The eBPF Advantage: Inspektor Gadget

Inspektor Gadget solves this by using eBPF (Extended Berkeley Packet Filter). Instead of intercepting every system call and context-switching between the kernel and user space, it runs sandboxed programs directly inside the Linux kernel. This reduces overhead to nearly zero—typically less than 1%—making it safe to use even under heavy production loads.

Feature	Traditional Tools (strace/tcpdump)	Inspektor Gadget (eBPF)
Performance Hit	Severe (can stall the app)	Negligible (<1% CPU overhead)
K8s Context	Manual mapping required	Native (filters by Pod, Label, Namespace)
Installation	Must be inside the container image	Zero instrumentation (lives on the node)
Security	High risk of crashing processes	Verified safe by the Kernel JIT compiler

What to Consider Before Starting

The Good: You don’t have to change a single line of code or rebuild your Docker images. It just works from the outside looking in.

The Reality Check: You need a Linux kernel version of 5.10 or higher to access the full suite of gadgets. Since it hooks into the kernel, you also need administrative (privileged) access to the cluster to deploy the agent.

Getting Your Environment Ready

In my experience, the cleanest way to manage this is via the kubectl plugin. It allows you to treat debugging commands just like any other kubectl operation.

1. Install the CLI

If you have Krew installed, you can get up and running in seconds:

kubectl krew install gadget
kubectl gadget version

2. Deploy the DaemonSet

Inspektor Gadget needs a “manager” on each node to load the eBPF programs. Run this command to deploy the necessary resources into the gadget namespace:

kubectl gadget deploy

Wait about 30 seconds, then verify that the pods are Running across your nodes.

Real-World Debugging Scenarios

Let’s move past the theory and solve some actual problems.

Catching Rogue Processes

Ever wonder if a container has been compromised or is running unauthorized scripts? The trace exec gadget captures every new process the moment it starts.

kubectl gadget trace exec --namespace production --pod-label app=payment-gw

This command will stream real-time data showing exactly what commands are being executed. If you see curl http://malicious-site.com or /bin/sh inside a container that should only be running a Java binary, you’ve found a major security hole.

Solving “Connection Refused” Mysteries

Networking in Kubernetes is complex. When a service fails to connect, is it a DNS issue, a NetworkPolicy block, or a dropped packet? Use trace tcp to see the handshake in real-time:

kubectl gadget trace tcp --namespace my-app

You’ll see the source IP, the destination port, and the TCP state. If you see a SYN sent but no SYN-ACK returned, your packets are likely being dropped by a firewall or a misconfigured Service Mesh.

Identifying Permission Denied Errors

When an app says “File not found,” but you can see the file when you exec into the pod, it’s usually a permission mismatch. The trace open gadget shows every file access attempt and its return code:

kubectl gadget trace open --namespace backend --pod-label component=db

Look for return codes like -13 (EACCES). It tells you exactly which file path caused the error, saving you from hours of manual ls -la checks.

Automating Least-Privilege Security

The advise network-policy gadget is a massive time-saver. Instead of manually writing YAML for NetworkPolicies, you can record your app’s actual traffic and let the tool generate the policy for you.

1. Record: Run kubectl gadget advise network-policy monitor --namespace staging --output ./traffic.json for 10 minutes while running your integration tests.
2. Generate: Run kubectl gadget advise network-policy report --input ./traffic.json.

It outputs a production-ready Kubernetes NetworkPolicy that only allows the specific traffic your app actually needs.

Operating Safely in Production

Even though eBPF is efficient, you shouldn’t leave these tracers running forever. I recommend these guardrails:

• Filter Aggressively: Never run a wide-open trace on a whole cluster. Always use --namespace or --pod-label to keep the data volume manageable.
• Targeted Sessions: Use gadgets like a surgeon’s scalpel. Turn them on to identify the bug, then turn them off. For long-term monitoring, stick to standard metrics.
• RBAC Control: Ensure only authorized SREs can run kubectl gadget, as it provides deep visibility into sensitive system data.

The Bottom Line

Inspektor Gadget turns the “black box” of Kubernetes into a transparent environment. By surfacing low-level kernel events and mapping them to high-level Kubernetes concepts, it eliminates the guesswork from debugging. Whether you are troubleshooting a failing TCP handshake or auditing file access, eBPF gives you the ground truth without crashing your production environment.