Real-World Problem: When iptables Hits the Scaling Wall
Last autumn, our primary Kubernetes cluster grew past 500 microservices across 40 bare-metal worker nodes. Total CPU and memory utilization looked calm at 45%. Yet our checkout and billing services were getting hammered by sudden latency spikes. During routine 15,000 req/sec traffic bursts, our p99 response times surged from an 8ms baseline all the way past 240ms.
Our monitoring dashboards flagged three critical issues:
- Worker node CPUs showed elevated kernel-space time (
system %), even when application throughput was modest. - Service routing updates took 20 to 45 seconds to propagate across all nodes whenever the Horizontal Pod Autoscaler (HPA) added replicas.
- Hunting down dropped packets meant attaching
tcpdumpto ephemeral pods during active incidents, which burned valuable debugging time.
The network subsystem was grinding to a halt under frequent pod churn.
Root Cause: Where Kube-Proxy Breaks Down at Scale
To fix this, we dug into standard Kubernetes networking internals. By default, kube-proxy programs Linux iptables (or IPVS) to direct incoming Service traffic to pod endpoints.
1. O(N) Sequential Packet Traversal
With iptables, the Linux kernel inspects firewall rules sequentially. If your cluster hosts 5,000 backend endpoints, a single inbound packet may evaluate thousands of string matches before hitting the correct target. This linear evaluation burns compute cycles directly in the kernel’s network fast path.
2. Full Table Synchronization Locks
Whenever a pod spins up, shuts down, or fails a readiness probe, kube-proxy regenerates and rewrites the entire iptables rule list. During rapid autoscaling events, the kernel spends critical processing cycles locking and copying memory buffers rather than routing application traffic.
3. Conntrack Saturation and Silent Drops
Traditional kube-proxy relies heavily on Linux connection tracking (conntrack) for Service NAT. When concurrent TCP sessions climb past 200,000, the table exhausts its allocation limit. The kernel starts throwing nf_conntrack: table full, dropping packet errors and dropping active client connections.
Comparing the Options: Calico vs. IPVS vs. Cilium eBPF
Before touching our production nodes, we ran synthetic benchmarks against three network architectures using 10,000 Service endpoints and 20,000 concurrent RPS:
| Architecture Metric | Calico (Standard iptables) | kube-proxy (IPVS Mode) | Cilium CNI (eBPF Native) |
|---|---|---|---|
| Service Lookup Complexity | O(N) sequential search | O(1) IPVS ipset hash | O(1) BPF Map direct memory lookup |
| Packet Hook Point | Netfilter hook chains | Netfilter + IPVS hooks | Socket layer & TC / XDP direct attach |
| Rule Update Latency (5k endpoints) | ~12–30 seconds | ~2–5 seconds | < 50 milliseconds (atomic map swap) |
| Layer 7 Network Telemetry | Requires Envoy sidecar injection | None built-in | Native via Hubble eBPF instrumentation |
| Kernel Overhead at Scale | High CPU consumption | Moderate | Lowest (~35% drop in node system CPU) |
IPVS solves the linear lookup problem with hash sets, but it remains tied to Netfilter conntrack overhead. Cilium takes a different path: it uses extended Berkeley Packet Filter (eBPF) to run sandboxed code directly at the Linux socket and Traffic Control (tc) hooks. Packets bypass the Netfilter stack entirely.
Step-by-Step: Deploying Cilium with Strict Kube-Proxy Replacement
We rolled out Cilium across our staging and production clusters with full kube-proxy replacement. Here is the exact installation workflow.
Step 1: Check Kernel and Mount the BPF Filesystem
Cilium requires modern eBPF helpers. Use a 5.4+ kernel (5.15+ LTS or 6.x is preferred). Ensure the BPF filesystem is mounted across all cluster nodes:
# Verify the running kernel version
uname -r
# Mount bpffs if not already present
mount | grep /sys/fs/bpf || sudo mount -t bpf bpffs /sys/fs/bpfStep 2: Remove the Existing Kube-Proxy DaemonSet
On new clusters provisioned with kubeadm, add --skip-phases=addon/kube-proxy during initialization. On an active cluster, remove the DaemonSet and wipe legacy iptables chains:
# Delete the kube-proxy DaemonSet and ConfigMap
kubectl -n kube-system delete daemonset kube-proxy
kubectl -n kube-system delete configmap kube-proxy
# Flush residual KUBE-* iptables rules across worker nodes (via SSH or Ansible)
sudo iptables-save | grep -v KUBE | sudo iptables-restoreStep 3: Install Cilium with Helm
Supply the API Server control plane endpoint explicitly so Cilium can establish cluster communication before networking pods are fully initialized:
# Add and update the Cilium Helm repository
helm repo add cilium https://helm.cilium.io/
helm repo update
# Deploy Cilium with full socket load balancing and Hubble observability
helm install cilium cilium/cilium --version 1.15.4 \
--namespace kube-system \
--set kubeProxyReplacement=true \
--set k8sServiceHost="10.0.0.100" \
--set k8sServicePort="6443" \
--set bpf.masquerade=true \
--set ipam.mode="kubernetes" \
--set hubble.enabled=true \
--set hubble.relay.enabled=true \
--set hubble.ui.enabled=true \
--set autoDirectNodeRoutes=trueReplace 10.0.0.100 with your cluster API Server load balancer VIP or primary control-plane IP.
Step 4: Verify eBPF Host Routing and Service Maps
Check the Cilium daemon status to verify that socket-level load balancing took over all routing responsibilities:
# Check overall Cilium cluster health
cilium status --wait
# Confirm that kube-proxy replacement is fully operating in strict mode
kubectl -n kube-system exec -ti ds/cilium -c cilium-agent -- cilium-dbg status | grep -A 10 "KubeProxyReplacement"The output should confirm KubeProxyReplacement: True (Strict). Cilium now manages NodePort, ClusterIP, HostPort, and external traffic directly through eBPF maps.
Step 5: Trace Traffic and Packet Drops in Real Time with Hubble
Hubble reads eBPF events directly from the kernel ring buffer, letting you inspect flows without sidecars or tcpdump overhead:
# Stream dropped packets in real time across the entire cluster
cilium hubble observe --type drop
# Trace traffic entering the checkout namespace
cilium hubble observe --namespace checkout --followProduction Results After Six Months
Migrating off kube-proxy transformed our cluster’s network stability under high load:
- Predictable Tail Latency: Payment microservice p99 latency dropped from 240ms down to a consistent 6ms during peak traffic spikes.
- Reclaimed Compute: Worker nodes recovered roughly 14% of CPU capacity that was previously wasted evaluating iptables chains and managing conntrack tables.
- Instant Autoscaling: Pod scale-outs of 100+ replicas update routes immediately via atomic BPF map insertions without causing network jitter.
If your Kubernetes infrastructure runs hundreds of services or handles thousands of concurrent connections, switching to Cilium eBPF delivers immediate performance and operational dividends.
