The Hidden Cost Nobody Talks About With Sidecars
Every pod in a traditional Istio mesh carries a sidecar — an Envoy proxy injected automatically alongside your application container. At small scale, that’s fine. At 200+ pods, you’re looking at 200 Envoy processes, each consuming 50–100 MB of RAM and a constant slice of CPU. The mesh overhead becomes a real line item in your cloud bill.
Istio isn’t the problem — the sidecar model is. One proxy per pod means N proxies for N pods, no sharing. The mesh scales linearly with workload count, not traffic volume.
Istio Ambient Mesh fixes this by removing sidecars entirely. Traffic interception and mTLS happen at the node level via a lightweight component called ztunnel, shared across all pods on that node. I’ve run this in production across clusters ranging from 50 to 300+ pods — proxy-related memory dropped noticeably from day one, pod startup got faster, and sidecar injection failures during rolling deploys became a thing of the past.
This guide walks through the ambient model hands-on: a working install in under five minutes, then L7 traffic policies with waypoint proxies.
Quick Start: Ambient Mesh in 5 Minutes
You need a Kubernetes cluster (1.28+) and kubectl configured. Start with a clean cluster — ambient mode and sidecar mode can coexist, but mixing them in the same namespace causes confusion during learning.
Step 1: Install istioctl
curl -L https://istio.io/downloadIstio | ISTIO_VERSION=1.22.0 sh -
cd istio-1.22.0/
export PATH=$PWD/bin:$PATH
istioctl versionStep 2: Deploy Istio with the Ambient Profile
The ambient profile installs the control plane plus two new components: istio-cni (DaemonSet for traffic redirection) and ztunnel (DaemonSet for L4 mTLS).
istioctl install --set profile=ambient --skip-confirmation
# Watch until all pods are Running
kubectl get pods -n istio-system -wExpected output after a minute or two:
NAME READY STATUS RESTARTS
istio-cni-node-xxxxx 1/1 Running 0
istio-ingressgateway-xxxxxxxxx-xxxxx 1/1 Running 0
istiod-xxxxxxxxx-xxxxx 1/1 Running 0
ztunnel-xxxxx 1/1 Running 0Step 3: Enroll a Namespace
Adding a namespace to the mesh takes a single label — no pod restarts, no injection webhooks.
kubectl create namespace demo
kubectl label namespace demo istio.io/dataplane-mode=ambient
# Deploy a sample app
kubectl apply -n demo -f https://raw.githubusercontent.com/istio/istio/release-1.22/samples/bookinfo/platform/kube/bookinfo.yaml
kubectl get pods -n demoPods come up with no istio-proxy container listed. The namespace is in the mesh — mTLS between pods is already active via ztunnel.
# Verify ambient is handling traffic
istioctl experimental ztunnel-config workload -n demoDeep Dive: How Ambient Mesh Actually Works
Ambient splits what a sidecar did into two separate layers, each optional and independently scoped.
Layer 1 — ztunnel (L4, Always On)
ztunnel runs as a DaemonSet, one pod per node. It handles:
- Mutual TLS for all pod-to-pod traffic on that node
- L4 authorization policies (IP, port, SPIFFE identity)
- Telemetry at the connection level (bytes in/out, connection count)
Traffic redirection is handled by istio-cni, which programs iptables rules in the pod network namespace without requiring the application container to run as root. The app knows nothing about any of this — it binds its port normally, and the kernel routes traffic through ztunnel transparently.
Under the hood, ztunnel instances speak HBONE (HTTP-Based Overlay Network Encapsulation) — HTTP/2 CONNECT tunneling with mutual TLS. It preserves source IP and plays well with existing network policies.
Layer 2 — Waypoint Proxies (L7, Optional)
Need HTTP routing, retries, circuit breaking, or header manipulation? That’s where the waypoint proxy comes in — a dedicated Envoy instance scoped to a namespace or service account. Traffic requiring L7 processing flows through the waypoint; everything else stays at L4 in ztunnel.
That’s the real value here: Envoy overhead only where you need L7 features, not baked into every pod by default.
Advanced Usage: Waypoint Proxies and Traffic Policies
Deploying a Waypoint Proxy
# Create a waypoint for the entire demo namespace
istioctl experimental waypoint apply -n demo --enroll-namespace
# Verify it's running
kubectl get pods -n demo -l istio.io/gateway-name=waypointApplying an AuthorizationPolicy
With L7 available, policies can target HTTP methods or paths — something ztunnel alone can’t do.
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
name: allow-reviews-get
namespace: demo
spec:
targetRefs:
- kind: Service
group: ""
name: reviews
action: ALLOW
rules:
- to:
- operation:
methods: ["GET"]
paths: ["/reviews/*"]kubectl apply -f authz-policy.yamlTraffic Management With VirtualService
With a waypoint in place, standard Istio VirtualService and DestinationRule objects work exactly as they do in sidecar mode.
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: reviews
namespace: demo
spec:
hosts:
- reviews
http:
- match:
- headers:
x-user:
exact: canary-tester
route:
- destination:
host: reviews
subset: v3
- route:
- destination:
host: reviews
subset: v1
weight: 90
- destination:
host: reviews
subset: v2
weight: 10Migrating From Sidecar Mode Incrementally
If you have an existing sidecar-based namespace, you can migrate without downtime:
# 1. Remove sidecar injection label
kubectl label namespace my-app istio-injection-
# 2. Add ambient label
kubectl label namespace my-app istio.io/dataplane-mode=ambient
# 3. Rolling restart to shed existing sidecar containers
kubectl rollout restart deployment -n my-app
# 4. Confirm no more istio-proxy containers
kubectl get pods -n my-app -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{range .spec.containers[*]}{.name}{" "}{end}{"\n"}{end}'Practical Tips From Running Ambient in Production
Measure Resource Savings Before Committing
Before migrating a busy namespace, snapshot current usage:
# CPU and memory per container, before migration
kubectl top pods -n my-app --containers | grep istio-proxy
# After migration — ztunnel usage per node (shared cost)
kubectl top pods -n istio-system -l app=ztunnelIn my clusters, namespaces with 20+ pods routinely dropped proxy-related memory by 60–70% after removing sidecars. The ztunnel DaemonSet uses roughly 30–50 MB per node regardless of pod count — that’s a fixed cost, not a per-workload one.
Don’t Add a Waypoint Unless You Need L7
The temptation is to add a waypoint everywhere “just in case.” Resist it. ztunnel alone gives you mTLS, peer authentication, and L4 policies — that covers the majority of security requirements. Add a waypoint only when a concrete need appears: header-based routing, HTTP fault injection, or path-level authorization. Each waypoint is a Deployment you manage.
Debugging Ambient Traffic Flow
# Check which workloads ztunnel sees
istioctl experimental ztunnel-config workload
# Inspect ztunnel's internal Envoy config on a specific node
istioctl experimental ztunnel-config all -n istio-system
# See if a waypoint proxy is intercepting a service
istioctl experimental waypoint status -n demo
# Live access log from ztunnel on a node
kubectl logs -n istio-system -l app=ztunnel -f --tail=50Compatibility Checklist
- CNI plugins: Calico, Cilium (without eBPF kube-proxy replacement), Flannel, and AWS VPC CNI are tested and supported.
- Kubernetes version: 1.28+ is recommended; 1.26 works but lacks some node feature gates.
- HostNetwork pods: Not supported in ambient mesh — use sidecar mode for those workloads specifically.
- Existing NetworkPolicy: Ambient mesh is additive. Your NetworkPolicy objects continue to work at the kernel level; ztunnel operates above them.
What Ambient Mesh Changes for Your Team
Ambient changes how you think about proxy management day to day. Upgrading sidecars used to mean coordinating rolling restarts across every deployment — manageable at 10 workloads, painful at 200. With ambient, a ztunnel upgrade is a single DaemonSet rollout, cluster-wide. L7 policies stop being something every pod silently carries and become deliberate choices scoped to specific services.
If sidecar overhead was the reason you shelved Istio, ambient addresses the core of that complaint. ztunnel gives you mTLS and an observability baseline with nearly zero configuration. Waypoints appear only where you actually need L7 — the mesh doesn’t assume you always will.
Istio 1.22 declared ambient production-ready, and the tooling reflects that. Migration docs are thorough, the debugging surface keeps improving, and the architecture has held up across several minor releases without surprises. Pick a non-critical namespace in staging, flip the label, and measure what happens to your proxy memory. The numbers tend to make the decision for you.
