Zero Trust in Practice: Scaling mTLS on Kubernetes with SPIRE and SPIFFE

Moving Beyond the Fragility of IP-Based Security

Traditional network security usually relies on the “castle and moat” strategy. We build high walls with firewalls and assume everything inside the perimeter is safe. But in a modern Kubernetes environment, IP addresses are fleeting. A single cluster might churn through 5,000 Pods a day. Relying on IP-based Access Control Lists (ACLs) in this environment is a recipe for burnout and security gaps.

Zero Trust solves this by ignoring location and focusing on identity. SPIFFE (Secure Production Identity Framework for Everyone) defines what that identity looks like. SPIRE (the SPIFFE Runtime Environment) is the engine that actually delivers those identities. After running this setup in production for half a year, I’ve found it much more reliable than managing a heavy Service Mesh like Istio just for mutual TLS (mTLS).

Setting Up SPIRE on Kubernetes

To issue identities, we need a solid foundation. In Kubernetes, this means deploying a SPIRE Server as the Certificate Authority and SPIRE Agents as a DaemonSet on every node.

Prerequisites

• A Kubernetes cluster (v1.22+ recommended).
• Helm v3 for deployment.
• A basic grasp of how Kubernetes ServiceAccounts work.

Deployment Steps

The official Helm charts are the most reliable path here. They handle the complex heavy lifting of configuring service accounts, roles, and the critical Workload API volume mounts. Start by adding the repository:

helm repo add spiffe https://spiffe.github.io/helm-charts-hardened/
helm repo update

Isolating security infrastructure is a best practice. I always deploy these components into a dedicated namespace to keep them away from general application traffic:

kubectl create namespace spire

# We typically allocate 2Gi of memory for the server in medium clusters
helm install spire spiffe/spire -n spire \
  --set spire-server.ca_key_type=rsa-2048 \
  --set spire-agent.trust_domain=prod.itfromzero.com

One hard-earned lesson: choose your trust_domain carefully. If you change example.org to internal.net later, you’ll have to re-issue every single identity in your fleet, causing a massive synchronization headache.

Orchestrating Identity: Configuration

Once the agents are live, they talk to the SPIRE Server to prove the identity of their host node. But how does a specific Pod get its unique ID? This is handled through “Workload Attestation.”

Defining the Trust Domain

Think of the Trust Domain as your root of trust, like prod.itfromzero.com. Every identity follows the SPIFFE ID format: spiffe://trust-domain/path/to/workload. It’s clean, hierarchical, and easy to audit.

Registering Workloads

Automating identity is better than manual work. The SPIRE Controller Manager watches for new Pods and creates registration entries automatically based on labels. However, for testing, you might want to see how a manual entry looks:

# Manual registration for a payment service
kubectl exec -n spire spire-server-0 -- \
/opt/spire/bin/spire-server entry create \
    -spiffeID spiffe://prod.itfromzero.com/ns/payments/sa/processor \
    -parentID spiffe://prod.itfromzero.com/spire/agent/k8s_psat/my-cluster/node-name \
    -selector k8s:ns:payments \
    -selector k8s:sa:processor

In a production environment, manual commands are a liability. I highly recommend the SPIRE Kubernetes Workload Registrar. It maps identities to Pods automatically, so your CI/CD pipeline doesn’t need to know SPIRE even exists.

Automating mTLS

The real payoff is automated mTLS. SPIRE issues SVIDs (SPIFFE Verifiable Identity Documents) as X.509 certificates. The SPIRE Agent rotates these automatically every few hours. Crucially, the private key stays in the node’s memory and never touches a Kubernetes Secret or persistent disk.

Real-World Performance and Monitoring

After 6 months, the biggest win was the total death of static credential management. We went from manually rotating 20+ secrets to zero. The system handled over 1,000 identity rotations daily without a single manual intervention.

Validating the Identity

You need to know if a Pod actually has its certificate. I use a simple check from within the application container to verify the Workload API status:

# Checking the identity from the app container
/opt/spire/bin/spire-agent api fetch x509 -socketPath /run/spire/sockets/agent.sock

A successful fetch returns the SPIFFE ID and the certificate expiration. If you see a “permission denied” error, check your selectors. Usually, the Pod is running with a different Service Account than the one you registered.

Monitoring and Logs

A Zero Trust foundation is only as good as its visibility. I focus on three specific metrics in our Grafana dashboards:

1. Registration Rate: Spikes here might indicate a deployment loop or a scaling issue.
2. Attestation Latency: We aim for under 200ms. If it climbs, the SPIRE Server is likely CPU-throttled.
3. SVID Expiration: We alert if any identity has less than 20% of its lifetime remaining.

Keep in mind that SPIRE agents cache SVIDs locally. If the SPIRE Server goes down for 10 minutes, your services won’t break immediately. They’ll keep using their cached certificates until they expire.

Lessons from 6 Months in Production

The tech works, but the mindset shift is the real challenge. Developers can’t just “curl” an internal endpoint anymore; they have to use the identities provided by the Workload API. Tools like the go-spiffe library or Envoy sidecars make this transition painless. If you want to move beyond basic firewall rules and truly harden your cluster, identity-based security with SPIRE is the most robust path forward.