The 2:15 AM Outage Nobody Wants to Debug
A few months back, our on-call engineer got paged at 2:15 AM. Our production payment gateway was suddenly dropping every checkout with ERR_CERT_DATE_INVALID. Within ninety seconds, transaction volume cratered from 140 orders per minute to zero. Web browsers flashed full-screen security warnings, and our native mobile apps severed API connections instantly due to certificate pinning.
We scrambled onto a bridge call assuming an edge proxy had crashed or a rollout had broken routing. It took twenty frantic minutes to spot the real culprit. An SSL certificate on our primary Ingress controller had quietly expired at midnight. The platform engineer who originally generated and rotated that wildcard certificate had switched teams four months prior. His recurring calendar invite sat buried in an unread mailbox.
Treating SSL/TLS lifecycles as a manual checklist on dynamic clusters guarantees downtime. It is not a matter of if, but when.
Root Cause: Why Manual TLS Fails in Kubernetes
Manual rotations work fine for three static VMs behind a hardware appliance. Kubernetes is a different beast entirely. Between fast-moving microservices, decoupled namespaces, and transient preview environments, hand-rolled TLS breaks down quickly:
- Shorter Certificate Lifespans: Major root programs dropped maximum validity from five years down to 398 days, then to 90 days. Current CA/Browser Forum proposals aim for 45-day lifespans. Renewing certs multiple times a year by hand multiplies human error.
- Namespace Secret Sprawl: Kubernetes isolates secrets by namespace. When engineers copy a wildcard
Secretacross 15 separate namespaces, drift happens fast. You end up with mismatched versions, forgotten duplicates, and silent breakages during rotation. - Brittle Operational Memory: Calendar pings, spreadsheets, and external uptime monitors are reactive. Once an alert triggers, an engineer still has to cut a private key, generate a CSR, pass challenge records, and base64-encode the payload into Kubernetes.
- Silent Ingress Failures: Pasting an incomplete certificate chain—or forgetting an intermediate bundle—leads to subtle TLS handshake failures on older mobile devices, often without surfacing any error in pod status checks.
Three Ways Teams Handle Kubernetes TLS
Before standardizing our cluster infrastructure, we weighed three common approaches for managing ingress encryption:
1. Edge Offloading at the Cloud Load Balancer (AWS ALB / Cloudflare / GCP HTTPS Proxy)
Terminate TLS at the cloud provider’s external load balancer before traffic ever reaches worker nodes.
- Pros: Zero certificate manifests inside Kubernetes; public CAs are handled by the cloud provider.
- Cons: Traffic traverses the VPC as unencrypted HTTP; limited portability across hybrid or bare-metal clusters; tricky to automate for dynamic multi-tenant subdomains.
2. Scheduled CronJobs Running Certbot
Run a scheduled in-cluster container with Certbot to solve HTTP-01 challenges, export keys, and execute kubectl create secret tls ... --dry-run=client -o yaml | kubectl apply -f -.
- Pros: Does not require installing custom resource definitions (CRDs).
- Cons: Shell scripts fail silently; wide RBAC permissions must be granted to the job across namespaces; zero reconciliation loop if someone accidentally deletes a secret.
3. Ingress-Native Automation with cert-manager
Run cert-manager as an in-cluster controller. It extends the Kubernetes API with resources like Issuer, ClusterIssuer, and Certificate.
- Pros: Declarative and GitOps-ready; binds directly to Ingress annotations; handles renewals automatically 30 days before expiration; supports HTTP-01 and DNS-01 challenge solvers.
- Cons: Adds three lightweight controller pods and CRDs to the cluster.
Complete Implementation: cert-manager + Let’s Encrypt
Pairing cert-manager with a cluster-wide ClusterIssuer provides the cleanest setup. The controller watches Ingress resources and provisions certificates automatically whenever it detects a TLS host rule.
Step 1: Install cert-manager with Helm
Deploy cert-manager using its official Helm chart. Enabling CRD installation directly in the values flag keeps custom resources in sync with the chart version:
# Add the Jetstack Helm repository
helm repo add jetstack https://charts.jetstack.io
helm repo update
# Install cert-manager and its CRDs
helm install cert-manager jetstack/cert-manager \
--namespace cert-manager \
--create-namespace \
--set crds.enabled=true
# Confirm all controller pods are running
kubectl get pods -n cert-managerWithin 30 seconds, you should see three running pods: cert-manager, cert-manager-cainjector, and cert-manager-webhook.
Step 2: Configure Staging and Production ClusterIssuers
A standard Issuer is scoped to a single namespace. A ClusterIssuer operates cluster-wide, serving Ingress objects in any namespace. Let’s Encrypt enforces strict production rate limits: 50 certificates per registered domain per week, and 5 duplicate certificates per week. Always test your ACME solver pipeline against their staging endpoint first.
Create a file named cluster-issuers.yaml:
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
name: letsencrypt-staging
spec:
acme:
server: https://acme-staging-v02.api.letsencrypt.org/directory
email: [email protected]
privateKeySecretRef:
name: letsencrypt-staging-account-key
solvers:
- http01:
ingress:
ingressClassName: nginx
---
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
name: letsencrypt-prod
spec:
acme:
server: https://acme-v02.api.letsencrypt.org/directory
email: [email protected]
privateKeySecretRef:
name: letsencrypt-prod-account-key
solvers:
- http01:
ingress:
ingressClassName: nginxApply the issuers to the cluster:
kubectl apply -f cluster-issuers.yaml
# Check registration status
kubectl get clusterissuerVerify that both issuers display READY: True under the status column.
Step 3: Annotate Your Ingress Resource
You rarely need to write standalone Certificate manifests by hand. Simply attach the cert-manager.io/cluster-issuer annotation to your Ingress. The controller reads your TLS block, submits the challenge, and stores the resulting key pair in the specified secret.
Here is an example Ingress manifest (payment-ingress.yaml):
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: payment-service-ingress
namespace: production
annotations:
# Instruct cert-manager to provision via Let's Encrypt Production
cert-manager.io/cluster-issuer: "letsencrypt-prod"
# Force HTTP to HTTPS redirection
nginx.ingress.kubernetes.io/ssl-redirect: "true"
spec:
ingressClassName: nginx
tls:
- hosts:
- api.example.com
secretName: payment-service-tls
rules:
- host: api.example.com
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: payment-service
port:
number: 8080Apply the manifest:
kubectl apply -f payment-ingress.yamlStep 4: Verify the ACME Challenge and Secret Generation
cert-manager monitors the annotation and spins up a temporary challenge pod alongside an ephemeral Ingress path at /.well-known/acme-challenge/<token>. Let’s Encrypt verifies domain ownership against that endpoint, validates the challenge, and hands back the signed certificate bundle.
Track issuance progress through each reconciliation stage:
# 1. Check issuance state
kubectl get certificate -n production
# 2. Inspect the ACME CertificateRequest
kubectl get certificaterequest -n production
# 3. View challenge state (should resolve within 15-45 seconds)
kubectl get challenges -n production
# 4. Debug failures if the challenge stays pending
kubectl describe order -n productionWhen issuance finishes, kubectl get certificate -n production prints READY: True. Check the newly populated TLS secret:
kubectl get secret payment-service-tls -n production -o yamlThe secret holds tls.crt and tls.key. Ingress-NGINX detects the updated secret immediately via its informer cache and reloads its SSL context with zero dropped connections.
Step 5: Test Automated Renewals
Let’s Encrypt certificates expire in 90 days. cert-manager initiates automated background renewals when 30 days of validity remain (at day 60). No cron jobs. No manual intervention.
You can test renewal behavior right away instead of waiting two months. Force an immediate renewal using the cmctl CLI tool:
# Trigger renewal on the target certificate
cmctl renew payment-service-tls -n productionStream controller logs to watch the renewal loop complete:
kubectl logs -n cert-manager -l app.kubernetes.io/name=cert-manager -fA declarative ClusterIssuer, automated Ingress annotations, and Kubernetes reconcilers take humans out of certificate rotation entirely. Set it up once, and your team can sleep through the night.
