Posted inSecurity

Hardening etcd and Kubernetes API Server: Encryption at Rest, mTLS, and Credential Leak Detection

Security tutorial - IT technology blog
Security tutorial - IT technology blog

The Silent Risk Sitting in Your etcd

After my server got hit by SSH brute-force attacks at midnight, I developed the right kind of security paranoia. So when I started managing production Kubernetes clusters, the first thing I did wasn’t deploy workloads — I went straight for the control plane and asked: what happens if someone gets into etcd?

The answer is uncomfortable. etcd is the brain of Kubernetes. It holds every Secret, ConfigMap, ServiceAccount token, and cluster state change ever written. By default, on many cluster bootstrapping setups — including some kubeadm defaults — Secrets are stored as base64-encoded text. Not encrypted. Just encoded. Anyone with etcd access can read your database passwords, TLS private keys, and API tokens with a single etcdctl get command.

Exposed etcd endpoints show up in real breach post-mortems as initial access vectors — not hypotheticals, but documented incidents. Getting this right requires deliberate configuration that most teams skip in the rush to get something running. Here’s what I do on every cluster: encryption at rest, verified mTLS across control plane components, and a credential leak scan. All three, every time.

Core Concepts Before Touching Any Configs

Encryption at Rest vs. In Transit

These solve two separate problems. In transit means data is encrypted as it moves between components — kube-apiserver talking to etcd over TLS, kubelet reporting node status over HTTPS. Kubernetes handles this reasonably well by default when bootstrapped correctly.

At rest means the data sitting on disk inside etcd is encrypted. This is off by default. If someone grabs the etcd disk volume — snapshot exfiltration, compromised VM, rogue cloud snapshot — they get plaintext Secrets. Enabling at-rest encryption uses an EncryptionConfiguration resource that tells kube-apiserver to encrypt specific resource types before writing to etcd.

Mutual TLS in the Control Plane

Standard TLS proves the server’s identity to the client. Mutual TLS (mTLS) goes both ways — both sides present certificates and verify each other. Inside a Kubernetes control plane:

  • kube-apiserver connects to etcd using a client certificate
  • etcd only accepts connections from clients presenting a cert signed by its trusted CA
  • kubelet presents a node certificate to kube-apiserver for every API call

kubeadm-bootstrapped clusters configure this correctly. In custom or managed environments, though, it’s easy to end up with missing flags, wrong CA chains, or silently expired certs. They look functional. They aren’t.

Where Credentials Actually Leak

Credentials escape a control plane through three main paths: plaintext data in etcd, JWT service account tokens mounted into pods that don’t need them, and kubeconfig files with long-lived admin credentials copied across machines. All three need active scrutiny.

Hands-On: Locking Down the Control Plane

Step 1 — Enable Encryption at Rest for Secrets

Create an encryption configuration file on your control plane node. This defines which resources get encrypted and with what provider:

# /etc/kubernetes/encryption-config.yaml
apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources:
      - secrets
      - configmaps
    providers:
      - aescbc:
          keys:
            - name: key1
              secret: <BASE64_ENCODED_32_BYTE_KEY>
      - identity: {}

Generate the 32-byte key:

head -c 32 /dev/urandom | base64

On a kubeadm cluster, edit the kube-apiserver static pod manifest at /etc/kubernetes/manifests/kube-apiserver.yaml to reference the config file:

spec:
  containers:
  - command:
    - kube-apiserver
    - --encryption-provider-config=/etc/kubernetes/encryption-config.yaml
    volumeMounts:
    - mountPath: /etc/kubernetes/encryption-config.yaml
      name: encryption-config
      readOnly: true
  volumes:
  - hostPath:
      path: /etc/kubernetes/encryption-config.yaml
      type: File
    name: encryption-config

After kube-apiserver restarts, new Secrets are encrypted. Existing ones aren’t re-encrypted automatically — force a rewrite:

kubectl get secrets --all-namespaces -o json | kubectl replace -f -

Verify the encryption is actually working by reading a Secret directly from etcd:

ETCDCTL_API=3 etcdctl \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  get /registry/secrets/default/my-secret | hexdump -C | head -5

If you see k8s:enc:aescbc:v1:key1 as a prefix in the output, encryption is active. If you see readable text, it isn’t.

Step 2 — Verify mTLS Between etcd and kube-apiserver

Check the running kube-apiserver flags to confirm it’s presenting client certificates to etcd:

ps aux | grep kube-apiserver | tr ' ' '\n' | grep '\-\-etcd-'

You should see --etcd-cafile, --etcd-certfile, and --etcd-keyfile all pointing to real files. If any are missing, etcd is accepting unauthenticated connections from the API server.

Certificate expiry is the one that quietly bites teams:

# Check all control plane cert expiry dates
kubeadm certs check-expiration

# Manual check for etcd certs specifically
openssl x509 -in /etc/kubernetes/pki/etcd/server.crt -noout -dates
openssl x509 -in /etc/kubernetes/pki/apiserver-etcd-client.crt -noout -dates

Next, verify etcd isn’t accidentally listening on all interfaces:

ss -tlnp | grep 2379

If you see 0.0.0.0:2379, etcd is exposed to the network. The --listen-client-urls flag in etcd’s config should be bound to 127.0.0.1 or the cluster’s internal IP only.

Step 3 — Scan for Leaked Credentials in Secrets

Even with encryption at rest enabled, credentials can leak through other paths. Start by mapping what’s actually stored in Secrets across the cluster:

# List all secrets and their keys across namespaces
kubectl get secrets --all-namespaces -o json | \
  jq -r '.items[] | .metadata.namespace + "/" + .metadata.name + ": " + \
  (.data | if . then (keys | join(", ")) else "(no data)" end)'

For Secrets you want to inspect manually:

kubectl get secret my-secret -n default \
  -o jsonpath='{.data.password}' | base64 -d

During incident response, I use ToolCraft’s Base64 Encoder/Decoder for quick manual inspections. It runs entirely in the browser — nothing leaves your machine. That matters when you’re handling sensitive encoded values. Pasting secrets into an online tool that logs input is exactly the exposure you’re trying to prevent.

Beyond secrets themselves, check which pods are receiving service account tokens they might not need:

# Find pods where automount is not explicitly disabled
kubectl get pods --all-namespaces -o json | \
  jq -r '.items[] | select(.spec.automountServiceAccountToken != false) | \
  .metadata.namespace + "/" + .metadata.name'

When you pull a JWT token from a running pod for inspection:

kubectl exec -n default my-pod -- \
  cat /var/run/secrets/kubernetes.io/serviceaccount/token

The ToolCraft JWT Decoder lets you read the claims, expiry, and issuer directly in-browser without sending the token anywhere. Useful for confirming whether a token scope is appropriately limited or when its expiry is.

Step 4 — Audit RBAC for Credential Blast Radius

A leaked credential is only as dangerous as the permissions attached to it. Check for service accounts with cluster-admin:

kubectl get clusterrolebindings -o json | \
  jq -r '.items[] | select(.roleRef.name == "cluster-admin") | \
  .metadata.name + ": " + ([.subjects[]?.name] | join(", "))'

Any service account with cluster-admin that isn’t explicitly documented and justified is a risk sitting in your cluster right now.

Making This Sustainable

Clusters drift. Certificates expire, new Secrets get added without a second thought, RBAC bindings accumulate quietly over months. I’ve built three recurring checks into my cluster runbooks:

  1. Monthly: run kubeadm certs check-expiration and flag any cert expiring within 90 days
  2. After any new deployment: verify the workload’s ServiceAccount isn’t mounting tokens it doesn’t use (automountServiceAccountToken: false in the pod spec)
  3. Quarterly: re-scan cluster-admin bindings and audit etcd encryption is still active

It’s the same discipline I apply to SSH key rotation and Fail2Ban thresholds on every Linux host — just pointed at the Kubernetes control plane instead.

Exposed etcd has been the entry point in real production incidents. The mitigations are built into Kubernetes itself, the verification steps take minutes, and the commands above cover the entire surface. Start with encryption at rest — it delivers the highest impact with the least risk to a running cluster.

Share:
Tags: