Posted inDevOps

Kubernetes Secret Management with External Secrets Operator (ESO): Syncing Secrets from AWS Secrets Manager and HashiCorp Vault

Storing plaintext API keys, database credentials, or TLS certificates inside Kubernetes manifests remains one of the fastest paths to a security incident. A common pitfall is committing a base64-encoded Kubernetes Secret to Git, assuming that obfuscation equals encryption. It does not. Base64 is merely an encoding scheme. Anyone with repository read access can decode your production credentials with a single shell command in less than two seconds.

This highlights a clear conflict in modern infrastructure. GitOps requires declarative, version-controlled state. Security compliance, on the other hand, demands that secrets live in hardened vaults with granular audit logs and automated rotation. Bridging this gap without leaking credentials into commit history requires a reliable synchronization layer.

Comparing Kubernetes Secret Management Approaches

Platform engineers typically weigh three primary strategies when handling cluster secrets:

1. Sealed Secrets (Bitnami)

Sealed Secrets uses asymmetric cryptography. You encrypt raw values client-side using a public key tied to an in-cluster controller, then commit the generated SealedSecret resource to Git. The controller decrypts the payload into a native Kubernetes Secret using its private key.

2. Secrets Store CSI Driver

The Secrets Store CSI Driver mounts external secrets directly into Pod volumes via Container Storage Interface (CSI) plugins. Secrets stream straight from external backends—such as AWS Secrets Manager, HashiCorp Vault, Azure Key Vault, or Google Secret Manager—into volatile memory inside the pod.

3. External Secrets Operator (ESO)

External Secrets Operator is a dedicated Kubernetes operator. It polls metadata from external APIs and generates standard v1/Secret resources inside your target namespaces.

Pros & Cons of Each Strategy

Every pattern involves specific operational trade-offs:

Sealed Secrets

  • Pros: Self-contained within Git; requires no cloud IAM plumbing; runs without vendor lock-in.
  • Cons: Secret rotation requires manual re-encryption and git commits; controller private key backups introduce single points of failure during disaster recovery; offers no integration with corporate secret engines.

Secrets Store CSI Driver

  • Pros: Credentials bypass etcd by default; payloads sit in memory-backed tmpfs volumes inside pods.
  • Cons: Tightly tied to pod lifecycles; requires volume mounts even for apps expecting simple environment variables; mirroring values back to native Secrets requires enabling alpha/beta controller flags and adds configuration overhead.

External Secrets Operator (ESO)

  • Pros: Generates standard v1/Secret objects, ensuring complete compatibility with existing Helm charts and Deployments; connects to multiple backends concurrently; provides built-in templating and configurable polling intervals; runs decoupled from pod start sequences.
  • Cons: Secrets persist in etcd (requiring KMS etcd encryption at rest); introduces an active controller that must be monitored and maintained.

Recommended Architecture: ESO with Enterprise Backends

Across production clusters managing hundreds of microservices, ESO consistently delivers the cleanest boundary between operations and development. Security teams centrally control IAM policies and lifecycle rules in AWS Secrets Manager or Vault. Meanwhile, application developers write concise ExternalSecret manifests right alongside their standard workloads.

When drafting YAML definitions for SecretStores and ExternalSecrets, verifying key hierarchies beforehand saves time. If you need to convert JSON payloads into YAML or test structured dictionaries, check out the ToolCraft YAML to JSON Converter.

ToolCraft runs all transformations entirely client-side in your browser without transmitting payloads, making it safe for mocking non-production schemas. To quickly inspect existing cluster secrets or verify base64 strings locally, the Base64 Encoder/Decoder is equally fast and secure.

Step-by-Step Implementation Guide

Step 1: Install External Secrets Operator

Deploy the ESO Helm chart into your cluster:

helm repo add external-secrets https://charts.external-secrets.io
helm repo update

helm install external-secrets \
  external-secrets/external-secrets \
  -n external-secrets \
  --create-namespace \
  --set installCRDs=true

Confirm that the controller, webhook, and cert-controller pods are up and healthy:

kubectl get pods -n external-secrets

Step 2: Connect AWS Secrets Manager via SecretStore

IAM Roles for Service Accounts (IRSA) provides secure token exchange on AWS EKS without hardcoded credentials. Grant your IAM role secretsmanager:GetSecretValue and secretsmanager:DescribeSecret permissions, bind it to your ServiceAccount, and declare a SecretStore:

cat <<EOF | kubectl apply -f -
apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: aws-secrets-backend
  namespace: default
spec:
  provider:
    aws:
      service: SecretsManager
      region: us-east-1
      auth:
        jwt:
          serviceAccountRef:
            name: eso-aws-sa
EOF

Step 3: Connect HashiCorp Vault via SecretStore

For HashiCorp Vault, the native Kubernetes auth method maps pod service accounts directly to Vault policies. This removes the need to juggle static root tokens:

cat <<EOF | kubectl apply -f -
apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: vault-backend
  namespace: default
spec:
  provider:
    vault:
      server: "https://vault.internal.corp:8200"
      path: "secret"
      version: "v2"
      auth:
        kubernetes:
          mountPath: "kubernetes"
          role: "app-role"
          serviceAccountRef:
            name: default
EOF

Step 4: Declare ExternalSecret Resources

Next, define an ExternalSecret manifest to pull values and shape the resulting Kubernetes Secret. If your upstream vault stores JSON dictionaries, format and inspect keys using the ToolCraft JSON Formatter before writing mappings.

cat <<EOF | kubectl apply -f -
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: payment-service-secret
  namespace: default
spec:
  refreshInterval: "1h"
  secretStoreRef:
    name: aws-secrets-backend
    kind: SecretStore
  target:
    name: payment-api-secret
    creationPolicy: Owner
  data:
    - secretKey: DB_PASSWORD
      remoteRef:
        key: production/payment-service
        property: password
    - secretKey: API_KEY
      remoteRef:
        key: production/payment-service
        property: api_key
EOF

Step 5: Verify Synchronization and Bind to Pods

Inspect the sync status of your resource:

kubectl get externalsecret payment-service-secret
kubectl describe externalsecret payment-service-secret

Look for Status: SecretSynced in the output. Then confirm that ESO populated the native secret:

kubectl get secret payment-api-secret -o yaml

Your application can now bind the synchronized keys as standard environment variables:

cat <<EOF | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  name: payment-service
  namespace: default
spec:
  replicas: 2
  selector:
    matchLabels:
      app: payment-service
  template:
    metadata:
      labels:
        app: payment-service
    spec:
      containers:
        - name: app
          image: nginx:alpine
          env:
            - name: DB_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: payment-api-secret
                  key: DB_PASSWORD
EOF

Whenever credentials rotate upstream, ESO retrieves the updated values within the configured refreshInterval. Pair this setup with a controller like Stakater Reloader to automatically trigger rolling pod updates the moment downstream secrets change.

Share:
Tags: