Deploy External Secrets Operator on Kubernetes: Sync Vault and Cloud Secret Managers

Table of Contents

Real-World Problem: Secret Sprawl and the GitOps Dilemma

During an incident drill on our staging cluster last year, our team caught a serious leak. A developer had accidentally pushed an unencrypted Kubernetes Secret manifest to an open GitHub mirror. The file contained credentials for our staging PostgreSQL database. The data looked garbled, but Base64 is merely an encoding scheme. It offers zero encryption, and anyone with a shell can decode it in two seconds flat.

As clusters grow past a dozen microservices, handling credentials becomes messy fast:

GitOps friction: Tools like ArgoCD and Flux thrive when every state change lives in Git. Pushing plain Kubernetes Secret manifests, however, exposes sensitive passwords to anyone with repository read access.
Rotation headaches: Rotating an RDS master password or a Stripe API key shouldn’t take all morning. Patching thirty namespace-scoped secrets manually wastes time and frequently breaks live connections.
Audit trail blind spots: HashiCorp Vault and AWS Secrets Manager log every single read request. Once a secret drops into Kubernetes etcd, however, those external audit logs stop tracking which pod reads it.

Before pushing any new root credentials to our central vault, generate high-entropy strings. We use the browser-based tool at toolcraft.app/en/tools/security/password-generator for server passwords. Because it executes client-side, sensitive tokens never cross the network. Once you have strong credentials, the next challenge is injecting them into pods cleanly.

Root Cause: Why Native Kubernetes Secrets Fall Short

To understand why this setup breaks down under production load, consider how Kubernetes stores secrets internally.

1. Base64 Provides Zero Confidentiality

Kubernetes manifests store secrets under the data field as Base64 strings. Run echo "<string>" | base64 -d, and your production tokens sit in clear text on the terminal screen. If you need to inspect raw tokens during local debugging, use the Base64 Encoder/Decoder on ToolCraft. Just make sure those decoded payloads never end up committed to version control.

2. The Lifecycle Disconnect

External managers like HashiCorp Vault, AWS Secrets Manager, and GCP Secret Manager handle automatic rotation, leases, and strict IAM boundaries. Kubernetes knows nothing about those changes. When an external database password rotates at 02:00, etcd keeps serving the stale value until a CI pipeline or an engineer pushes an update.

3. Broad Cluster Read Access

Engineers often hold read-only permissions across non-production namespaces. Running kubectl get secret -o yaml immediately yields every credential in that namespace, unless you back the cluster with dedicated KMS envelope encryption and tight RBAC guards.

Evaluating the Options

Over the past four years, our infrastructure team evaluated four common patterns for bridging external vaults to Kubernetes pods.

Approach	Mechanism	Pros	Cons
Sealed Secrets (Bitnami)	Asymmetric encryption via an in-cluster controller key.	GitOps-ready; encrypted manifests sit safely in public repositories.	No integration with enterprise vaults; password rotation requires manual re-encryption.
Vault Agent Sidecar	Mutating webhook injects a sidecar container into each pod to render secrets onto shared memory volumes.	Direct dynamic leases; never generates a native Kubernetes Secret.	Adds 40–80 MB of RAM overhead per pod; apps must read file paths rather than standard environment variables.
Secrets Store CSI Driver	Mounts external provider secrets directly as local container volumes.	Official Kubernetes SIG project; wide cloud provider support.	Syncing to native `Secret` objects requires extra CRDs and flags; pods must mount the volume to trigger sync.
External Secrets Operator (ESO)	A single controller syncs external APIs into standard Kubernetes `Secret` objects.	Works across Vault, AWS, GCP, and Azure; works with standard environment variables; handles drift detection out of the box.	Leaves standard Kubernetes `Secret` objects inside the namespace.

Step-by-Step: Deploying External Secrets Operator (ESO)

For most engineering teams running GitOps pipelines, External Secrets Operator (ESO) strikes the best balance. Your applications continue reading standard environment variables, while HashiCorp Vault or Cloud Secret Manager remains the single source of truth.

Step 1: Install ESO with Helm

First, pull the official Helm chart and install the operator into its own namespace:

helm repo add external-secrets https://charts.external-secrets.io
helm repo update

helm install external-secrets \
  external-secrets/external-secrets \
  -n external-secrets \
  --create-namespace \
  --set installCRDs=true

Verify that all three core pods—the controller, the webhook, and the cert-controller—are healthy:

kubectl get pods -n external-secrets

Step 2: Configure Authentication to Your Upstream Store

ESO provides two custom resources for upstream connections: SecretStore for single namespaces and ClusterSecretStore for cluster-wide backends. Here is a production-tested configuration connecting to HashiCorp Vault with a dedicated Kubernetes ServiceAccount:

# Create a dedicated service account for Vault authentication
kubectl create serviceaccount vault-auth-sa -n production

Save this manifest as vault-secretstore.yaml:

apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: vault-backend
  namespace: production
spec:
  provider:
    vault:
      server: "https://vault.internal.mycompany.com:8200"
      path: "secret"
      version: "v2"
      auth:
        kubernetes:
          mountPath: "kubernetes"
          role: "production-app-role"
          serviceAccountRef:
            name: vault-auth-sa

Apply the file:

kubectl apply -f vault-secretstore.yaml

If you run on AWS EKS instead, point ESO to AWS Secrets Manager using IAM Roles for Service Accounts (IRSA):

apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: aws-backend
  namespace: production
spec:
  provider:
    aws:
      service: SecretsManager
      region: us-east-1
      auth:
        jwt:
          serviceAccountRef:
            name: aws-irsa-sa

Step 3: Define the ExternalSecret Custom Resource

With authentication established, define the mapping between your external keys and the native Kubernetes secret. If your external secret contains nested JSON, validate the payload structure beforehand using the JSON Formatter & Validator on ToolCraft. A single malformed bracket in Vault can block operator reconciliation across an entire namespace.

Create app-externalsecret.yaml:

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: payment-service-secret-sync
  namespace: production
spec:
  refreshInterval: "1h"
  secretStoreRef:
    name: vault-backend
    kind: SecretStore
  target:
    name: payment-service-db-credentials
    creationPolicy: Owner
  data:
    - secretKey: DB_PASSWORD
      remoteRef:
        key: production/payment-service
        property: password
    - secretKey: DB_USER
      remoteRef:
        key: production/payment-service
        property: username

Apply the manifest:

kubectl apply -f app-externalsecret.yaml

Step 4: Confirm Synchronization Status

Check the status flag on the custom resource:

kubectl get externalsecret payment-service-secret-sync -n production

Confirm that the STATUS column displays SecretSynced:

NAME                          STORE           REFRESH INTERVAL   STATUS         READY
payment-service-secret-sync   vault-backend   1h                 SecretSynced   True

Now inspect the generated Kubernetes Secret:

kubectl get secret payment-service-db-credentials -n production -o yaml

Step 5: Bind the Secret to Your Deployment

Your application pods require zero custom client SDKs or sidecars. Inject the synced keys through standard environment variables:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: payment-service
  namespace: production
spec:
  replicas: 2
  selector:
    matchLabels:
      app: payment-service
  template:
    metadata:
      labels:
        app: payment-service
    spec:
      containers:
        - name: app
          image: payment-service:v1.2.0
          env:
            - name: DATABASE_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: payment-service-db-credentials
                  key: DB_PASSWORD
            - name: DATABASE_USER
              valueFrom:
                secretKeyRef:
                  name: payment-service-db-credentials
                  key: DB_USER

Production Best Practices

A few hard-earned lessons from managing ESO across multi-tenant production clusters:

Watch polling costs: Avoid setting refreshInterval to aggressive windows like 5s. AWS Secrets Manager charges $0.05 per 10,000 API requests. Polling 50 secrets every five seconds generates over 25 million calls monthly, adding $130 in unexpected API charges. A window of 1h to 4h works well for static application credentials.
Automate pod restarts: Kubernetes does not automatically cycle pods when an underlying Secret updates. Deploy Stakater Reloader alongside ESO. Adding the reloader.stakater.com/auto: "true" annotation to your Deployment triggers a clean rolling restart the moment ESO updates the secret.
Keep Git clean: Commit only ExternalSecret and SecretStore declarations to Git. The sensitive credentials remain insulated inside your central key vault.