Posted inDevOps

Automating Kubernetes Secret Management with External Secrets Operator: Syncing AWS and Vault

DevOps tutorial - IT technology blog
DevOps tutorial - IT technology blog

The Problem with Native Kubernetes Secrets

Native Kubernetes Secrets are a bit of a misnomer. They aren’t actually encrypted; they are just Base64-encoded strings. If you commit these to a Git repository, anyone with read access can run echo 'encoded-string' | base64 --decode and own your production database in three seconds. I’ve seen many teams learn this the hard way after a junior developer accidentally pushes a .yaml file containing a Stripe API key to a public repo.

Most organizations eventually graduate to dedicated tools like AWS Secrets Manager or HashiCorp Vault. The friction starts when you try to get those values into your pods. While you could write custom Python scripts or use heavy init-containers, these methods add significant maintenance debt. Mastering the bridge between external vaults and Kubernetes is a non-negotiable skill for building a secure, modern CI/CD pipeline.

The External Secrets Operator (ESO) fills this gap perfectly. It functions as a background controller that pulls data from your provider and injects it into native Kubernetes Secrets for your apps to consume.

Understanding the Core Concepts of ESO

ESO relies on two primary custom resources to handle the heavy lifting:

  • SecretStore / ClusterSecretStore: These act as the connection string. They define how to talk to AWS, Vault, or GCP. Use a SecretStore for single-namespace isolation or a ClusterSecretStore to share one connection across the entire cluster.
  • ExternalSecret: This is the manifest that defines what to fetch. It instructs the operator to locate a specific key, like prod/db/password, and map it to a local Kubernetes Secret.

I frequently toggle between YAML and JSON formats when debugging these manifests to ensure my nested data structures are correct. Keeping a YAML ↔ JSON Converter handy is helpful here. Using a client-side tool ensures you aren’t leaking sensitive architecture patterns to a third-party server during the conversion process.

Step 1: Installing External Secrets Operator

Helm is the standard choice for installation. It simplifies version management and makes the cleanup process straightforward if you ever need to migrate.

helm repo add external-secrets https://charts.external-secrets.io

helm install external-secrets \
  external-secrets/external-secrets \
  -n external-secrets \
  --create-namespace

Verify the deployment by checking the status of the controller pods:

kubectl get pods -n external-secrets

Step 2: Connecting to AWS Secrets Manager

Avoid static AWS access keys at all costs. Instead, use IAM Roles for Service Accounts (IRSA) to grant the cluster permission to read secrets. You will need an IAM policy that specifically allows secretsmanager:GetSecretValue for your target resources.

Once your IAM role is linked to a Kubernetes ServiceAccount, apply a ClusterSecretStore like this:

apiVersion: external-secrets.io/v1beta1
kind: ClusterSecretStore
metadata:
  name: aws-secretsmgr
spec:
  provider:
    aws:
      service: SecretsManager
      region: us-east-1
      auth:
        jwt:
          serviceAccountRef:
            name: eso-service-account
            namespace: external-secrets

Step 3: Connecting to HashiCorp Vault

Vault setups usually require a bit more configuration. You can use Tokens or AppRoles, but the Kubernetes Auth Method is the cleanest for internal cluster traffic. Below is a standard configuration for a Kubernetes-native auth mount:

apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: vault-backend
  namespace: my-app
spec:
  provider:
    vault:
      server: "https://vault.example.com"
      path: "secret"
      version: "v2"
      auth:
        kubernetes:
          mountPath: "kubernetes"
          role: "my-role"
          serviceAccountRef:
            name: "my-app-sa"

Authentication failures are common during the initial setup. If the operator fails to sync, I use a JWT Decoder to check the expiration and claims of the service account token. This quickly reveals if the issue is an expired token or a mismatched role name.

Step 4: Syncing Your First Secret

Let’s put it into practice. Suppose you have an AWS secret named /production/api-key. You want this available in your my-app namespace as a secret named api-credentials.

Apply this ExternalSecret manifest:

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: api-key-sync
  namespace: my-app
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: aws-secretsmgr
    kind: ClusterSecretStore
  target:
    name: api-credentials
    creationPolicy: Owner
  data:
  - secretKey: API_TOKEN
    remoteRef:
      key: /production/api-key
      property: api_token_value

ESO will now fetch the value and generate a standard Kubernetes Secret. Your application pods can mount this as an environment variable or a file volume. If you are migrating legacy secrets, use a Base64 Decoder to compare the new synced values against your old manual ones. This ensures the application won’t crash due to unexpected formatting changes.

Practical Tips from the Field

Managing hundreds of secrets across production clusters has taught me a few hard lessons:

  1. Watch Your API Costs: AWS Secrets Manager costs $0.40 per secret/month plus $0.05 per 10,000 API calls. Don’t set a 10-second refreshInterval. A 1-hour interval is usually plenty for most use cases and keeps your bill low.
  2. Standardize Paths: Use a hierarchy like /env/namespace/app/key. This allows you to write granular IAM or Vault policies that prevent one team from accidentally viewing another team’s credentials.
  3. Monitor Status: If a sync fails, kubectl describe externalsecret <name> is your best friend. It provides clear event logs detailing whether the issue is a 403 Forbidden error or a network timeout.
  4. Least Privilege: Never give ESO cluster-wide admin access to your vault. Grant it read-only access to specific paths to minimize the blast radius in case of a compromise.

Summary

Ditching manual kubectl create secret commands is a massive win for security and sanity. By implementing External Secrets Operator, you maintain a single source of truth in a hardened vault while keeping your GitOps manifests clean. It automates rotation, reduces human error, and ensures that your sensitive data never touches your source control in plaintext.

The initial setup takes about an hour. However, the time saved on troubleshooting and the security peace of mind it provides are worth far more. Once you see your secrets updating automatically across multiple environments, you’ll never go back to manual encoding.

Share:
Tags: