Posted inDevOps

Stop the Template Madness: A Practical Guide to Kustomize

DevOps tutorial - IT technology blog
DevOps tutorial - IT technology blog

The 2:14 AM Production Disaster

2:14 AM. My phone buzzed on the nightstand with a PagerDuty alert that felt like a punch in the gut. The production API was bleeding 503 errors. I pulled up the logs, squinting through the harsh blue light of my monitor. The culprit? A hardcoded staging database URI—db-staging.internal:5432—had snuck into the production cluster via a manual ‘kubectl apply’ session.

The next 45 minutes were spent untangling a Helm chart that had mutated into a 12-layer nesting nightmare. A single missing indentation in a YAML file had caused the configuration to drift. That night, it became clear: Helm was overkill for this project. We didn’t need a heavy package manager. We needed a clean, native way to manage environment-specific shifts without the template hell. That’s when I migrated the stack to Kustomize.

Quick Start: From Zero to Deployed in 5 Minutes

Kustomize lives inside your existing kubectl binary. No extra downloads. No server-side ‘Tiller’ ghosts to manage. It operates on a straightforward logic: you define a Base (your common denominator) and Overlays (your environment-specific deltas).

This is the standard directory structure I use for every new microservice:

myapp/
├── base/
│   ├── deployment.yaml
│   ├── service.yaml
│   └── kustomization.yaml
└── overlays/
    ├── dev/
    │   └── kustomization.yaml
    └── prod/
        └── kustomization.yaml

First, draft a lean base/deployment.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: web-server
spec:
  replicas: 1
  selector:
    matchLabels:
      app: web-server
  template:
    metadata:
      labels:
        app: web-server
    spec:
      containers:
      - name: main
        image: my-repo/api:latest
        ports:
        - containerPort: 8080

Next, use base/kustomization.yaml to register your resource files:

resources:
- deployment.yaml
- service.yaml

To scale your production environment to 10 replicas, your overlays/prod/kustomization.yaml only needs to contain the difference:

resources:
- ../../base

patches:
- target:
    kind: Deployment
    name: web-server
  patch: |-
    - op: replace
      path: /spec/replicas
      value: 10

Deploying is a one-liner:

kubectl apply -k overlays/prod/

No complex variable injection. No ‘if-else’ logic. Just raw Kubernetes YAML patched at runtime.

How the Overlay Pattern Actually Works

In the real world, this pattern forces you to treat infrastructure as layers rather than a monolithic script. Helm often leaves you with a 1,500-line values.yaml where you have no idea where a specific variable actually lands until you run a dry-run. Kustomize changes that.

By using Strategic Merge Patching, Kustomize blends your changes into the base. This is a massive win for security. I can lock down a base deployment with strict corporate resource limits and security headers, allowing developers to focus strictly on their environment variables in the overlays.

The ‘Stale Config’ Trap: ConfigMap Generators

Updating a ConfigMap and forgetting to restart the Pods is a classic rookie mistake. Kustomize fixes this via configMapGenerator. It appends a unique 10-character hash (like api-config-f98h2g6) to the ConfigMap name. Every time you tweak a value, Kustomize generates a new name, which automatically triggers a rolling update of your Deployment.

# overlays/dev/kustomization.yaml
configMapGenerator:
- name: api-config
  literals:
  - DB_URL=jdbc:mysql://dev-db:3306/mydb
  - DEBUG=true

Beyond the Basics: Patching for Reality

Literal replacements only get you so far. As your traffic grows, you’ll need to swap image tags via CI/CD pipelines or inject global metadata. Kustomize handles this with built-in transformers that act like a surgical scalpel for your YAML.

Automated Image Tagging

Manually editing YAML for every build is a recipe for burnout. Your CI tool (GitHub Actions or GitLab CI) can update the image tag dynamically during the build process:

# Inside overlays/prod/kustomization.yaml
images:
- name: my-repo/api
  newName: my-repo/api
  newTag: v1.4.2

Standardizing Metadata

I always include a commonLabels field. It makes filtering logs or debugging with kubectl get pods -l env=prod instantaneous. You can also enforce a namespace for the entire overlay to prevent a ‘dev’ resource from accidentally leaking into ‘production-apps’.

namespace: production-apps
commonLabels:
  variant: prod
  team: backend

Hard-Learned Lessons from the Trenches

When you’re fighting a fire at 2 AM, ambiguity is the enemy. Here are four rules I live by to keep my clusters sane:

  • Keep the Base Minimal: The base should be the ‘skeleton’ of your app. Don’t bake in environment-specific defaults there; push those into a ‘default’ or ‘dev’ overlay.
  • Preview Before You Apply: Run kubectl kustomize overlays/prod first. It dumps the final, merged YAML to your terminal. This has saved me from deploying broken configs more times than I can count.
  • Secure Your Secrets: If you use secretGenerator, never commit the source files to Git. Use an external secret manager like HashiCorp Vault or SealedSecrets for anything sensitive.
  • Avoid Nesting Hell: Stick to two levels (Base and Overlays). If you find yourself building base -> staging -> prod-region-1 -> prod-region-1-zone-a, you are over-engineering.

The bottom line: Kustomize isn’t a total replacement for Helm. Helm is still the king for distributing complex third-party tools like Prometheus. But for your own microservices? Kustomize offers a level of clarity that templates can’t touch. It lets me sleep better knowing that the YAML on my screen is exactly what’s running in the cluster.

Share:
Tags: