Kubernetes Cleanup: Using Kube-janitor to Slash Dev and Staging Costs

The Cost of “Ghost” Resources in Kubernetes

Running a Kubernetes cluster for a growing engineering team eventually leads to resource sprawl. In busy Dev or Staging environments, developers constantly spin up new namespaces for feature testing or PR previews. The mess starts when the task ends, but the resources remain active.

Forgotten namespaces don’t just sit there. They consume VPC IP addresses, occupy load balancer slots, and hog CPU and memory reservations. On managed services like EKS or GKE, this waste is expensive. In one recent audit, I found a cluster where 40% of the nodes were supporting projects finished weeks ago. That translates to hundreds of dollars wasted every month on idle EC2 instances.

Why Manual Cleanup and Custom Scripts Fail

The first instinct for most teams is a manual approach. You might ask a senior engineer to spend Friday afternoons hunting down old namespaces. This is a poor use of expensive engineering time. Eventually, someone writes a “quick” bash script to run kubectl delete via a cron job.

Custom scripts are often brittle and dangerous. They lack detailed logging and rarely handle complex dependencies well. One slight regex error could accidentally wipe your production database. We need a declarative, Kubernetes-native way to manage object lifecycles. Kube-janitor solves this by treating cleanup as a policy, not a chore.

Setting Up Kube-janitor

Kube-janitor is a lightweight controller created by Zalando. It monitors your cluster and removes resources based on specific rules or metadata. Unlike a blind script, it understands Kubernetes objects and respects safety boundaries.

I recommend using Helm for the installation to manage RBAC permissions properly. Run these commands to get started:

# Add the Zalando repository
helm repo add kube-janitor https://zalando.github.io/kube-janitor/
helm repo update

# Install the controller
helm install kube-janitor kube-janitor/kube-janitor \
  --namespace kube-janitor \
  --create-namespace

Kube-janitor remains idle after installation. It won’t delete anything until you provide specific instructions through ConfigMaps or resource annotations.

Configuring Cleanup Policies

Using annotations is the most effective way to manage resources. This shifts the responsibility of the resource lifespan to the person who created it. I have used this method in several production clusters with great success.

1. Setting a Time-to-Live (TTL)

You can tell Kube-janitor to delete a namespace exactly 4 hours after its creation. This is ideal for CI/CD pipelines that generate short-lived preview environments. Simply add the janitor/ttl annotation to your manifest.

apiVersion: v1
kind: Namespace
metadata:
  name: feature-x-preview
  annotations:
    janitor/ttl: "4h"
spec: {}

2. Setting a Specific Expiry Date

Sometimes you know exactly when a test phase ends, such as the final day of a sprint. In these cases, use an absolute timestamp. This ensures resources vanish even if you are out of the office.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: temporary-worker
  annotations:
    janitor/expires: "2024-12-31T23:59:59Z"
spec:
  # ... deployment specs

3. Creating Global Rules

The reality is that developers often forget to add annotations. You can define global rules in your values.yaml to catch these omissions. For example, you can target any namespace starting with “test-” that is older than seven days.

rules:
  - name: "cleanup-test-namespaces"
    resources:
      - namespaces
    condition: |
      metadata.name =~ "^test-.*" and age > "7d"
    action: delete

Verification and Safety

Never let a cleanup tool run wild without verification. Kube-janitor includes a “dry-run” mode that is vital for initial testing. Always check the logs before enabling destructive actions.

Use this command to see what the controller is targeting:

kubectl logs -f deployment/kube-janitor -n kube-janitor

You will see logs like: Resource namespaces/test-app-123 is expired, deleting.... If dry-run is active, the logs will state that the resource would have been deleted without actually removing it.

Protecting Critical Systems

To prevent the accidental deletion of kube-system or production, use an exclusion annotation. I recommend applying this to all core infrastructure namespaces immediately.

kubectl annotate namespace production janitor/policy=keep

Best Practices

After managing Kube-janitor across multiple clusters, I have found three strategies that work best. First, set a default 14-day TTL for all staging namespaces. This forces teams to justify why a resource should stay longer.

Second, integrate annotations into your CI/CD templates. Whether you use GitHub Actions or GitLab CI, your deployment scripts should automatically calculate and inject the janitor/ttl. Finally, send your logs to a tool like Grafana Loki. If you see a sudden spike in deletions, you can investigate before it impacts a developer’s workflow.

Automated cleanup moves you away from reactive cost management. You no longer need to scramble when the bill arrives. Instead, your cluster stays lean, predictable, and cost-efficient by design.