The monthly AWS bill arrived and the number next to our EKS dev cluster made me stop scrolling. $847 for a cluster that developers actively use maybe 10 hours a day, 5 days a week. That’s 50 hours of real usage versus 718 hours of billing.
Fifty dollars a day for idle compute is hard to rationalize: dev and staging workloads were running 24/7 even though nobody touched them after 7 PM or on weekends. Nodes stayed up, pods kept running, and AWS kept charging. Classic cloud waste that sneaks up on you — and the kind that a single YAML file can largely solve.
The Real-World Problem: Kubernetes Never Sleeps
Our staging cluster had 14 Deployments and 3 StatefulSets across two namespaces. On a typical Tuesday at 11 PM, CPU utilization was under 2% across all pods. Every process was idle, waiting for load that wouldn’t come until 8 AM the next morning. But the nodes didn’t know that. They just kept running, because that’s what Kubernetes nodes do.
Weekends made it worse, not just overnight hours. Forty-eight hours of zero-productivity compute, repeated every single week. On a 5-day work week with 8 AM to 8 PM active hours, you’re paying for workloads that are actively useful roughly 35% of the time.
Root Cause: Kubernetes Was Designed for High Availability, Not Cost Optimization
When you deploy something to Kubernetes, it runs until you explicitly stop it. There’s no native concept of “office hours” scheduling. The platform assumes you want your workloads always available — which makes perfect sense for production, but creates unnecessary waste in dev and staging environments.
Horizontal Pod Autoscaler (HPA) scales based on CPU and memory metrics. It helps with traffic spikes, but HPA enforces a minimum replica count of 1 — it cannot scale workloads to zero at all. That’s a hard architectural limit, not a configuration issue. Reaching true zero requires KEDA or a schedule-based tool like kube-green.
Dev and staging clusters inherit production’s always-on behavior, even when nobody is watching them at 3 AM on a Sunday. Nothing in the platform pushes back on that.
Solutions I Compared Before Settling on Kube-green
Manual kubectl Scaling
The team agreed to run kubectl scale deployment --replicas=0 before leaving each evening. This worked for about two weeks. Then people started forgetting, or leaving a deployment at 1 replica “just in case.” Human processes don’t survive developer context switches.
Custom Bash CronJob
I wrote a Kubernetes CronJob that scaled all Deployments in certain namespaces to zero at 8 PM and back up at 8 AM. It worked — until it didn’t. The script didn’t store original replica counts anywhere, so everything woke up with 1 replica. Services that needed 3+ replicas for health checks to pass started failing, and I spent a morning debugging what looked like application crashes but was really just missing replicas.
KEDA (Kubernetes Event-Driven Autoscaler)
KEDA can scale workloads to zero based on external event sources. It’s excellent for queue-driven services, but using it purely to turn things off at night felt like using a sledgehammer on a finishing nail. Every Deployment would need its own ScaledObject, and managing those across namespaces added more operational overhead than the original problem.
Kube-green
Purpose-built for schedule-based suspension. One CRD per namespace, define when to sleep and when to wake, done. It stores original replica counts as annotations before scaling to zero, then restores them precisely on wakeup. Six months running this across dev and staging — zero phantom restarts, zero missed wakeups, zero on-call pages because staging forgot to come back online. The results have been consistently stable.
Setting Up Kube-green: The Practical Guide
Installation via Helm
helm repo add kube-green https://kube-green.dev/helm-charts
helm repo update
helm install kube-green kube-green/kube-green \
--namespace kube-green \
--create-namespaceVerify the operator is running:
kubectl get pods -n kube-green
# NAME READY STATUS RESTARTS AGE
# kube-green-xxxxxxxxxx-xxxxx 1/1 Running 0 2mThe SleepInfo Resource
This is the entire configuration surface of kube-green — one YAML file per namespace:
apiVersion: kube-green.com/v1alpha1
kind: SleepInfo
metadata:
name: dev-sleep-schedule
namespace: development
spec:
weekdays: "1-5" # Monday to Friday
sleepAt: "20:00" # Sleep at 8 PM
wakeUpAt: "08:00" # Wake at 8 AM
timeZone: "Asia/Tokyo"
suspendDeployments: true
suspendStatefulSets: true
suspendCronJobs: truekubectl apply -f dev-sleep-schedule.yamlKube-green writes the original replica count to each resource’s annotations before scaling to zero. On wakeup, it reads those annotations and restores exactly what was there before. No configuration drift, no guessing.
Staging with Different Hours and Exclusions
Staging usually stays active later for QA runs. You can configure a separate schedule, and exclude specific workloads from suspension:
apiVersion: kube-green.com/v1alpha1
kind: SleepInfo
metadata:
name: staging-sleep-schedule
namespace: staging
spec:
weekdays: "1-5"
sleepAt: "23:00" # Later for QA testing
wakeUpAt: "07:30" # Early for morning smoke tests
timeZone: "Asia/Tokyo"
suspendDeployments: true
suspendStatefulSets: false # Keep databases running
excludeRef:
- apiVersion: apps/v1
kind: Deployment
name: monitoring-agent # Always keep this aliveUse excludeRef for monitoring agents or any service that external health checks depend on. Kube-green skips those entirely during the suspension cycle.
Checking Status and Debugging
# See current sleep state
kubectl get sleepinfo -n development
# NAME SLEEP AT WAKE UP AT WEEKDAYS TIMEZONE STATUS
# dev-sleep-schedule 20:00 08:00 1-5 Asia/Tokyo Sleeping
# Check stored replica counts on suspended deployments
kubectl get deployments -n development -o jsonpath=\
'{range .items[*]}{.metadata.name}: replicas={.metadata.annotations.sleepinfo\.kube-green\.dev/replicas}{"\n"}{end}'Manual Override When Needed
When a developer needs the environment up on Saturday, they just scale the deployment directly:
kubectl scale deployment my-app --replicas=2 -n developmentKube-green won’t fight you. It reads current replica counts at sleep time and saves whatever is there. On the next scheduled wakeup, normal operations resume without any manual cleanup.
What Kube-green Doesn’t Cover
Kube-green suspends workloads — it doesn’t directly manage infrastructure. On managed Kubernetes (EKS, GKE, AKS), your control plane runs regardless. The node cost savings only materialize if your cluster autoscaler actually removes nodes when pods are gone. Make sure node groups support scale-to-zero:
# Verify EKS node group minimum size
aws eks describe-nodegroup \
--cluster-name my-cluster \
--nodegroup-name dev-workers \
--query 'nodegroup.scalingConfig'
# Should show minSize: 0 or 1Also: PersistentVolume costs don’t stop when pods sleep. EBS volumes bill by the hour whether or not anything is attached. For truly low-cost dev environments, think about ephemeral storage for non-critical data, or accept that storage costs are the smaller line item worth leaving alone.
Actual Numbers After Six Months
Before kube-green, our dev cluster averaged 8 nodes running 24/7. After — with cluster autoscaler removing nodes once pods were gone — we typically drop to 1–2 nodes during off-hours. Monthly bill went from ~$847 to ~$290. Staging had similar results: roughly 65% reduction across both environments.
Operational cost was near zero. After the initial 30 minutes of setup and a week of monitoring wakeup times, the system has run without any intervention. Developers rarely notice the schedule — morning wakeup timing means environments are ready before people finish their first coffee.
The more interesting side effect has been cultural. Before, nobody thought about whether dev resources were necessary at a given scale. Now cost visibility has sparked conversations about right-sizing: why does a dev Deployment need 3 replicas? Why is that StatefulSet configured for 4Gi memory when staging tests never come close to stressing it? Kube-green made resource ownership visible in a way that billing dashboards alone never did.
If your dev or staging cluster is running around the clock with no suspension strategy, this is the fastest path to meaningful savings — thirty minutes of setup, $557 back in your monthly budget, every month. The CRD is simple, the operator is lightweight, and those savings stack up again every weekend.
