Posted inDevOps

Kubernetes Cost Monitoring with OpenCost: A Practical FinOps Guide for Cloud-Native Teams

DevOps tutorial - IT technology blog
DevOps tutorial - IT technology blog

Why Kubernetes Costs Sneak Up on You

You deploy a few microservices, add a staging cluster, and six months later your cloud bill has quietly doubled. Nobody changed the architecture intentionally. Resource requests were set too conservatively, idle namespaces piled up unnoticed, and the team had no clear picture of which workloads were actually burning money.

Cost observability is one of those skills that separates a proactive DevOps engineer from one who’s perpetually surprised by monthly cloud invoices. You can have zero alerts firing and 100% uptime across every Grafana panel — and still be completely in the dark about where your $8,000 monthly AWS bill is actually going.

OpenCost fills that gap. It’s an open-source CNCF sandbox project that gives you per-namespace, per-workload cost data without vendor lock-in or a $500/month SaaS subscription.

Approach Comparison: How Teams Usually Handle Kubernetes Cost Monitoring

Three approaches dominate the space, each with genuine trade-offs:

1. Cloud Provider Native Tools

AWS Cost Explorer, GCP Billing Reports, and Azure Cost Management all show you total cluster spend. The problem is granularity — they report EC2 node costs, but they can’t break that down by namespace, deployment, or label. You see the bill, not the breakdown.

2. Commercial FinOps Platforms

Tools like Kubecost (commercial tier), CloudHealth, or Apptio Cloudability give you deep allocation reports. They work well. The catch: Kubecost Pro starts around $500/month for a mid-size cluster, and CloudHealth runs into the thousands. For smaller teams trying to find waste they haven’t quantified yet, that’s a hard sell.

3. OpenCost (Open Source)

OpenCost was open-sourced by Kubecost in 2022 and donated to the CNCF. It runs inside your cluster, scrapes Prometheus metrics, and applies real cloud pricing to your actual resource consumption. Clean API, lightweight UI, free to run. No data leaves your infrastructure, and it integrates natively with Grafana.

Pros and Cons of OpenCost

Pros

  • Truly free and open source — Apache 2.0 license, no usage limits or seat fees
  • Namespace and workload-level allocation — costs broken down by deployment, daemonset, statefulset, label, or annotation
  • Works on any cloud or on-prem — supports AWS, GCP, Azure, and custom pricing for bare metal
  • CNCF sandbox project — active community, well-maintained Helm chart, good documentation
  • Grafana dashboard included — pre-built dashboards available on Grafana.com
  • Prometheus native — if you already run kube-prometheus-stack, integration takes about 10 minutes

Cons

  • No built-in anomaly alerts — you need to write Prometheus alerting rules yourself for cost spikes
  • Cloud billing reconciliation requires extra setup — matching actual discounts and reserved instance pricing needs cloud billing API access
  • UI is functional but minimal — the commercial Kubecost UI is significantly richer; OpenCost covers the basics
  • Persistent storage recommended for history — without it, cost history resets on pod restart

Recommended Setup

For most teams, this four-component stack covers 90% of what you actually need:

  • OpenCost — cost engine and UI
  • kube-prometheus-stack — Prometheus + Grafana (if not already deployed)
  • OpenCost Grafana dashboards — imported from Grafana.com
  • Persistent Volume for Prometheus — so cost history survives pod restarts

If you already have Prometheus running in your cluster, skip straight to Step 2.

Implementation Guide

Step 1: Install kube-prometheus-stack (skip if already installed)

OpenCost depends on Prometheus for metrics. The community Helm chart is the fastest path:

# Add the Helm repo
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update

# Install into the monitoring namespace
helm install kube-prometheus-stack prometheus-community/kube-prometheus-stack \
  --namespace monitoring \
  --create-namespace \
  --set prometheus.prometheusSpec.retention=15d \
  --set prometheus.prometheusSpec.storageSpec.volumeClaimTemplate.spec.resources.requests.storage=20Gi

The retention=15d flag keeps 15 days of metrics — enough for meaningful weekly cost trends. Adjust the storage claim based on cluster scale; 20Gi is a reasonable baseline for 10–30 nodes.

Step 2: Install OpenCost

OpenCost ships its own Helm chart that deploys the cost engine alongside a lightweight UI:

# Add OpenCost Helm repo
helm repo add opencost https://opencost.github.io/opencost-helm-chart
helm repo update

# Create values file
cat > opencost-values.yaml << 'EOF'
opencost:
  exporter:
    cloudProviderApiKey: ""  # Leave empty for on-demand pricing without discounts
  ui:
    enabled: true
  prometheus:
    internal:
      enabled: false  # We're using the external prometheus we installed
    external:
      enabled: true
      url: "http://kube-prometheus-stack-prometheus.monitoring.svc.cluster.local:9090"
EOF

# Install
helm install opencost opencost/opencost \
  --namespace opencost \
  --create-namespace \
  -f opencost-values.yaml

Pay attention to the Prometheus URL format — it uses Kubernetes service DNS. If your Prometheus service name or namespace differs, update the URL before installing.

Step 3: Verify the Deployment

# Check pods are running
kubectl get pods -n opencost

# Expected output:
# NAME                        READY   STATUS    RESTARTS   AGE
# opencost-xxxxxxxxx-xxxxx    2/2     Running   0          2m

# Check OpenCost logs for any Prometheus connectivity issues
kubectl logs -n opencost deployment/opencost -c opencost

Look for Prometheus connection successful in the logs. Connection refused errors almost always mean the Prometheus URL in your values file needs adjusting.

Step 4: Access the OpenCost UI

# Port-forward the UI to your local machine
kubectl port-forward -n opencost service/opencost-ui 9090:9090

Open http://localhost:9090. Within a few minutes, you’ll see cost data broken down by namespace. OpenCost backfills historical data from Prometheus, so you’ll immediately have cost estimates stretching back 15 days — the full length of our retention window.

Step 5: Query Costs via the OpenCost API

The HTTP API is where OpenCost really shines for automation. Script cost reports, feed data into Slack, or pipe it into internal dashboards:

# Port-forward the API
kubectl port-forward -n opencost service/opencost 9003:9003

# Query costs for the last 7 days, aggregated by namespace
curl -s "http://localhost:9003/allocation/compute?window=7d&aggregate=namespace" | \
  python3 -m json.tool | head -80

# Query costs aggregated by deployment label
curl -s "http://localhost:9003/allocation/compute?window=24h&aggregate=label:app" | \
  python3 -m json.tool

The aggregate parameter is the key to granular analysis. Group by namespace, controller, pod, node, or any custom label your team applies. Label-based grouping is what lets you tie spend directly back to individual product teams.

Step 6: Import Grafana Dashboards

OpenCost publishes official Grafana dashboards. Port-forward Grafana first:

# Port-forward Grafana
kubectl port-forward -n monitoring service/kube-prometheus-stack-grafana 3000:80

Open http://localhost:3000 (default credentials: admin/prom-operator), navigate to Dashboards → Import, and enter dashboard ID 15714. Set your Prometheus instance as the data source when prompted.

Step 7: Set Up Cloud Provider Pricing (Optional but Recommended)

Out of the box, OpenCost uses on-demand pricing from cloud provider APIs. For AWS clusters, you can get more accurate numbers by feeding in your billing configuration:

cat > aws-config.json << 'EOF'
{
  "provider": "AWS",
  "description": "AWS Provider Configuration",
  "AWS_ACCESS_KEY_ID": "your-access-key",
  "AWS_SECRET_ACCESS_KEY": "your-secret-key",
  "awsSpotDataBucket": "",
  "awsSpotDataRegion": "ap-northeast-1",
  "projectID": "123456789"
}
EOF

# Create a secret from the config
kubectl create secret generic opencost-aws-config \
  --from-file=aws-config.json=aws-config.json \
  -n opencost

Update your opencost-values.yaml to reference this secret, then run helm upgrade to apply.

What to Look at First

Three numbers are worth checking the moment your dashboard loads:

  • Idle cost by namespace — a namespace requesting 4 CPU but consuming 400m is burning 90% of its compute budget for nothing; these are your fastest right-sizing opportunities
  • Cost by controller type — DaemonSets run on every node, so a bloated DaemonSet with 500m CPU requests on a 20-node cluster consumes 10 CPU cores cluster-wide; audit them carefully
  • Cost over time trends — gradual upward drift usually means resource request creep, where developers stack safety buffers each sprint without ever trimming the old ones

A Few Tips from Production Use

A few patterns appear in almost every production cluster once you have real cost visibility.

Staging namespaces are almost always the biggest offenders. Developers copy production configs and forget to scale down requests — finding a staging pod claiming 2 CPU and 4Gi RAM while actually consuming 200m CPU and 512Mi is more common than you’d expect. Start there. Fixing staging resource requests can cut 15–30% of that namespace’s cost in an afternoon.

Alert on cost growth rate, not absolute costs. Absolute thresholds go stale as your product scales. A 30% week-over-week increase in any namespace is worth investigating regardless of the baseline — that signal stays relevant whether you’re spending $200/month or $20,000/month on the cluster.

Label-based aggregation changes the dynamic of cost conversations entirely. When engineering managers can see their team’s infrastructure spend on the same Grafana instance as SLOs and error rates, budget discussions shift from estimates to data. That context shift often matters more than any single optimization finding.

Share:
Tags: