Kubernetes Resource Optimization with Goldilocks and VPA: Sizing Requests and Limits

At 2:14 AM, my phone buzzed with an urgent PagerDuty alert: NodeMemoryPressure, followed by cascading OOMKilled errors across two worker nodes. Within minutes, six pods in our payment processing namespace were in CrashLoopBackOff. The culprit was painfully familiar.

A newly deployed microservice requested just 100Mi of memory but had no limit configured, allowing it to balloon past 9.2Gi during batch reconciliation. Next to it, an internal webhook service held reservations for 4 CPU cores while consuming barely 15 millicores all day. The cluster was starving on paper while running nearly idle in reality.

Setting Kubernetes requests and limits by guesswork carries a steep cost. You either burn thousands of dollars on unallocated headroom, or you scramble at 2 AM when the Linux kernel terminates critical workloads. Combining Fairwinds Goldilocks with the Kubernetes Vertical Pod Autoscaler (VPA) eliminates the blind guessing. Together, they turn raw consumption metrics into precise, actionable pod resource recommendations.

Table of Contents

Quick start (5 min)

During an active incident, you do not need architectural essays. You need immediate visibility into what containers consume versus what you booked for them.

Step 1: Verify Metrics Server

VPA depends on active node and pod metrics. First, verify that your cluster metrics pipeline is running:

kubectl top nodes
kubectl top pods -A

If both commands return resource tables, you are good to go. If you get error: Metrics API not available, deploy the official metrics-server manifest before moving ahead.

Step 2: Install the Vertical Pod Autoscaler

Goldilocks relies on the VPA Recommender engine to calculate baselines. Clone the autoscaler repository and install the standard CRDs and controllers:

git clone https://github.com/kubernetes/autoscaler.git
cd autoscaler/vertical-pod-autoscaler
./hack/vpa-up.sh

Confirm all three core VPA controllers are active:

kubectl get pods -n kube-system | grep vpa

Look for vpa-recommender, vpa-updater, and vpa-admission-controller in the Running state.

Step 3: Deploy Goldilocks via Helm

Goldilocks inspects your deployments, generates VPA objects automatically, and presents recommendations through a lightweight visual dashboard. Add the repository and install the Helm chart:

helm repo add fairwinds-stable https://charts.fairwinds.com/stable
helm repo update

helm install goldilocks fairwinds-stable/goldilocks \
  --namespace goldilocks \
  --create-namespace \
  --set dashboard.enabled=true

Step 4: Enable Goldilocks on your target namespace

Goldilocks monitors only namespaces that opt in. Tag your target namespace with the inspection label:

kubectl label namespace production goldilocks.fairwinds.com/enabled=true

Forward the dashboard service port to your workstation:

kubectl -n goldilocks port-forward svc/goldilocks-dashboard 8080:80

Navigate to http://localhost:8080. Goldilocks displays every deployment in the namespace alongside concrete recommendations for both Guaranteed and Burstable Quality of Service (QoS) profiles.

Deep dive

To prevent recurring outages, engineering teams must understand how the VPA engine calculates these values and how Kubernetes interprets them.

How VPA Calculates Target Resources

Once you label a namespace, the Goldilocks controller creates a corresponding VerticalPodAutoscaler custom resource for each deployment. It sets updateMode: "Off" by default. This design keeps evaluation safe: the recommender observes production traffic without restarting pods or mutating running manifests.

You can inspect any auto-generated VPA object directly:

kubectl get vpa -n production
kubectl get vpa payment-service -n production -o yaml

Look inside status.recommendation.containerRecommendations. You will find four distinct calculations evaluated over an 8-day sliding window:

LowerBound: The minimum threshold. Dropping requests below this number creates a severe risk of CPU throttling or memory starvation.
Target: The calculated sweet spot. Use this exact value for your container requests.
UpperBound: The observed ceiling during peak usage spikes. This value informs container memory limits.
UncappedTarget: The raw target calculation before applying any user-defined minimum or maximum constraints in the VPA policy.

Guaranteed vs Burstable QoS in Goldilocks

Every pod receives a Quality of Service tier based on how you declare CPU and memory:

Guaranteed: requests match limits for every container. The Kubernetes scheduler reserves the entire block exclusively on the node. Goldilocks sets both fields to the VPA Target. Use this tier for critical databases and queue consumers.
Burstable: requests match the VPA Target, but limits are set higher or left open for CPU. This configuration allows microservices to absorb temporary traffic spikes without wasting node reservations during quiet hours.

During our 2 AM incident, our primary payment proxy was running under the BestEffort class because it lacked resource declarations entirely. When node memory ran out, the Linux kernel targeted that proxy first. BestEffort pods carry an oom_score_adj of 1000, making them the first victims during node pressure.

Advanced usage

Standard settings cover basic stateless apps. Stateful workloads, telemetry sidecars, and spiky background workers require more deliberate configurations.

Defining Custom VPA Constraints

Sometimes you need hard boundaries. For example, an unpatched memory leak should not cause VPA to recommend a 32Gi reservation on a node pool with 16Gi instances. You can define a custom VPA manifest with explicit guards:

apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  name: checkout-worker-vpa
  namespace: production
spec:
  targetRef:
    apiVersion: "apps/v1"
    kind: Deployment
    name: checkout-worker
  updatePolicy:
    updateMode: "Off"
  resourcePolicy:
    containerPolicies:
      - containerName: checkout-worker
        minAllowed:
          cpu: 100m
          memory: 256Mi
        maxAllowed:
          cpu: 2000m
          memory: 4Gi
        controlledResources: ["cpu", "memory"]

Apply this manifest to override automatic defaults. The VPA recommender will keep its targets within your specified safety envelope.

Excluding Service Mesh Sidecars

Telemetry sidecars like Envoy, Linkerd, or Datadog agents distort application sizing. Goldilocks lets you filter them out so recommendations focus purely on your application containers.

Update the Goldilocks Helm release with an exclusion list:

helm upgrade goldilocks fairwinds-stable/goldilocks \
  --namespace goldilocks \
  --set "controller.flags.exclude-containers=istio-proxy\,datadog-agent"

This keeps your metrics actionable by isolating core application performance from proxy overhead.

Exporting Goldilocks Metrics for CI/CD Automation

Manual dashboard inspections do not scale across dozens of clusters. Export Goldilocks data straight into Prometheus instead:

helm upgrade goldilocks fairwinds-stable/goldilocks \
  --namespace goldilocks \
  --reuse-values \
  --set controller.metrics.enabled=true

Once exported, you can flag misconfigured workloads in Grafana or CI linting steps. For example, alert teams whenever configured requests exceed recommended targets by more than 2x, or whenever requests sit below 50% of the target threshold.

Practical tips

Fine-tuning production workloads is an iterative practice. Keep these four rules in mind before rolling out changes across your clusters:

1. Beware the CPU Limit Throttling Trap

Memory is an incompressible resource. Exceed your limit, and the kernel terminates the container immediately. CPU, by contrast, is compressible. Exceed your CPU limit, and the Completely Fair Scheduler (CFS) throttles container run time across quota periods.

Tight CPU limits frequently cause artificial latency spikes. In one API service handling 1,200 req/sec, removing a 500m CPU limit dropped p99 response times from 840ms down to 38ms. Set CPU requests based on the Goldilocks Target, but consider leaving CPU limits unset or generous to prevent synthetic throttling.

2. Account for JVM and Runtime Memory Overhead

Runtimes like Java, Node.js, and Python consume memory well beyond their application heaps. A JVM running with -Xmx2g still allocates off-heap memory for thread stacks, Metaspace, and GC metadata. Setting container limits right at 2Gi guarantees an OOMKill. Goldilocks reports real Resident Set Size (RSS). For a 2Gi heap, provision at least 2.8Gi to 3.0Gi in container limits to accommodate runtime overhead safely.

3. Never Combine VPA Auto-Mode with HPA on CPU or Memory

Avoid running VPA in updateMode: "Auto" alongside a Horizontal Pod Autoscaler driven by CPU or memory metrics. The two controllers will fight each other. HPA will spin up additional replicas while VPA increases pod sizes, triggering an aggressive feedback loop that exhausts node capacity. If you want automatic horizontal scaling, base HPA on custom business metrics like SQS queue depth or HTTP request rates.

4. Establish a Weekly Sizing Review Routine

Dedicate 20 minutes every Monday morning to resource hygiene. Review workloads highlighted in Goldilocks, then update your GitOps manifests in ArgoCD or Flux. Right-sizing twenty microservices by just 500m CPU and 1Gi memory each can easily reclaim $1,200 every month in cloud infrastructure bills.