tutorial

Monitoring Goldilocks: A Practical Guide for Kubernetes Resource Right-Sizing

Goldilocks is a Kubernetes tool by Fairwinds that makes Vertical Pod Autoscaler (VPA) recommendations accessible to developers. It creates VPA objects for ev...

Goldilocks is a Kubernetes tool by Fairwinds that makes Vertical Pod Autoscaler (VPA) recommendations accessible to developers. It creates VPA objects for every deployment in a namespace, collects resource usage history, and surfaces request and limit recommendations through a visual dashboard. When Goldilocks is healthy, teams can right-size workloads and cut cloud costs. When it's not, VPA objects go stale, recommendations drift from reality, and engineers fly blind on resource sizing.

Why Monitoring Goldilocks Matters

Goldilocks operates on a continuous feedback loop: its controller watches namespaces, creates and manages VPA objects, and the dashboard reads from those VPA objects to show recommendations. A failure anywhere in this loop causes silent degradation:

  • Controller failure: VPA objects are not created for new deployments — recommendations are missing entirely
  • VPA objects going stale: VPA recommender stops updating recommendations — old suggestions become misleading
  • Dashboard UI unavailability: engineers can't access recommendations even if VPA data is current
  • Namespace controller gap: namespaces labeled for Goldilocks are not being managed

The subtlety of Goldilocks failures makes monitoring non-negotiable. Unlike an app that crashes visibly, Goldilocks can appear to work (the dashboard loads) while serving months-old recommendations.

Architecture Overview

Goldilocks Controller → VPA Objects → VPA Recommender → Goldilocks Dashboard
      ↑
Namespace labels (goldilocks.fairwinds.com/enabled: "true")

The controller watches for namespace labels and creates/deletes VPA objects accordingly. The VPA recommender (a separate component) continuously updates those VPA objects with resource recommendations. The dashboard reads VPA objects and presents them.

What to Monitor

1. Goldilocks Controller Health

The controller manages the lifecycle of VPA objects. Verify it's running and processing namespaces:

# Check controller pod status
kubectl get pods -n goldilocks -l app.kubernetes.io/name=goldilocks,app.kubernetes.io/component=controller

# Check controller logs for errors
kubectl logs -n goldilocks -l app.kubernetes.io/component=controller --since=15m | \
  grep -E "(error|failed|panic)"

A healthy controller log shows namespace reconciliation events. Absence of recent reconcile logs — or a backlog of errors — indicates the controller is stuck.

Verify the controller is actually managing VPA objects for labeled namespaces:

# List namespaces that Goldilocks should be managing
kubectl get ns -l goldilocks.fairwinds.com/enabled=true

# Count VPA objects Goldilocks has created
kubectl get vpa --all-namespaces -l app.kubernetes.io/managed-by=goldilocks | wc -l

If the VPA object count is significantly lower than the number of deployments in labeled namespaces, the controller is not keeping up or has missed reconciliations.

2. VPA Object Health and Freshness

VPA objects contain the actual recommendations. Check that they exist and contain current data:

# List all Goldilocks-managed VPA objects and their conditions
kubectl get vpa --all-namespaces -l app.kubernetes.io/managed-by=goldilocks \
  -o json | jq '.items[] | {
    namespace: .metadata.namespace,
    name: .metadata.name,
    conditions: .status.conditions,
    lastUpdated: .status.recommendation.containerRecommendations[0].upperBound
  }'

For each VPA object, check that recommendation is non-null and the VPA recommender is providing updates. If all VPA recommendations are null, the VPA recommender component is not running or not functioning:

# Check if VPA recommender is running (prerequisite for Goldilocks)
kubectl get pods -n kube-system -l app=vpa-recommender

# Or in the VPA namespace
kubectl get pods -n vertical-pod-autoscaler -l app=vpa-recommender

3. Recommendation Staleness Detection

VPA recommendations update as the recommender collects more utilization samples. A recommendation that hasn't changed in an unusually long time suggests the recommender has stalled:

# Check recommendation update timestamps
kubectl get vpa --all-namespaces -l app.kubernetes.io/managed-by=goldilocks \
  -o json | jq '
    .items[] |
    {
      namespace: .metadata.namespace,
      name: .metadata.name,
      lastTransitionTime: (.status.conditions[]? | select(.type=="RecommendationProvided") | .lastTransitionTime)
    }
  '

If lastTransitionTime for RecommendationProvided is days old on a cluster with active workloads, the recommender has stopped updating — Goldilocks will show stale data.

You can also monitor VPA object conditions:

# Find VPA objects where recommendations are NOT being provided
kubectl get vpa --all-namespaces -o json | jq '
  .items[] |
  select(.status.conditions[]? | .type == "RecommendationProvided" and .status != "True") |
  {namespace: .metadata.namespace, name: .metadata.name}
'

4. Namespace Controller Coverage

Goldilocks uses a label-based controller to decide which namespaces to manage. Verify that labeled namespaces have corresponding VPA objects for all their deployments:

# For each enabled namespace, compare deployments vs VPA objects
for ns in $(kubectl get ns -l goldilocks.fairwinds.com/enabled=true -o name | cut -d/ -f2); do
  DEPLOYMENTS=$(kubectl get deploy -n "$ns" --no-headers | wc -l)
  VPAS=$(kubectl get vpa -n "$ns" -l app.kubernetes.io/managed-by=goldilocks --no-headers 2>/dev/null | wc -l)
  echo "Namespace: $ns | Deployments: $DEPLOYMENTS | VPAs: $VPAS"
done

A mismatch (more deployments than VPAs) means the controller hasn't created VPA objects for all workloads — new deployments are not being right-sized.

5. Dashboard UI Availability

The Goldilocks dashboard exposes recommendations via a web UI. Monitor its availability:

# Check dashboard pod
kubectl get pods -n goldilocks -l app.kubernetes.io/component=dashboard

# Test dashboard HTTP response
kubectl port-forward -n goldilocks svc/goldilocks-dashboard 8080:80 &
curl -sf -o /dev/null -w "%{http_code}" http://localhost:8080/

The dashboard serves on port 80 by default. A non-200 response means engineers cannot access VPA recommendations — even if the data is healthy.

Prometheus Alerting Rules

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: goldilocks-alerts
  namespace: goldilocks
spec:
  groups:
  - name: goldilocks
    rules:
    - alert: GoldilocksControllerDown
      expr: |
        kube_pod_status_ready{
          namespace="goldilocks",
          pod=~"goldilocks-controller.*",
          condition="true"
        } == 0
      for: 5m
      labels:
        severity: critical
      annotations:
        summary: "Goldilocks controller is not ready"
        description: "Goldilocks controller is down — VPA objects are not being managed."

    - alert: GoldilocksDashboardDown
      expr: |
        kube_pod_status_ready{
          namespace="goldilocks",
          pod=~"goldilocks-dashboard.*",
          condition="true"
        } == 0
      for: 5m
      labels:
        severity: warning
      annotations:
        summary: "Goldilocks dashboard is not ready"
        description: "Goldilocks dashboard is down — engineers cannot access resource recommendations."

    - alert: GoldilocksVPARecommenderDown
      expr: |
        kube_pod_status_ready{
          pod=~"vpa-recommender.*",
          condition="true"
        } == 0
      for: 10m
      labels:
        severity: critical
      annotations:
        summary: "VPA Recommender is not running"
        description: "Without VPA Recommender, Goldilocks cannot generate or update recommendations."

    - alert: GoldilocksVPACoverageGap
      expr: |
        (
          count(kube_deployment_labels) 
          - count(kube_customresource_info{customresource_group="autoscaling.k8s.io", customresource_kind="VerticalPodAutoscaler"})
        ) > 5
      for: 30m
      labels:
        severity: warning
      annotations:
        summary: "Goldilocks VPA coverage gap detected"
        description: "More deployments exist than Goldilocks VPA objects — some workloads are not being right-sized."

Operational Runbook Snippets

Force Goldilocks to reconcile a namespace:

# Remove and re-add the label to trigger reconciliation
kubectl label ns <namespace> goldilocks.fairwinds.com/enabled-
kubectl label ns <namespace> goldilocks.fairwinds.com/enabled=true

Restart controller to clear stuck state:

kubectl rollout restart deployment/goldilocks-controller -n goldilocks
kubectl rollout status deployment/goldilocks-controller -n goldilocks

Verify VPA Recommender has enough data (needs 8+ days for stable recommendations):

kubectl get vpa -n <namespace> <vpa-name> -o json | \
  jq '.status.recommendation.containerRecommendations[] | {container: .containerName, target: .target}'

If target is null, the recommender hasn't collected enough data samples yet — wait for more utilization history.

Integrating Vigilmon for End-to-End Uptime Monitoring

Goldilocks's dashboard is a critical tool for platform engineers performing periodic right-sizing reviews. Vigilmon ensures the dashboard is always available when engineers need it:

  1. Expose the Goldilocks dashboard via Ingress or a LoadBalancer service with authentication (OAuth proxy or basic auth).
  2. Add a Vigilmon monitor targeting your Goldilocks dashboard URL: https://goldilocks.yourdomain.com/.
  3. Check interval: every 2 minutes is appropriate for a non-critical-path tool; use 1 minute if right-sizing reviews happen continuously.
  4. Keyword assertion: check for "Goldilocks" or "VPA Recommendations" in the response body to validate the dashboard is serving real content, not a login redirect or error page.
  5. Alert channels: notify your platform team's Slack channel — this is typically not an on-call emergency, but engineers should know within 10 minutes if the dashboard is down during a right-sizing sprint.

For teams who expose VPA objects via a custom API aggregator, add a second Vigilmon monitor targeting that endpoint to validate end-to-end recommendation data flow.

Monitoring Coverage Summary

| Component | What to Check | Tool | Alert Threshold | |-----------|--------------|------|-----------------| | Controller pod | Ready state | Prometheus / kubectl | Not ready >5m | | Dashboard pod | Ready state + HTTP 200 | Prometheus + Vigilmon | Not ready or HTTP error >5m | | VPA Recommender | Running state | Prometheus | Not ready >10m | | Namespace coverage | Deployments vs VPA count | kubectl comparison script | Gap >5 objects for >30m | | Recommendation freshness | lastTransitionTime on VPA | kubectl / custom metric | Stale >24h on active cluster |

Conclusion

Goldilocks is the interface between raw VPA data and engineering decisions about resource sizing. A broken controller silently leaves new deployments unoptimized; a down dashboard means cost-saving recommendations are inaccessible during right-sizing sprints. By monitoring controller health, VPA object freshness, namespace coverage, and dashboard availability — with Vigilmon providing external uptime assurance — you ensure that Goldilocks actually delivers on its promise: right-sizing Kubernetes workloads to cut cloud costs and improve reliability.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles