Monitoring OpenCost: A Practical Guide for Kubernetes Cost Observability

OpenCost is the CNCF-incubated open-source standard for Kubernetes cost monitoring. Unlike proprietary solutions, OpenCost integrates directly with cloud provider billing APIs and Prometheus to deliver real-time cost allocation without vendor lock-in. But as with any observability platform, OpenCost itself needs to be monitored — a stale billing feed or broken API endpoint silently undermines every cost metric it produces.

Why Monitoring OpenCost Is Non-Negotiable

OpenCost runs as a single pod alongside your cost store (typically Prometheus), so any failure cascades directly into zero cost visibility. Common failure modes include:

• Cloud billing API rate limits causing cost data gaps
• Prometheus integration breaking after a scrape config change
• The OpenCost API returning errors that dashboards silently swallow
• Billing data ingestion lagging hours behind real spend

A monitoring blind spot in OpenCost means your platform engineering team doesn't know when cost data becomes unreliable — until an engineer notices the dashboard looks wrong.

Architecture Overview

OpenCost exposes a REST API on port 9003 (by default) and scrapes node pricing data from cloud provider APIs or a local pricing CSV. The data flow is:

Cloud Billing API → OpenCost Pod → Prometheus → Grafana / OpenCost UI

Each link in this chain needs monitoring.

What to Monitor

1. API Endpoint Availability

The primary health signal is the /allocation endpoint. A successful response confirms the API is operational and data is being served:

# Check allocation API
curl -sf "http://opencost.opencost.svc.cluster.local:9003/allocation/compute?window=1d&aggregate=namespace&accumulate=true" | \
  jq '.code'

Expected: 200. Any other code — or a timeout — indicates the OpenCost API process has failed or is overwhelmed.

For a lighter liveness check, use the root endpoint:

curl -sf -o /dev/null -w "%{http_code}" \
  http://opencost.opencost.svc.cluster.local:9003/

OpenCost also exposes a /healthz endpoint in recent versions:

curl -sf http://opencost.opencost.svc.cluster.local:9003/healthz

2. Cloud Billing Data Ingestion Health

OpenCost pulls cloud billing data (AWS Cost and Usage Reports, GCP BigQuery billing exports, Azure Cost Management) to reconcile on-demand pricing with actual spend. When this feed breaks, cost data reverts to list pricing — often significantly underreporting actual costs.

Monitor the cloud cost integration status:

# Check cloud cost reconciliation status
curl -sf "http://opencost.opencost.svc.cluster.local:9003/cloudCost/status" | \
  jq '.data.integration.status'

Look for "success". A "failure" status with a recent lastFailure timestamp means your cloud billing credentials have expired or the billing export pipeline is broken.

For AWS specifically, validate that Cost and Usage Reports are being ingested:

curl -sf "http://opencost.opencost.svc.cluster.local:9003/cloudCost?window=2d&aggregate=service" | \
  jq '.data | length'

A return of 0 or null when you expect cloud cost data signals a broken ingestion pipeline.

3. Cost Allocation Accuracy

Beyond raw availability, validate that allocation data is numerically sane. A misconfigured node pricing file or broken cloud integration can return 0 costs for all resources:

# Check if any cost data exists for the past hour
TOTAL=$(curl -sf "http://opencost.opencost.svc.cluster.local:9003/allocation/compute?window=1h&aggregate=cluster&accumulate=true" | \
  jq '[.data[].totalCost] | add // 0')

echo "Total cluster cost (last hour): $TOTAL"

# Alert if total is suspiciously zero
[ "$(echo "$TOTAL > 0" | bc)" = "1" ] || echo "ALERT: zero cost allocation detected"

In a real cluster running workloads, totalCost for the past hour should always be greater than zero.

4. Prometheus Integration Health

OpenCost queries Prometheus for resource utilization metrics. A broken Prometheus connection causes OpenCost to return zero utilization for all workloads:

# Verify OpenCost can reach Prometheus
curl -sf "http://opencost.opencost.svc.cluster.local:9003/prometheusQuery?query=up" | \
  jq '.data.result | length'

If this returns 0, OpenCost has lost connectivity to Prometheus. Check the PROMETHEUS_SERVER_ENDPOINT environment variable in the OpenCost deployment:

kubectl get deploy opencost -n opencost -o jsonpath='{.spec.template.spec.containers[0].env}' | \
  jq '.[] | select(.name=="PROMETHEUS_SERVER_ENDPOINT")'

Also monitor the OpenCost-specific Prometheus metrics that indicate scrape health:

# OpenCost containers scraped by Prometheus
up{job="opencost"}

# Node CPU allocation metric freshness
(time() - timestamp(node_cpu_hourly_cost)) > 300

5. Pod and Container Health

# Check OpenCost pod status
kubectl get pods -n opencost -l app=opencost

# Watch for OOMKilled restarts — pricing data loading is memory-intensive
kubectl get pod -n opencost -l app=opencost -o jsonpath='{.items[0].status.containerStatuses[0].restartCount}'

High restart counts often indicate the pod is OOMKilled when loading large pricing datasets. Increase memory limits if restartCount keeps climbing.

Prometheus Alerting Rules

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: opencost-alerts
  namespace: opencost
spec:
  groups:
  - name: opencost
    rules:
    - alert: OpenCostAPIDown
      expr: probe_success{job="opencost-api"} == 0
      for: 5m
      labels:
        severity: critical
      annotations:
        summary: "OpenCost API is unreachable"
        description: "The OpenCost /allocation endpoint has been failing for 5 minutes. Cost data is unavailable."

    - alert: OpenCostPodCrashLooping
      expr: rate(kube_pod_container_status_restarts_total{namespace="opencost"}[15m]) > 0
      for: 5m
      labels:
        severity: warning
      annotations:
        summary: "OpenCost pod is crash-looping"
        description: "OpenCost has restarted {{ $value }} times in the last 15 minutes."

    - alert: OpenCostPrometheusDisconnected
      expr: up{job="opencost"} == 0
      for: 10m
      labels:
        severity: critical
      annotations:
        summary: "OpenCost is not being scraped by Prometheus"

    - alert: OpenCostCloudBillingStale
      expr: (time() - opencost_cloud_cost_last_run_time) > 86400
      for: 1h
      labels:
        severity: warning
      annotations:
        summary: "OpenCost cloud billing data is more than 24 hours old"

Integrating Vigilmon for External Monitoring

Prometheus-based alerting catches internal failures, but you also need to know when OpenCost is reachable from the outside — dashboards, CI cost-gate scripts, and finance tooling all need the API to be externally available.

Vigilmon provides continuous HTTP monitoring from outside your cluster:

1. Open Vigilmon and go to Monitors → New Monitor.
2. Set the URL to your externally exposed OpenCost endpoint, e.g. https://opencost.yourdomain.com/allocation/compute?window=1h&aggregate=namespace&accumulate=true.
3. Check interval: 1 minute for critical cost-visibility infrastructure.
4. Add a keyword assertion: check for "code":200 or "data" to validate actual API response, not just TCP connectivity.
5. Alert channels: wire up Slack or PagerDuty so your FinOps or platform team is paged within 1-2 minutes of an outage.

For teams that expose OpenCost's Prometheus metrics externally, add a second Vigilmon monitor targeting the /metrics endpoint and check for opencost_build_info to confirm the process is alive.

Monitoring Checklist

| Signal | Method | Threshold | |--------|--------|-----------| | API availability | HTTP probe on /healthz | Any failure >5m | | Allocation data presence | JSON response length | 0 results for >15m | | Total cost > 0 | Query totalCost field | Zero for >30m in running cluster | | Cloud billing freshness | lastRun timestamp | >24h stale | | Prometheus connectivity | up{job="opencost"} | 0 for >10m | | Pod restarts | restartCount | >3 in 15m window |

Conclusion

OpenCost's simplicity is a strength, but it also means a single pod failure takes out your entire cost monitoring layer. By combining internal Prometheus alerts with Vigilmon's external HTTP checks, you get defense in depth: internal failures trigger Prometheus rules, while external availability and API correctness are continuously validated. For open-source K8s cost monitoring, this layered approach ensures cost data remains trustworthy when engineering and finance teams depend on it most.