tutorial

Monitoring cast.ai: Ensure Your K8s Cost Optimization Engine Stays Healthy

How to monitor cast.ai's autoscaler, rebalancer, and workload availability so Kubernetes cost cuts never break production uptime.

cast.ai automatically right-sizes nodes, replaces overpriced on-demand instances with spot, and rebalances workloads across node pools to cut Kubernetes cloud costs by 50–70%. But when cast.ai's rebalancer evicts pods aggressively or the autoscaler provisions insufficient capacity, your applications go down — and your cloud bill savings mean nothing if services are unavailable.

This tutorial shows you how to monitor the health of your cast.ai-managed cluster so cost optimization and reliability coexist.

What Is cast.ai?

cast.ai is an autonomous Kubernetes cost optimization platform that:

  • Autoscales nodes — removes underutilized nodes and adds capacity only when needed
  • Spot instance automation — migrates workloads to spot/preemptible instances and handles spot interruptions transparently
  • Rebalancing — periodically evicts and reschedules pods to defragment node utilization
  • Right-sizing — identifies over-provisioned CPU and memory requests and recommends reductions

When these mechanisms work well, costs drop. When they malfunction — rebalancer evicting pods without respecting Pod Disruption Budgets, autoscaler undershooting during traffic spikes, spot interruptions outpacing the fallback buffer — production workloads degrade without any alert from cast.ai itself.

Why You Need External Monitoring

cast.ai provides an internal dashboard showing node savings and rebalancing activity. It does not monitor whether your application endpoints are actually reachable during and after its operations. External monitoring from Vigilmon fills that gap:

Rebalancing windows — cast.ai rebalances at intervals you configure. During rebalancing, pods are evicted and rescheduled. If PDBs are misconfigured or the cluster is undersized, services can lose quorum momentarily. Vigilmon detects the dip immediately.

Spot interruption cascades — spot nodes can be reclaimed with 30 seconds notice. If cast.ai's interrupt handler doesn't drain the node fast enough, pods are force-deleted. Vigilmon catches the resulting 502/503 responses before your users do.

Autoscaler lag — cast.ai scales nodes down aggressively but must scale up reactively. During sudden traffic spikes, there's a window where pending pods wait for new nodes. Vigilmon's response time monitoring shows latency degradation during scale-up events.

Cast.ai agent connectivity — cast.ai's in-cluster agent must maintain a persistent connection to the cast.ai control plane. If the agent loses connectivity, autonomous operations stop silently.

What You'll Set Up

  • HTTP monitors for your critical application endpoints
  • Heartbeat monitors to verify cast.ai rebalancing completes cleanly
  • Response time threshold monitoring for scale-up lag detection
  • Alert routing to Slack or PagerDuty

You'll need a free Vigilmon account — no credit card required.

Step 1: Add a Health Endpoint to Your Applications

Before configuring Vigilmon, make sure each critical service has a /health endpoint that checks real dependencies:

# FastAPI example with dependency checks
from fastapi import FastAPI, Response
import asyncpg
import os

app = FastAPI()

@app.get("/health")
async def health(response: Response):
    checks = {}
    
    try:
        conn = await asyncpg.connect(os.environ["DATABASE_URL"])
        await conn.execute("SELECT 1")
        await conn.close()
        checks["database"] = "ok"
    except Exception as e:
        checks["database"] = f"error: {e}"
        response.status_code = 503
        return {"status": "degraded", "checks": checks}
    
    return {"status": "ok", "checks": checks}

This endpoint is what both your Kubernetes readiness probe and Vigilmon will check — but from entirely different network paths.

Step 2: Monitor Your Application Endpoints

Set up an HTTP monitor for each production service:

  1. Log in to vigilmon.online and go to Monitors → New Monitor
  2. Choose HTTP / HTTPS
  3. Set the URL: https://api.yourdomain.com/health
  4. Set the check interval: 1 minute
  5. Configure expected response:
    • Status code: 200
    • Response body contains: "status":"ok"
    • Response time threshold: 2000ms
  6. Save the monitor

Repeat for each service that cast.ai manages. When a rebalancing cycle evicts pods that don't recover cleanly, Vigilmon fires an alert within 2 minutes.

What to Monitor During Rebalancing Windows

| Endpoint | Why it matters during rebalancing | |---|---| | API gateway /health | First to show 502s if pods are evicted | | Database proxy /health | Spot eviction can hit stateful proxies | | Message queue consumer /health | Eviction causes consumer lag spikes | | Background job endpoint | Rebalancing drops in-flight jobs |

Step 3: Response Time Monitoring for Scale-Up Lag

cast.ai scale-up introduces latency before new node capacity is available. Configure a response time threshold monitor to catch this window:

  1. Edit your existing HTTP monitor
  2. Set Response time threshold to 3000ms
  3. Enable Latency alerting — alert when P95 exceeds threshold for 2 consecutive checks

This fires during scale-up lag without requiring a full service outage. You can correlate the alert with cast.ai's autoscaler logs to tune scaling policies.

Step 4: Heartbeat Monitor for Rebalancing Job Completion

Run a post-rebalancing verification script as a CronJob that pings Vigilmon when the cluster is healthy after a rebalancing cycle:

#!/bin/bash
# cast-ai-post-rebalance-check.sh
set -e

NAMESPACE="${1:-default}"
HEARTBEAT_URL="${VIGILMON_HEARTBEAT_URL}"

# Check that no pods are in Pending state (stuck waiting for nodes)
PENDING=$(kubectl get pods -n "$NAMESPACE" --field-selector=status.phase=Pending \
  -o json | jq '.items | length')

if [ "$PENDING" -gt "5" ]; then
  echo "WARNING: $PENDING pods pending — possible scale-up lag or eviction issue"
  exit 1
fi

# Check that no pods are in CrashLoopBackOff
CRASHES=$(kubectl get pods -n "$NAMESPACE" -o json | \
  jq '[.items[].status.containerStatuses[]? | 
      select(.state.waiting.reason == "CrashLoopBackOff")] | length')

if [ "$CRASHES" -gt "0" ]; then
  echo "WARNING: $CRASHES pods in CrashLoopBackOff — post-rebalance health check failed"
  exit 1
fi

echo "Cluster healthy: 0 stuck pending pods, 0 crash loops"
curl -fsS "$HEARTBEAT_URL" > /dev/null
echo "Heartbeat sent."

Deploy as a CronJob running every 15 minutes:

apiVersion: batch/v1
kind: CronJob
metadata:
  name: cast-ai-health-heartbeat
  namespace: monitoring
spec:
  schedule: "*/15 * * * *"
  jobTemplate:
    spec:
      template:
        spec:
          restartPolicy: OnFailure
          serviceAccountName: cast-ai-monitor-sa
          containers:
            - name: check
              image: bitnami/kubectl:latest
              env:
                - name: VIGILMON_HEARTBEAT_URL
                  valueFrom:
                    secretKeyRef:
                      name: vigilmon-secrets
                      key: cast-ai-heartbeat-url
              command: ["/bin/sh", "/scripts/cast-ai-post-rebalance-check.sh", "production"]
              volumeMounts:
                - name: scripts
                  mountPath: /scripts
          volumes:
            - name: scripts
              configMap:
                name: cast-ai-check-scripts
                defaultMode: 0755

Set up the heartbeat in Vigilmon:

  1. Go to Monitors → New Monitor → Heartbeat
  2. Name it cast.ai cluster health check
  3. Expected interval: 15 minutes
  4. Grace period: 10 minutes
  5. Copy the ping URL into your Kubernetes Secret

Step 5: Monitor the cast.ai Agent

The cast.ai in-cluster agent maintains a WebSocket connection to the control plane. If it disconnects, autonomous cost optimization stops. Add a liveness check:

# Check cast.ai agent pod is running and recently active
kubectl get pods -n castai-agent -l app=castai-agent \
  -o jsonpath='{.items[0].status.containerStatuses[0].ready}'
# Expected: true

Expose this as a ClusterIP service + Ingress health endpoint and add it as an HTTP monitor in Vigilmon with a 2-minute check interval.

Step 6: Configure Alert Channels

  1. In Vigilmon go to Alert Channels → New Channel → Webhook
  2. Add your Slack Incoming Webhook URL for #infrastructure-alerts
  3. Assign all cast.ai monitors to this channel

Sample alert when rebalancing causes a service dip:

{
  "monitor_name": "API Gateway /health",
  "status": "down",
  "url": "https://api.yourdomain.com/health",
  "started_at": "2026-07-12T03:45:00Z",
  "duration_seconds": 45,
  "regions_failing": ["us-east", "us-west"]
}

A 45-second outage at 3:45 AM correlating with cast.ai's scheduled rebalancing window tells you immediately what caused it.

Recommended Alert Thresholds

| Monitor | Interval | Threshold | Escalation | |---|---|---|---| | Application /health endpoints | 1 min | 200 < 2s | Page on-call | | Response time P95 | 1 min | > 3000ms | Alert SRE | | Cluster health heartbeat | 15 min | Grace 10 min | Alert SRE | | cast.ai agent liveness | 2 min | 200 OK | Page on-call | | Pending pods count | 5 min | > 5 pods | Alert SRE |

Summary

cast.ai saves money by aggressively managing your cluster. External monitoring ensures those savings don't come at the cost of reliability:

| cast.ai operation | Risk | Vigilmon monitor | |---|---|---| | Pod rebalancing | Eviction causing service gaps | HTTP endpoint monitor | | Spot interruptions | Pod force-deletion | HTTP + heartbeat | | Autoscaler scale-up | Response time lag | Latency threshold monitor | | Agent disconnection | Optimization stops silently | Agent liveness HTTP monitor |

Start free at vigilmon.online and have your first cast.ai coverage monitor running in under two minutes.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles