How to Monitor SkyPilot with Vigilmon

SkyPilot is an open-source cloud-agnostic ML job orchestration framework that automatically selects the cheapest available GPU instances across AWS, GCP, Azure, Lambda Labs, and RunPod. It handles spot instance preemption and auto-recovery, distributed training setup, and cluster lifecycle management — letting ML teams run training jobs and inference clusters without cloud vendor lock-in.

SkyPilot jobs can run for hours or days. A training job that silently stalls due to a preempted spot instance that failed to recover, a cluster that never scales back up, or an OOM error that terminates the job without a notification can waste days of GPU budget before anyone notices. This guide covers how to use Vigilmon to keep your SkyPilot workloads observable.

Why Monitor SkyPilot Jobs

SkyPilot abstracts away cloud infrastructure, but the failure modes are real and expensive:

1. Spot instance preemption without recovery — SkyPilot handles most preemptions automatically, but occasionally recovery loops fail; the job stalls with no output
2. Training divergence — a job runs to completion but the model loss curve diverges; without metric monitoring you don't know until evaluation
3. Cluster lifecycle failures — sky launch fails to provision a cluster due to capacity constraints; the job never starts
4. OOM kills — a GPU runs out of memory mid-training, crashes the process, and the job ends without reaching the checkpoint
5. Scheduled sweep jobs — hyperparameter sweep jobs that should run nightly stop silently when the orchestration script has a bug or the cloud account hits quota limits
6. Serving cluster degradation — SkyPilot-managed inference clusters can have individual replicas fail while the load balancer continues routing traffic

Vigilmon adds an independent external layer: heartbeats from job scripts to confirm completion, and HTTP monitors for any serving endpoints SkyPilot manages.

Key Metrics to Monitor

| Metric | What it indicates | |--------|------------------| | Job completion heartbeat | Training/sweep job completed successfully | | Heartbeat interval regularity | Scheduled jobs are running on time | | Serving cluster HTTP availability | Inference endpoint is up and responding | | Response time on inference endpoint | GPU cluster is not overloaded | | Checkpoint file timestamp | Training is making progress (via health script) | | Cluster resource availability | GPU capacity is not exhausted |

Setup Guide

1. Add Heartbeats to SkyPilot Job Scripts

The most critical monitor for any SkyPilot training job is a heartbeat: a ping sent after the job successfully completes. Without it, a failed job is indistinguishable from a job that never ran.

In Vigilmon:

1. Monitors → New Monitor
2. Type: Heartbeat
3. Name: SkyPilot — Nightly Training Run
4. Expected interval: 24 hours
5. Grace period: 2 hours (training jobs have variable runtime)
6. Save — copy the ping URL

Add the ping to your training script's final success block:

# train.py — submitted to SkyPilot
import torch
import requests
import os

VIGILMON_HB = os.environ.get("VIGILMON_HEARTBEAT_URL", "")

def train():
    # ... your training loop ...
    model.save_pretrained("./output/checkpoint-final")
    print("[train] Training complete. Final checkpoint saved.")

    if VIGILMON_HB:
        try:
            requests.get(VIGILMON_HB, timeout=10)
            print("[train] Vigilmon heartbeat sent.")
        except Exception as e:
            print(f"[train] Vigilmon ping failed (non-fatal): {e}")

if __name__ == "__main__":
    train()

Pass the URL via SkyPilot's envs configuration:

# skypilot_train.yaml
name: nightly-training

resources:
  accelerators: A100:1
  cloud: aws
  use_spot: true

envs:
  VIGILMON_HEARTBEAT_URL: "https://vigilmon.online/ping/abc123"

run: |
  pip install -r requirements.txt
  python train.py

Launch with:

sky launch -c training-cluster skypilot_train.yaml

2. Add Mid-Job Progress Heartbeats

For long-running jobs (multi-day training runs), add per-epoch or per-checkpoint heartbeats so you know the job is progressing, not just that it eventually completes:

# train.py with per-checkpoint heartbeats
import requests
import os

VIGILMON_PROGRESS_HB = os.environ.get("VIGILMON_PROGRESS_HB_URL", "")

class VigilmonCallback:
    """Sends a Vigilmon ping after each checkpoint save."""

    def on_save(self, args, state, control, **kwargs):
        if VIGILMON_PROGRESS_HB and state.is_world_process_zero:
            try:
                requests.get(VIGILMON_PROGRESS_HB, timeout=5)
                print(f"[callback] Heartbeat sent at step {state.global_step}")
            except Exception:
                pass  # Non-fatal — training continues
        return control

Set the Vigilmon heartbeat interval to match your checkpoint frequency (e.g. every 2 hours if you checkpoint every 2000 steps). A missed heartbeat means the job stalled mid-training.

3. Monitor SkyPilot-Managed Serving Endpoints

SkyPilot can manage inference clusters that expose HTTP endpoints. Add Vigilmon HTTP monitors for each serving cluster:

In Vigilmon:

1. Monitors → New Monitor
2. Type: HTTP
3. URL: http://your-skypilot-serving-endpoint/health (from sky status --endpoint)
4. Expected status: 200
5. Interval: 2 minutes
6. Save

For OpenAI-compatible serving (vLLM, SGLang on SkyPilot):

# Get your serving endpoint
sky status --endpoint serving-cluster

# Test it manually
curl http://<endpoint>/v1/models \
  -H "Authorization: Bearer token"

Add Vigilmon's keyword check for "object":"list" to confirm the model API is fully initialized, not just that the HTTP server started.

4. Export Cluster Health via a Health Script

SkyPilot's sky status command can be polled from a separate health monitor. Run a lightweight probe script on a schedule that checks cluster state and exposes it as an HTTP endpoint:

# cluster_health_server.py — run on your orchestration machine
import subprocess
import json
from fastapi import FastAPI

app = FastAPI()

@app.get("/health/skypilot")
def skypilot_health():
    try:
        result = subprocess.run(
            ["sky", "status", "--output", "json"],
            capture_output=True,
            text=True,
            timeout=30,
        )
        clusters = json.loads(result.stdout) if result.returncode == 0 else []
        all_ok = all(c.get("status") in ("UP", "INIT") for c in clusters)
        return {
            "ok": all_ok,
            "cluster_count": len(clusters),
            "clusters": [
                {"name": c["name"], "status": c.get("status")} for c in clusters
            ],
        }
    except Exception as e:
        return {"ok": False, "error": str(e)}

Point Vigilmon at https://your-orchestration-host.com/health/skypilot with a keyword check on "ok":true.

5. Monitor the Underlying Cloud Provider APIs

SkyPilot provisions resources via cloud APIs. If AWS, GCP, or Azure has an incident in your target region, SkyPilot jobs will fail to launch or recover. Monitor your primary cloud's status:

# AWS health check (us-east-1 EC2)
https://status.aws.amazon.com/

# GCP status
https://status.cloud.google.com/

# Azure status
https://status.azure.com/

Add HTTP monitors for the cloud status API endpoints relevant to your SkyPilot configuration. A degraded cloud region before a major training run is worth knowing about.

Alerting

Configure Vigilmon alerts for your SkyPilot monitors:

1. Open each monitor → Alerts
2. Add notification channels:
   • Slack: #ml-ops for job failure and serving cluster alerts
   • Email: ML engineer on-call for missed training heartbeats
   • PagerDuty: Production serving cluster failures (if the cluster serves live traffic)

Recommended thresholds:

| Monitor | Alert after | Escalate after | |---------|------------|----------------| | Training job heartbeat | 1 missed heartbeat | N/A (single run) | | Per-checkpoint heartbeat | 1 missed heartbeat | 2 consecutive misses | | Serving cluster HTTP | 2 consecutive failures | 3 consecutive failures | | Cluster health endpoint | 1 failure | 10-minute sustained failure |

Set the response time warning threshold on serving endpoints to 30 000ms — GPU inference is slower than typical HTTP services, but latency above 30 seconds usually indicates an overloaded or misconfigured cluster.

Monitoring a Multi-Cloud SkyPilot Setup

If you run SkyPilot across multiple cloud providers, create a monitor group per cloud:

1. Groups → New Group
2. Name: "SkyPilot — AWS us-east-1"
3. Add all training heartbeats, serving monitors, and cluster health checks for that region
4. Repeat for GCP, Azure, etc.

This makes it easy to correlate a monitor failure with a specific cloud region incident.

Conclusion

SkyPilot makes it economical to run GPU workloads across clouds, but the distributed and long-running nature of ML jobs makes silent failures especially costly. Heartbeat monitors catch training jobs that stall or stop without error. HTTP monitors catch serving cluster degradation. Cluster health endpoints surface infrastructure failures before they cascade into missed model updates.

Add a Vigilmon heartbeat to every SkyPilot training script today — it's five lines of code and catches the most expensive failure mode: a training job that quietly died hours ago.

Further Reading

• Monitoring AI/LLM API Dependencies with Vigilmon
• Monitoring Kubernetes Clusters with Vigilmon
• Monitoring Docker Deployments with Vigilmon