Ray is the open-source framework for scaling Python ML workloads — from hyperparameter tuning with Ray Tune to distributed training with Ray Train, online serving with Ray Serve, and reinforcement learning with RLlib. Teams running Ray on Kubernetes, AWS EC2, or on-premises clusters depend on it for production model training and inference. When the Ray head node goes down or a Serve deployment crashes, model training stalls and API predictions fail silently.
This tutorial covers how to monitor Ray clusters, Ray Serve deployments, and distributed training jobs using Vigilmon — external uptime monitoring that catches what the Ray Dashboard misses.
Why Ray clusters need external monitoring
Ray's distributed architecture creates failure modes that internal dashboards don't surface well:
- Head node failure — when the Ray head node crashes, the entire cluster loses coordination; workers keep running but no new tasks are scheduled, and jobs stall indefinitely
- Ray Serve replica crashes — a Serve deployment's replicas OOM and crash; Ray restarts them but during the restart window all incoming requests return 503, and the Serve dashboard may show the deployment as "healthy" before replicas finish restarting
- Plasma store exhaustion — the shared memory object store fills up; workers start spilling to disk and task throughput drops by 10–100x without any visible error
- Dashboard proxy failure — the Ray dashboard is itself an HTTP service that can crash independently of the cluster; you lose observability exactly when you need it most
- Job stalls from dead actors — a long-running Ray actor handles training coordination; if it crashes without supervision, the training loop blocks forever on
ray.get()
External monitoring adds an independent layer that checks the cluster's observable behavior — not just what it reports internally.
What you'll need
- A Ray cluster (Ray 2.x, on Kubernetes with KubeRay or standalone)
- A publicly reachable Ray Serve deployment or custom health endpoint
- A free Vigilmon account
Step 1: Monitor the Ray Dashboard
The Ray Dashboard runs an HTTP server on port 8265 by default. Vigilmon can probe it directly:
- Log in to vigilmon.online → Monitors → New Monitor → HTTP / HTTPS
- URL:
http://<ray-head-node-ip>:8265 - Expected status:
200 - Check interval: 1 minute
- Save
This verifies that the head node's HTTP stack is alive. If the head node crashes or the dashboard service exits, Vigilmon opens an incident immediately.
For the Ray REST API (available on port 8265 under /api):
- New Monitor → HTTP / HTTPS
- URL:
http://<ray-head-node-ip>:8265/api/cluster_status - Expected status:
200 - Save
Step 2: Monitor Ray Serve deployments
Ray Serve deployments expose HTTP endpoints that your models serve predictions on. These are the most critical monitors: if the deployment is down, predictions fail.
Add a /health route to your Serve deployment
import ray
from ray import serve
from fastapi import FastAPI
app = FastAPI()
@serve.deployment(
num_replicas=2,
ray_actor_options={"num_cpus": 1}
)
@serve.ingress(app)
class ModelDeployment:
def __init__(self):
# Load your model here
self.model = load_model("s3://models/my-model-v3")
self._ready = True
@app.get("/health")
def health(self):
if not self._ready:
from fastapi.responses import JSONResponse
return JSONResponse(
{"status": "loading"},
status_code=503
)
return {"status": "ok", "model_version": "v3"}
@app.post("/predict")
async def predict(self, payload: dict):
result = self.model.predict(payload["features"])
return {"prediction": result}
Add a Vigilmon monitor on /health:
- Monitors → New Monitor → HTTP / HTTPS
- URL:
https://your-ray-serve-host.example.com/health - Expected status:
200 - Keyword:
"status":"ok" - Check interval: 1 minute
- Save
Step 3: Build a cluster health endpoint
Ray's internal APIs expose cluster health information. Wrap them in a lightweight service:
# ray_cluster_health.py — run as a separate Flask process on the head node
from flask import Flask, jsonify
import ray, time, os
app = Flask(__name__)
# Initialize Ray connection (connects to the running cluster)
ray.init(address="auto", ignore_reinit_error=True)
@app.route("/cluster-health")
def cluster_health():
try:
nodes = ray.nodes()
alive_nodes = [n for n in nodes if n.get("Alive", False)]
dead_nodes = [n for n in nodes if not n.get("Alive", False)]
# Get cluster resources
available = ray.available_resources()
total = ray.cluster_resources()
cpu_used = total.get("CPU", 0) - available.get("CPU", 0)
cpu_total = total.get("CPU", 0)
# Healthy if: at least 1 alive node, head node alive, < 95% CPU used
cpu_pct = (cpu_used / cpu_total * 100) if cpu_total > 0 else 0
status = "ok" if alive_nodes and cpu_pct < 95 else "degraded"
return jsonify({
"status": status,
"alive_nodes": len(alive_nodes),
"dead_nodes": len(dead_nodes),
"cpu_used_pct": round(cpu_pct, 1),
"gpus_available": available.get("GPU", 0),
"gpus_total": total.get("GPU", 0)
}), 200 if status == "ok" else 503
except Exception as e:
return jsonify({"status": "error", "error": str(e)}), 503
@app.route("/serve-health")
def serve_health():
"""Check Ray Serve deployment status via the Serve HTTP proxy."""
from ray.serve._private.client import ServeControllerClient
try:
client = serve.get_deployment_handle # just check connectivity
status = serve.status()
apps = status.applications
unhealthy = [
name for name, app in apps.items()
if app.status not in ("RUNNING", "HEALTHY")
]
if unhealthy:
return jsonify({
"status": "degraded",
"unhealthy_apps": unhealthy
}), 503
return jsonify({"status": "ok", "apps": len(apps)}), 200
except Exception as e:
return jsonify({"status": "error", "error": str(e)}), 503
if __name__ == "__main__":
app.run(host="0.0.0.0", port=8266)
Add Vigilmon monitors for /cluster-health and /serve-health.
Step 4: Monitor Ray training jobs
Long-running distributed training jobs need progress monitoring — a job that stops logging progress is likely stuck:
# Inside your Ray Train training loop
import ray
from ray.train import Checkpoint
import time, json
@ray.remote
class TrainingProgressReporter:
"""Actor that tracks training progress and serves a health endpoint."""
def __init__(self):
self._last_step = None
self._last_step_time = time.time()
self._total_steps = 0
def report_step(self, step: int, loss: float):
self._last_step = step
self._last_step_time = time.time()
self._total_steps += 1
def get_health(self):
if self._last_step is None:
return {"status": "not_started"}
age = time.time() - self._last_step_time
# Alert if no progress in 10 minutes
status = "ok" if age < 600 else "stalled"
return {
"status": status,
"last_step": self._last_step,
"seconds_since_last_step": int(age)
}
Expose the reporter's get_health() from a Flask endpoint:
reporter = TrainingProgressReporter.remote()
@app.route("/training-health")
def training_health():
result = ray.get(reporter.get_health.remote())
code = 200 if result["status"] == "ok" else 503
return jsonify(result), code
Add this as a Vigilmon HTTP monitor with keyword check "status":"ok".
Step 5: Monitor the Ray GCS (Global Control Service)
The Global Control Service manages cluster state. It runs on port 6379 (Redis-compatible) by default:
- In Vigilmon → Monitors → New Monitor → TCP Port
- Host:
<ray-head-node-ip> - Port:
6379 - Save
Also monitor the GCS gRPC port (10001):
- New Monitor → TCP Port
- Port:
10001 - Save
These TCP checks catch GCS crashes that wouldn't show up in HTTP monitoring.
Step 6: Configure Vigilmon alerts
Route incidents to your team immediately:
- Alert Channels → Add Channel → Webhook
- Enter your Slack incoming webhook URL
- Assign to all Ray monitors
Push alerts directly from your training script on failure:
import requests, os
def push_training_alert(message: str):
api_key = os.environ.get("VIGILMON_API_KEY")
monitor_id = os.environ.get("VIGILMON_MONITOR_ID")
if not api_key or not monitor_id:
return
try:
requests.post(
f"https://vigilmon.online/api/monitors/{monitor_id}/incidents",
headers={"Authorization": f"Bearer {api_key}"},
json={"message": message, "severity": "critical"},
timeout=5
)
except Exception:
pass
# In your training entrypoint
try:
trainer.fit()
except Exception as e:
push_training_alert(f"Ray Train job failed: {type(e).__name__}: {e}")
raise
Step 7: Monitor on KubeRay
If you run Ray on Kubernetes with KubeRay, add a Kubernetes readiness probe alongside Vigilmon:
# raycluster.yaml excerpt
headGroupSpec:
template:
spec:
containers:
- name: ray-head
readinessProbe:
httpGet:
path: /api/cluster_status
port: 8265
initialDelaySeconds: 30
periodSeconds: 10
livenessProbe:
httpGet:
path: /api/cluster_status
port: 8265
initialDelaySeconds: 60
periodSeconds: 30
The Kubernetes probe restarts the container on failure. Vigilmon monitors the external-facing Serve endpoints to catch cases where Kubernetes shows the pod as healthy but the service is unreachable from outside the cluster.
Step 8: Create a status page for your ML platform
- In Vigilmon → Status Pages → New Status Page
- Add monitors: Ray Dashboard, Serve deployment health, cluster health, GCS TCP ports
- Publish and share with your ML engineering team and model consumers
Monitoring checklist for Ray
| What to monitor | Vigilmon type | Alert condition |
|---|---|---|
| Ray Dashboard (port 8265) | HTTP | Non-200 response |
| Ray Serve /health | HTTP | Non-200 or "status":"loading" |
| Cluster health endpoint | HTTP | "status":"degraded" or non-200 |
| Serve deployment status | HTTP | unhealthy_apps non-empty |
| Training job progress | HTTP | "status":"stalled" |
| GCS Redis port (6379) | TCP Port | Connection refused |
| GCS gRPC port (10001) | TCP Port | Connection refused |
Summary
Ray is critical infrastructure for teams running distributed ML — and it fails in subtle ways that the Ray Dashboard doesn't surface. With Vigilmon you get:
- 1-minute probes on Ray Serve deployments catching inference outages
- Cluster health monitoring that alerts before workers start dropping tasks
- Training job progress checks that catch stalled jobs before training hours are wasted
- TCP monitoring of the GCS ports that coordinate cluster state
- A public status page for your ML platform