Ray Serve is the serving layer of the Ray ecosystem — a scalable framework for deploying machine learning models and LLM inference pipelines as HTTP endpoints. Because Ray Serve manages dynamic replica scaling, model multiplexing, and request batching under the hood, its failure modes look nothing like a traditional web service going down: deployments can enter degraded states where some replicas are healthy and some are not, throughput collapses silently, or latency percentiles spike while HTTP 200s keep flowing.
In this tutorial you'll set up comprehensive uptime and latency monitoring for Ray Serve using Vigilmon — free tier, no credit card required.
Why Ray Serve deployments need external monitoring
Ray Serve's distributed architecture means internal Prometheus metrics and Ray Dashboard only tell part of the story. An external monitor sees what your end users actually experience: whether the endpoint responds, how fast, and whether the response content indicates a healthy model.
Key failure modes that internal metrics miss:
- Replica health divergence — the head node reports N replicas but some are in
UNHEALTHYstatus while still accepting and silently degrading requests - Model loading failures on autoscale — when Ray Serve scales up a new replica, the model weights may fail to load; the replica appears in the pool but returns 500s until restarted
- LLM context length saturation — inference backends (vLLM, TGI) fail requests above context limits with non-obvious error codes that look like application errors, not infrastructure failures
- Batching queue saturation — request queues fill faster than replicas drain them; end-to-end latency climbs past SLOs while CPU utilization looks normal
- GCS (Global Control Store) unavailability — Ray's distributed metadata store goes down; new requests are dropped but existing replicas keep running and returning stale or incorrect results
- Head node restart — the Ray head restarts and re-registers deployments; there's a gap where the HTTP proxy is live but the deployment backend isn't ready yet
An external HTTP monitor with response-content validation catches all of these from the user's perspective.
What you'll need
- A running Ray Serve deployment with at least one HTTP endpoint
- The endpoint accessible from a public or VPN-reachable IP/hostname
- A free Vigilmon account (sign up in 30 seconds)
Step 1: Add a health endpoint to your Ray Serve application
Ray Serve's @serve.deployment decorator exposes HTTP handlers. Add a /health route that validates the model is ready to serve:
# serve_app.py
import ray
from ray import serve
from starlette.requests import Request
from starlette.responses import JSONResponse
import time
ray.init()
serve.start()
@serve.deployment(
num_replicas=2,
ray_actor_options={"num_cpus": 1},
health_check_period_s=10,
health_check_timeout_s=5,
)
class TextClassifier:
def __init__(self):
# Load model weights during init — failures here cause replica UNHEALTHY
from transformers import pipeline
self.model = pipeline("text-classification", model="distilbert-base-uncased-finetuned-sst-2-english")
self._ready = True
async def __call__(self, request: Request):
body = await request.json()
text = body.get("text", "")
result = self.model(text)
return JSONResponse({"label": result[0]["label"], "score": result[0]["score"]})
def check_health(self):
# Ray Serve calls this on health_check_period_s intervals
if not self._ready:
raise RuntimeError("Model not loaded")
# Expose a standalone health endpoint that reports deployment status
@serve.deployment(route_prefix="/health")
class HealthCheck:
async def __call__(self, request: Request):
status = serve.status()
deployments = status.deployment_statuses
all_ok = all(
d.status == "HEALTHY"
for d in deployments.values()
)
if not all_ok:
unhealthy = [
name for name, d in deployments.items()
if d.status != "HEALTHY"
]
return JSONResponse(
{"status": "degraded", "unhealthy_deployments": unhealthy},
status_code=503,
)
return JSONResponse({"status": "ok", "deployments": len(deployments)})
serve.run(TextClassifier.bind(), route_prefix="/classify")
serve.run(HealthCheck.bind(), route_prefix="/health")
Verify the endpoint:
curl http://localhost:8000/health
# {"status": "ok", "deployments": 2}
Step 2: Add a latency probe endpoint
Latency percentiles matter for ML serving. Add a /probe endpoint that runs a minimal inference and returns timing:
@serve.deployment(route_prefix="/probe")
class LatencyProbe:
def __init__(self):
self._start_time = time.time()
async def __call__(self, request: Request):
t0 = time.time()
# Minimal model call — use your actual model handle here
elapsed_ms = (time.time() - t0) * 1000
uptime_s = time.time() - self._start_time
return JSONResponse({
"status": "ok",
"probe_latency_ms": round(elapsed_ms, 2),
"uptime_seconds": round(uptime_s),
})
serve.run(LatencyProbe.bind(), route_prefix="/probe")
Step 3: Expose Ray Serve behind a reverse proxy
Ray Serve's HTTP proxy binds to 0.0.0.0:8000 by default. For production, put nginx in front to handle TLS and provide a stable public URL:
# /etc/nginx/sites-available/ray-serve
server {
listen 443 ssl;
server_name ml.your-domain.com;
ssl_certificate /etc/letsencrypt/live/ml.your-domain.com/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/ml.your-domain.com/privkey.pem;
location / {
proxy_pass http://127.0.0.1:8000;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_read_timeout 120s;
}
}
sudo ln -s /etc/nginx/sites-available/ray-serve /etc/nginx/sites-enabled/
sudo nginx -t && sudo systemctl reload nginx
Step 4: Add a Vigilmon HTTP monitor for deployment health
Log in to Vigilmon and go to Monitors → Add Monitor.
| Field | Value |
|-------|-------|
| Monitor type | HTTP |
| Name | Ray Serve — deployment health |
| URL | https://ml.your-domain.com/health |
| Check interval | 1 minute |
| HTTP method | GET |
| Expected status | 200 |
| Response must contain | "status":"ok" |
| Regions | Select 2+ regions geographically close to your inference cluster |
Click Save Monitor. Vigilmon will immediately begin polling from all selected regions.
Step 5: Add a latency monitor for the probe endpoint
Add a second monitor targeting the /probe endpoint. This catches throughput degradation and replica overload even when the /health endpoint still returns 200:
| Field | Value |
|-------|-------|
| Monitor type | HTTP |
| Name | Ray Serve — inference latency probe |
| URL | https://ml.your-domain.com/probe |
| Check interval | 2 minutes |
| Expected status | 200 |
| Response must contain | "status":"ok" |
| Timeout | 10 seconds (alert if inference takes longer than this) |
Setting a tight timeout on this monitor means Vigilmon will flag latency regressions before your users notice them.
Step 6: Monitor the Ray head node dashboard port
Ray's head node exposes a dashboard on port 8265 that goes down when the head node is unhealthy. Add a TCP monitor to detect head node outages:
# Verify the port is reachable
curl -I http://<ray-head-ip>:8265
In Vigilmon, add a TCP monitor:
| Field | Value |
|-------|-------|
| Monitor type | TCP |
| Name | Ray head node dashboard |
| Host | <ray-head-external-ip> |
| Port | 8265 |
| Check interval | 1 minute |
Step 7: Configure autoscaling health via a custom metrics endpoint
Ray Serve's autoscaler can stall — adding replicas that never become healthy. Expose autoscaler state as a Vigilmon-pollable endpoint:
# autoscale-status/app.py
from flask import Flask, jsonify
import requests
app = Flask(__name__)
RAY_DASHBOARD = "http://localhost:8265"
@app.route("/health/autoscale")
def autoscale_health():
try:
resp = requests.get(f"{RAY_DASHBOARD}/api/serve/applications/", timeout=5)
resp.raise_for_status()
data = resp.json()
except Exception as e:
return jsonify({"status": "error", "detail": str(e)}), 503
apps = data.get("applications", {})
stalled = []
for name, app_info in apps.items():
for dep_name, dep in app_info.get("deployments", {}).items():
replicas = dep.get("replica_states", {})
starting = replicas.get("STARTING", 0)
unhealthy = replicas.get("UNHEALTHY", 0)
if unhealthy > 0 or starting > 5:
stalled.append(f"{name}/{dep_name}")
if stalled:
return jsonify({"status": "degraded", "stalled": stalled}), 503
return jsonify({"status": "ok", "apps": list(apps.keys())})
if __name__ == "__main__":
app.run(host="0.0.0.0", port=9100)
Add a Vigilmon monitor for this endpoint:
| Field | Value |
|-------|-------|
| Monitor type | HTTP |
| Name | Ray Serve — autoscaler health |
| URL | http://<your-server>:9100/health/autoscale |
| Expected status | 200 |
| Response must contain | "status":"ok" |
Step 8: Configure alerts
In Vigilmon, go to Alerts → Alert Channels and connect your notification channels (email, Slack, PagerDuty, webhook).
Apply an alert policy to each Ray Serve monitor:
| Setting | Recommended value | |---------|-------------------| | Alert after | 2 consecutive failures | | Recovery alert | Enabled | | Alert channel | Your ML ops on-call channel |
For the latency probe monitor, a single-failure threshold is appropriate — a single timeout already indicates a serious problem for inference workloads.
Step 9: Set up a status page
If you offer ML inference as a service to customers, create a public status page:
- Navigate to Status Pages → Create Status Page
- Add your deployment health monitor as "Inference API"
- Add the latency probe as "Inference Performance"
- Leave the autoscaler health monitor off the public page — it's internal infrastructure
What you're monitoring now
| Monitor | What it detects | |---------|-----------------| | Deployment health HTTP | Unhealthy replicas, model load failures, GCS outages | | Latency probe HTTP | Throughput degradation, batching queue saturation, timeout spikes | | Ray head node TCP | Head node crashes, dashboard unavailability | | Autoscaler health HTTP | Stalled scale-up, replicas stuck in STARTING/UNHEALTHY state |
Conclusion
Ray Serve's distributed architecture makes it powerful for ML serving but opaque when things go wrong. Internal Prometheus metrics and the Ray Dashboard tell you what Ray thinks is happening — Vigilmon tells you what your users are actually experiencing. With HTTP content checks, timeout-based latency monitoring, and multi-region coverage, you'll catch replica failures, model loading errors, and throughput collapses before they reach your users.
Sign up for a free Vigilmon account and add your first Ray Serve monitor today.