tutorial

KubeRay Monitoring: How to Detect Ray Cluster Failures Before They Kill Your ML Workloads

Learn how to monitor KubeRay Ray clusters on Kubernetes — head node health, worker availability, job failures, and GPU utilization — using Vigilmon external monitoring and heartbeat checks.

KubeRay makes it easy to deploy and manage Ray clusters on Kubernetes. Ray powers distributed ML training, hyperparameter tuning, and reinforcement learning at scale. But when a Ray head node crashes, when worker pods fail to join the cluster, or when a Ray job silently hangs, your ML pipeline stalls — and nothing in the Kubernetes control plane tells you users are blocked.

Vigilmon adds the external layer KubeRay lacks: continuous probes from multiple geographic regions that alert you the moment a Ray cluster endpoint goes dark, and heartbeat monitors that catch hung or missing Ray jobs before your data scientists notice stale model outputs.

Why KubeRay Needs External Monitoring

KubeRay manages Ray cluster lifecycle through Kubernetes CRDs (RayCluster, RayJob, RayService). The operator will restart crashed pods according to its reconciliation loop — but reconciliation does not guarantee that the Ray cluster is actually serving traffic or processing jobs. Common failure modes that go undetected:

Head node TCP port is unreachable from outside the cluster. The head pod may be running but the Service or Ingress routing to port 8265 (Ray Dashboard) or 10001 (Ray client) could be misconfigured after a node pool rotation or Ingress controller update.

Ray Dashboard HTTP endpoint returns errors. If the Ray Dashboard is restarting or the head node is in a degraded state, the Dashboard HTTP API will return 500s or connection timeouts — but the pod's liveness probe can still pass if the process is alive.

RayJob completes without pinging your pipeline. A RayJob resource may show Succeeded in Kubernetes while the actual training script exited early due to an OOM kill or GPU driver fault. External heartbeat monitoring catches these silent failures.

Worker nodes fail to register. After a preemption event or spot instance interruption, worker pods may fail to re-join the cluster, leaving Ray with fewer resources than the job requires — causing jobs to queue indefinitely.

Key Metrics and Endpoints to Monitor

| Signal | Endpoint / Method | What Failure Looks Like | |--------|-------------------|------------------------| | Ray Dashboard liveness | http://<ingress>/ — HTTP 200 | 5xx or timeout means head is degraded | | Ray Dashboard API | http://<ingress>/api/cluster_status | Non-200 or "status": "error" | | RayJob completion | Heartbeat ping from job script | Missing ping within grace period | | Ray Serve health | http://<ingress>/api/serve/applications/ | Non-200 means model serving is broken | | Worker count | Dashboard API num_workers field | Drop below expected minimum |

What You'll Set Up

  • HTTP monitor for the Ray Dashboard endpoint
  • HTTP monitor for Ray Serve health (if using model serving)
  • Heartbeat monitor for RayJob completion
  • Alert channel to Slack or PagerDuty

You'll need a free Vigilmon account — no credit card required.

Step 1: Expose the Ray Dashboard via Ingress

If you haven't already exposed the Ray Dashboard externally, create an Ingress for it:

# ray-dashboard-ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ray-dashboard
  namespace: ray-system
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /
spec:
  ingressClassName: nginx
  rules:
    - host: ray.yourdomain.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: raycluster-head-svc
                port:
                  number: 8265

Apply it:

kubectl apply -f ray-dashboard-ingress.yaml

Verify externally:

curl -s https://ray.yourdomain.com/api/cluster_status | jq .status

Step 2: Monitor the Ray Dashboard HTTP Endpoint

  1. Log in to vigilmon.online and go to Monitors → New Monitor
  2. Choose HTTP / HTTPS
  3. Set the URL to https://ray.yourdomain.com/api/cluster_status
  4. Set the check interval to 1 minute
  5. Under Expected response, configure:
    • Status code: 200
    • Response time threshold: 5000ms (Ray Dashboard can be slow under load)
  6. Save the monitor

This catches head node unavailability, Ingress routing failures, and TLS certificate expiry — none of which Kubernetes internal probes detect.

Step 3: Monitor Ray Serve Health (If Using Model Serving)

If you deploy ML models with Ray Serve, add a dedicated monitor for the Serve health endpoint:

  1. Add a second monitor with URL: https://ray.yourdomain.com/api/serve/applications/
  2. Expected status code: 200
  3. Response body contains: "status" (confirms JSON payload is present)
  4. Check interval: 1 minute

When a Serve deployment rolls out a broken model version, this endpoint starts returning 500s before users experience errors — giving you a chance to roll back.

Step 4: Heartbeat Monitoring for RayJob Completion

RayJob resources don't expose an HTTP endpoint that Vigilmon can probe. The correct approach is a heartbeat: your job pings Vigilmon on success, and if the ping doesn't arrive, Vigilmon alerts you.

Create the Heartbeat Monitor

  1. In Vigilmon, go to Monitors → New Monitor → Heartbeat
  2. Name it: ray-training-job
  3. Expected interval: match your job schedule (e.g., 6 hours for a twice-daily training pipeline)
  4. Grace period: 30 minutes
  5. Save — copy the heartbeat URL: https://vigilmon.online/heartbeat/your-unique-token

Wire It Into Your RayJob Script

Add a Vigilmon ping at the end of your training script:

# train.py — called by your RayJob
import os
import urllib.request
import ray

ray.init()

@ray.remote
def train_epoch(epoch: int) -> dict:
    # your training logic here
    return {"epoch": epoch, "loss": 0.01}

def main():
    results = ray.get([train_epoch.remote(i) for i in range(10)])
    print("Training complete:", results)

    heartbeat_url = os.environ.get("VIGILMON_HEARTBEAT_URL")
    if heartbeat_url:
        urllib.request.urlopen(heartbeat_url, timeout=10)
        print("Heartbeat sent to Vigilmon.")

if __name__ == "__main__":
    main()

Configure the RayJob to Pass the Heartbeat URL

# rayjob.yaml
apiVersion: ray.io/v1
kind: RayJob
metadata:
  name: model-training
  namespace: ray-system
spec:
  entrypoint: python /app/train.py
  rayClusterSpec:
    headGroupSpec:
      template:
        spec:
          containers:
            - name: ray-head
              image: your-registry/ray-train:latest
              env:
                - name: VIGILMON_HEARTBEAT_URL
                  valueFrom:
                    secretKeyRef:
                      name: vigilmon-secrets
                      key: training-heartbeat-url
  shutdownAfterJobFinishes: true
  ttlSecondsAfterFinished: 300

Store the heartbeat URL as a Kubernetes Secret:

kubectl create secret generic vigilmon-secrets \
  --namespace ray-system \
  --from-literal=training-heartbeat-url='https://vigilmon.online/heartbeat/your-unique-token'

Now if your training job hangs on a deadlocked Ray actor, runs out of GPU memory and exits silently, or is never scheduled because the cluster has no capacity, Vigilmon will alert you when the heartbeat window expires.

Step 5: Configure Alert Channels

Slack

  1. Create a Slack Incoming Webhook for your #ml-alerts channel
  2. In Vigilmon, go to Alert Channels → New Channel → Webhook
  3. Paste the webhook URL
  4. Assign it to your KubeRay monitors

PagerDuty

For production model serving, route Ray Serve health alerts to an on-call rotation:

  1. Create a PagerDuty service and copy the integration key
  2. In Vigilmon, go to Alert Channels → New Channel → PagerDuty
  3. Enter the integration key
  4. Assign only the Ray Serve monitor to this channel — dashboard downtime can go to Slack, but model serving downtime should page on-call

Recommended Alert Thresholds

| Monitor | Check Interval | Alert After | Notes | |---------|---------------|-------------|-------| | Ray Dashboard | 1 min | 2 failures | Head node restarts take ~60s | | Ray Serve health | 1 min | 1 failure | Model serving downtime is immediate user impact | | RayJob heartbeat | Per job schedule | Grace: 30 min | Adjust grace to match job variance |

Summary

KubeRay's reconciliation loop keeps pods running but cannot tell you whether Ray is actually serving models or completing jobs. Vigilmon gives you the external perspective:

| Failure Mode | KubeRay Operator | Vigilmon | |---|---|---| | Head pod crash | Restarts pod | Alerts on endpoint downtime | | Ingress routing broken | ✗ | ✓ | | Ray Serve model error | ✗ | ✓ via /api/serve/applications/ | | RayJob hangs silently | ✗ | ✓ via heartbeat | | TLS certificate expired | ✗ | ✓ | | GPU OOM silent exit | ✗ | ✓ via heartbeat |

Start monitoring your Ray clusters free at vigilmon.online — setup takes under five minutes.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles