tutorial

Monitoring DeepSpeed Distributed Training Jobs (Free Uptime Alerts)

DeepSpeed trains trillion-parameter models across GPU clusters, but a single worker crash or NCCL hang can silently stall your entire run. Learn how to monitor training worker health, GPU memory, checkpointing, and throughput with free external monitoring.

Monitoring DeepSpeed Distributed Training Jobs (Free Uptime Alerts)

DeepSpeed makes billion-parameter model training practical by partitioning optimizer state, gradients, and parameters across GPUs using the Zero Redundancy Optimizer (ZeRO). But distributed training introduces failure modes that don't exist in single-GPU workloads: one worker crash stalls the entire job, NCCL communication hangs freeze all nodes silently, and a failed checkpoint write loses days of GPU time with no notification.

By the end of this guide you'll have HTTP health probes for your training coordinator, heartbeat monitors for training workers, alerts for NCCL failures and checkpoint errors, and a public status page — all on the free tier of Vigilmon.

The failure modes DeepSpeed users miss

Silent worker crashes — DeepSpeed launches one training process per GPU. If a single worker dies due to an OOM kill, SIGKILL, or Python exception, the remaining workers hang waiting for collective operations that will never complete. Your job consumes GPU-hours doing nothing.

NCCL communication deadlocks — GPU-to-GPU communication via NVLink or InfiniBand can time out or deadlock. The symptoms look identical to a slow training run for minutes before it becomes obvious something is wrong. Without external monitoring, you may not notice until you check manually.

Checkpoint write failures — ZeRO-partitioned checkpoints require all GPUs to write simultaneously to a shared filesystem. A single permission error or full disk silently causes checkpoint failure, losing all training progress since the last successful save.

Gradient overflow in fp16 — Mixed-precision training scales loss to avoid underflow. When gradients overflow to inf or NaN, the optimizer skips the update and adjusts the loss scale. Persistent overflow means your model isn't learning, and the training run should be restarted.

Step 1: Expose a training health endpoint

The standard pattern is a small HTTP server that your training coordinator (or rank-0 process) runs alongside the training loop. It exposes the health of the overall run.

# training_health_server.py
import threading
import time
import os
import json
from http.server import HTTPServer, BaseHTTPRequestHandler

# Shared state updated by the training loop
training_state = {
    "status": "initializing",
    "step": 0,
    "loss": None,
    "loss_scale": None,
    "overflow_steps": 0,
    "last_checkpoint_step": 0,
    "last_checkpoint_time": None,
    "workers_alive": 0,
    "started_at": time.time(),
}
state_lock = threading.Lock()


class HealthHandler(BaseHTTPRequestHandler):
    def do_GET(self):
        if self.path != "/health":
            self.send_response(404)
            self.end_headers()
            return

        with state_lock:
            state = dict(training_state)

        # Mark degraded if training stalled (no step progress in 10 minutes)
        stalled = (
            state["status"] == "training"
            and state.get("last_step_time")
            and time.time() - state["last_step_time"] > 600
        )

        healthy = state["status"] in ("training", "checkpointing") and not stalled
        code = 200 if healthy else 503

        self.send_response(code)
        self.send_header("Content-Type", "application/json")
        self.end_headers()
        self.wfile.write(json.dumps({
            "status": "ok" if healthy else "degraded",
            "training": state,
            "stalled": stalled,
        }).encode())

    def log_message(self, *args):
        pass  # suppress request logs


def start_health_server(port: int = 8765):
    """Start the health server in a background thread."""
    server = HTTPServer(("0.0.0.0", port), HealthHandler)
    thread = threading.Thread(target=server.serve_forever, daemon=True)
    thread.start()
    return server

In your DeepSpeed training script (rank-0 only):

import deepspeed
import torch
import time
from training_health_server import start_health_server, training_state, state_lock

def train():
    # Only rank 0 runs the health server
    local_rank = int(os.environ.get("LOCAL_RANK", 0))
    if local_rank == 0:
        start_health_server(port=8765)

    model_engine, optimizer, _, _ = deepspeed.initialize(
        args=args,
        model=model,
        model_parameters=model.parameters(),
    )

    with state_lock:
        training_state["status"] = "training"
        training_state["workers_alive"] = torch.distributed.get_world_size()

    for step, batch in enumerate(dataloader):
        loss = model_engine(batch)
        model_engine.backward(loss)

        # Check for gradient overflow (fp16 mixed precision)
        overflow = model_engine.optimizer.overflow
        with state_lock:
            training_state["step"] = step
            training_state["loss"] = loss.item()
            training_state["loss_scale"] = model_engine.optimizer.loss_scale
            training_state["last_step_time"] = time.time()
            if overflow:
                training_state["overflow_steps"] += 1

        model_engine.step()

        # Checkpoint
        if step % save_every == 0:
            with state_lock:
                training_state["status"] = "checkpointing"
            model_engine.save_checkpoint(output_dir, step)
            with state_lock:
                training_state["status"] = "training"
                training_state["last_checkpoint_step"] = step
                training_state["last_checkpoint_time"] = time.time()

Test your health endpoint:

curl http://localhost:8765/health
# {"status":"ok","training":{"status":"training","step":1024,...},"stalled":false}

Step 2: Monitor training health with Vigilmon HTTP checks

With the health endpoint running, point Vigilmon at it:

  1. Sign up at vigilmon.online — free tier, no credit card
  2. Click New Monitor → HTTP
  3. Enter http://your-training-node:8765/health
  4. Set the check interval to 1 minute (training jobs need fast feedback)
  5. Save

For multi-node training, monitor the rank-0 node's health endpoint — it reflects the state of the full distributed job.

| Monitor | What it catches | |---|---| | http://rank0:8765/health | Job stall, status transitions | | http://rank0:8765/health (keyword: "overflow_steps":0) | Gradient overflow accumulation |

Step 3: Heartbeat monitoring for training milestones

HTTP checks confirm the health server is up, but they won't alert you if training silently stops making progress. Use heartbeat monitors for this.

The pattern: ping Vigilmon's heartbeat URL after each checkpoint save. If you don't receive a ping within 2× your checkpoint interval, an alert fires.

import httpx
import os
import logging

logger = logging.getLogger(__name__)

CHECKPOINT_HEARTBEAT_URL = os.environ.get("CHECKPOINT_HEARTBEAT_URL")
THROUGHPUT_HEARTBEAT_URL = os.environ.get("THROUGHPUT_HEARTBEAT_URL")


def ping_heartbeat(url: str, params: dict = None):
    if not url:
        return
    try:
        httpx.get(url, params=params, timeout=5)
    except Exception as e:
        logger.warning("Heartbeat ping failed: %s", e)


# After each successful checkpoint:
def on_checkpoint_saved(step: int, tokens_per_second: float):
    ping_heartbeat(CHECKPOINT_HEARTBEAT_URL)
    logger.info("Checkpoint saved at step %d (%.0f tok/s)", step, tokens_per_second)


# After each training step (ping a throughput heartbeat every N steps):
def on_training_step(step: int, tokens_per_second: float):
    if step % 100 == 0:
        ping_heartbeat(THROUGHPUT_HEARTBEAT_URL)

In Vigilmon:

  1. Click New Monitor → Heartbeat
  2. For checkpoint heartbeat: set expected interval to checkpoint_interval × 2
  3. For throughput heartbeat: set expected interval to 5 minutes (100 steps should complete in under 5 min for most workloads)
  4. Copy each ping URL and add to your environment:
# .env
CHECKPOINT_HEARTBEAT_URL=https://vigilmon.online/api/heartbeat/your-checkpoint-token
THROUGHPUT_HEARTBEAT_URL=https://vigilmon.online/api/heartbeat/your-throughput-token

Step 4: Monitor GPU memory and NCCL health

For GPU memory and NCCL monitoring, expose additional health data from your health endpoint:

import pynvml
import subprocess

def get_gpu_health() -> dict:
    """Get GPU memory usage across all visible GPUs."""
    try:
        pynvml.nvmlInit()
        gpus = []
        for i in range(pynvml.nvmlDeviceGetCount()):
            handle = pynvml.nvmlDeviceGetHandleByIndex(i)
            mem = pynvml.nvmlDeviceGetMemoryInfo(handle)
            utilization = pynvml.nvmlDeviceGetUtilizationRates(handle)
            gpus.append({
                "index": i,
                "memory_used_gb": round(mem.used / 1e9, 2),
                "memory_total_gb": round(mem.total / 1e9, 2),
                "memory_pct": round(mem.used / mem.total * 100, 1),
                "gpu_utilization_pct": utilization.gpu,
            })
        # Flag any GPU above 95% VRAM — OOM imminent
        any_oom_risk = any(g["memory_pct"] > 95 for g in gpus)
        return {"gpus": gpus, "oom_risk": any_oom_risk}
    except Exception as e:
        return {"error": str(e)}


def check_nccl_rendezvous(port: int = 29500) -> dict:
    """Check if the NCCL rendezvous port is listening."""
    import socket
    try:
        sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
        sock.settimeout(2)
        result = sock.connect_ex(("localhost", port))
        sock.close()
        return {"listening": result == 0, "port": port}
    except Exception as e:
        return {"error": str(e), "listening": False}

Add to your health handler:

# Inside HealthHandler.do_GET
gpu_health = get_gpu_health()
nccl_health = check_nccl_rendezvous()

healthy = (
    state["status"] in ("training", "checkpointing")
    and not stalled
    and not gpu_health.get("oom_risk", False)
    and nccl_health.get("listening", False)
)

Step 5: Checkpoint write health via disk space probe

ZeRO checkpoint writes fail silently when the target filesystem is full. Add a pre-checkpoint disk space check:

import shutil

def check_checkpoint_disk(path: str, min_free_gb: float = 50.0) -> dict:
    """Return error if checkpoint directory has less than min_free_gb available."""
    try:
        usage = shutil.disk_usage(path)
        free_gb = usage.free / 1e9
        ok = free_gb >= min_free_gb
        return {
            "path": path,
            "free_gb": round(free_gb, 1),
            "ok": ok,
            "message": None if ok else f"Only {free_gb:.1f} GB free, need {min_free_gb} GB",
        }
    except Exception as e:
        return {"path": path, "ok": False, "error": str(e)}

Wire it into your health response so Vigilmon's 503 fires before the checkpoint fails:

disk_health = check_checkpoint_disk(
    os.environ.get("CHECKPOINT_DIR", "/checkpoints")
)
healthy = healthy and disk_health.get("ok", False)

Step 6: Slack and email alerts

Configure alert delivery in Vigilmon:

For Slack:

  1. Create an incoming webhook in your Slack workspace
  2. In Vigilmon go to Notifications → New Channel → Slack
  3. Paste your webhook URL and enable it on your monitors

For email: Go to Notifications → New Channel → Email and enter your address.

You'll receive alerts like:

🔴 DOWN: rank0-node:8765/health
Status: 503 degraded — training stalled for 15 minutes
GPU OOM risk: true (GPU 2 at 96.4% VRAM)

🔴 MISSED: deepspeed-checkpoint heartbeat
Expected every: 60 minutes
Last ping: 97 minutes ago

Step 7: CPU offload and elastic training health

If you're using ZeRO-3 CPU offload or ZeRO-Infinity NVMe offload, add I/O health probes:

import psutil
import time

def check_cpu_memory(min_free_gb: float = 16.0) -> dict:
    mem = psutil.virtual_memory()
    free_gb = mem.available / 1e9
    return {
        "free_gb": round(free_gb, 1),
        "ok": free_gb >= min_free_gb,
    }

def check_nvme_io(nvme_path: str = "/nvme") -> dict:
    """Check NVMe mount availability for ZeRO-Infinity offloading."""
    try:
        usage = shutil.disk_usage(nvme_path)
        return {"ok": True, "free_gb": round(usage.free / 1e9, 1)}
    except Exception as e:
        return {"ok": False, "error": str(e)}

For DeepSpeed Elastic training, monitor the coordinator endpoint (default port 29400) with an additional HTTP monitor in Vigilmon.

Step 8: Public status page

When a training run fails on shared infrastructure, stakeholders need visibility.

In Vigilmon:

  1. Go to Status Pages → New Status Page
  2. Name it ML Training Infrastructure
  3. Add your training health, checkpoint, and throughput monitors
  4. Share the URL with your team

The status page automatically shows current and historical incident data — useful for post-mortems on long training runs.

Summary

| Monitor type | What it catches | |---|---| | HTTP on /health (1-min interval) | Job stall, NCCL rendezvous down, OOM risk, disk full | | Checkpoint heartbeat | Checkpoint interval exceeded — likely worker crash or hang | | Throughput heartbeat | Training throughput dropped to zero | | Slack/email alerts | Immediate notification without polling dashboards |

DeepSpeed distributed training is complex enough that silent failures are the norm, not the exception. External monitoring with Vigilmon closes the gap: you find out within minutes, not hours.

Get started free at vigilmon.online →

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles