tutorial

Monitoring ColossalAI Distributed Training and Chat Servers (Free Alerts)

ColossalAI's Gemini memory manager and pipeline parallelism can hide failures that stall your entire training run. Learn to monitor worker health, GPU memory, the chat inference server, and checkpoint writes — free with Vigilmon.

Monitoring ColossalAI Distributed Training and Chat Servers (Free Alerts)

ColossalAI provides a unified API for tensor parallelism, pipeline parallelism, data parallelism, and sequence parallelism — plus the Gemini memory manager that dynamically moves tensors between GPU HBM, CPU RAM, and NVMe to lower peak memory requirements. But that added flexibility introduces new failure surfaces: Gemini OOM events are different from standard CUDA OOM errors, pipeline micro-batch scheduling can silently stall, and the ColossalAI Chat inference server on port 8080 is a separate process that can die without taking the training nodes with it.

This guide sets up HTTP health monitoring, heartbeat monitors for training milestones, and alerts for the chat inference server — all free on Vigilmon.

The failure modes ColossalAI users miss

Gemini memory manager failures — ColossalAI's Gemini system moves tensors between GPU HBM, CPU RAM, and NVMe dynamically. When the memory manager encounters an error (invalid CUDA device, NVMe mount missing, CPU memory exhausted), it can fail silently or raise exceptions that don't bubble up to the training loop cleanly. Training appears to hang.

Pipeline scheduling stalls — Pipeline parallelism breaks the model into stages and schedules micro-batches. A deadlock between pipeline stages causes all ranks to freeze waiting for messages that never arrive. Without external monitoring, this looks like a slow training run.

Chat server downtime — ColossalAI Chat server (port 8080) is an inference endpoint for trained models, entirely separate from the training cluster. It can crash due to OOM, model loading errors, or dependency failures without affecting the training job — but users on the inference endpoint are immediately affected.

CUDA kernel compilation failures — ColossalAI JIT-compiles custom CUDA kernels on first run. Kernel compilation failures at startup leave you with an unusable environment, but the Python process may still be running.

Step 1: Health endpoint for the training coordinator

Add a health server that runs on rank-0 alongside your training loop:

# colossalai_health_server.py
import threading
import time
import json
import os
from http.server import HTTPServer, BaseHTTPRequestHandler

training_state = {
    "status": "initializing",
    "step": 0,
    "loss": None,
    "pipeline_stage": None,
    "gemini_oom_events": 0,
    "last_step_time": None,
    "last_checkpoint_step": 0,
    "world_size": 0,
    "kernels_compiled": False,
}
state_lock = threading.Lock()


class HealthHandler(BaseHTTPRequestHandler):
    def do_GET(self):
        if self.path != "/health":
            self.send_response(404)
            self.end_headers()
            return

        with state_lock:
            state = dict(training_state)

        stalled = (
            state["status"] == "training"
            and state.get("last_step_time")
            and time.time() - state["last_step_time"] > 600
        )

        healthy = (
            state["status"] in ("training", "checkpointing")
            and not stalled
            and state.get("kernels_compiled", False)
        )

        self.send_response(200 if healthy else 503)
        self.send_header("Content-Type", "application/json")
        self.end_headers()
        self.wfile.write(json.dumps({
            "status": "ok" if healthy else "degraded",
            "training": state,
            "stalled": stalled,
        }).encode())

    def log_message(self, *args):
        pass


def start_health_server(port: int = 8766):
    server = HTTPServer(("0.0.0.0", port), HealthHandler)
    t = threading.Thread(target=server.serve_forever, daemon=True)
    t.start()

In your ColossalAI training script:

import colossalai
import torch
import time
from colossalai_health_server import start_health_server, training_state, state_lock

def train():
    colossalai.launch_from_torch(config=config)
    local_rank = torch.distributed.get_rank()

    if local_rank == 0:
        start_health_server(port=8766)

    # Signal kernel compilation complete
    with state_lock:
        training_state["kernels_compiled"] = True
        training_state["status"] = "training"
        training_state["world_size"] = torch.distributed.get_world_size()

    for step, batch in enumerate(dataloader):
        try:
            loss = engine(batch)
            engine.backward(loss)
            engine.step()

            with state_lock:
                training_state["step"] = step
                training_state["loss"] = loss.item()
                training_state["last_step_time"] = time.time()

        except RuntimeError as e:
            if "out of memory" in str(e).lower():
                with state_lock:
                    training_state["gemini_oom_events"] += 1
            raise

        if step % save_every == 0 and local_rank == 0:
            with state_lock:
                training_state["status"] = "checkpointing"
            save_checkpoint(engine, step)
            with state_lock:
                training_state["status"] = "training"
                training_state["last_checkpoint_step"] = step

Step 2: HTTP monitoring with Vigilmon

Point Vigilmon at your training health endpoint:

  1. Sign up at vigilmon.online — free, no credit card
  2. Click New Monitor → HTTP
  3. Enter http://your-rank0-node:8766/health
  4. Set interval to 1 minute
  5. Save

| Monitor | What it catches | |---|---| | Training health endpoint | Stall, kernel compilation failure, Gemini OOM accumulation | | ColossalAI Chat server (port 8080) | Inference server downtime |

For the chat server, add a second HTTP monitor:

  • URL: http://your-chat-server:8080/health (or any endpoint that returns 200 when the server is up)
  • Interval: 1 minute

If ColossalAI Chat doesn't expose a /health route, monitor the root path — a 200 or 405 both confirm the server is listening.

Step 3: Heartbeat monitors for training milestones

import httpx
import os
import logging

logger = logging.getLogger(__name__)

CHECKPOINT_HEARTBEAT_URL = os.environ.get("CHECKPOINT_HEARTBEAT_URL")
STEP_HEARTBEAT_URL = os.environ.get("STEP_HEARTBEAT_URL")


def ping(url: str):
    if not url:
        return
    try:
        httpx.get(url, timeout=5)
    except Exception as e:
        logger.warning("Heartbeat failed: %s", e)


# Call after each checkpoint save (rank-0 only):
def on_checkpoint_saved(step: int):
    ping(CHECKPOINT_HEARTBEAT_URL)


# Call every 50 steps to signal throughput health:
def on_training_step(step: int):
    if step % 50 == 0:
        ping(STEP_HEARTBEAT_URL)

In Vigilmon, create two Heartbeat monitors:

| Heartbeat | Expected interval | |---|---| | Checkpoint heartbeat | checkpoint_interval_minutes × 2 | | Step heartbeat | 10 minutes (50 steps should complete in under 10 min) |

# .env
CHECKPOINT_HEARTBEAT_URL=https://vigilmon.online/api/heartbeat/your-checkpoint-token
STEP_HEARTBEAT_URL=https://vigilmon.online/api/heartbeat/your-step-token

Step 4: ColossalAI Chat server health check

The chat inference server is a FastAPI application. Add a health route if it doesn't already have one:

# In your ColossalAI Chat server code (if you control it)
from fastapi import FastAPI
import time

app = FastAPI()
server_start_time = time.time()
model_loaded = False


@app.get("/health")
async def health():
    if not model_loaded:
        return JSONResponse(
            status_code=503,
            content={"status": "loading", "uptime_seconds": time.time() - server_start_time}
        )
    return {"status": "ok", "uptime_seconds": time.time() - server_start_time}

If you're running the upstream ColossalAI Chat container without modifying it, create a TCP monitor in Vigilmon instead:

  1. New Monitor → Port
  2. Host: your-chat-server
  3. Port: 8080
  4. Interval: 1 minute

A TCP monitor confirms the port is open even without HTTP. Pair it with an HTTP monitor on the chat API's root path for full coverage.

Step 5: GPU memory and communication backend health

import pynvml
import socket


def get_gpu_memory_health() -> dict:
    try:
        pynvml.nvmlInit()
        gpus = []
        for i in range(pynvml.nvmlDeviceGetCount()):
            h = pynvml.nvmlDeviceGetHandleByIndex(i)
            m = pynvml.nvmlDeviceGetMemoryInfo(h)
            gpus.append({
                "index": i,
                "used_gb": round(m.used / 1e9, 2),
                "total_gb": round(m.total / 1e9, 2),
                "pct": round(m.used / m.total * 100, 1),
            })
        oom_risk = any(g["pct"] > 95 for g in gpus)
        return {"gpus": gpus, "oom_risk": oom_risk}
    except Exception as e:
        return {"error": str(e)}


def check_nccl_port(port: int = 29500) -> bool:
    try:
        s = socket.create_connection(("localhost", port), timeout=2)
        s.close()
        return True
    except Exception:
        return False

Add these to your health endpoint response. Set the HTTP monitor to expect "oom_risk":false using Vigilmon's keyword assertion if your plan supports it — otherwise the 503 status code from your health handler covers this automatically.

Step 6: NVMe offload and checkpoint disk health

import shutil


def check_disk_space(paths: dict) -> dict:
    """Check free space on checkpoint and NVMe paths."""
    results = {}
    for name, path in paths.items():
        try:
            usage = shutil.disk_usage(path)
            free_gb = usage.free / 1e9
            results[name] = {
                "free_gb": round(free_gb, 1),
                "ok": free_gb > 20.0,
            }
        except Exception as e:
            results[name] = {"ok": False, "error": str(e)}
    all_ok = all(v.get("ok", False) for v in results.values())
    return {"disks": results, "ok": all_ok}


# In your health handler:
disk_health = check_disk_space({
    "checkpoints": os.environ.get("CHECKPOINT_DIR", "/checkpoints"),
    "nvme_offload": os.environ.get("NVME_PATH", "/nvme"),
})
healthy = healthy and disk_health["ok"]

Step 7: Slack and email alerts

In Vigilmon:

Slack:

  1. Create an incoming webhook
  2. Notifications → New Channel → Slack → paste URL
  3. Enable on all training and chat server monitors

Email: Notifications → New Channel → Email

Alerts you'll receive:

🔴 DOWN: rank0-node:8766/health
Status: 503 — training stalled (last step 12 minutes ago)
Gemini OOM events: 3

🔴 DOWN: chat-server:8080
Status: connection refused

🔴 MISSED: colossalai-checkpoint heartbeat
Expected every: 60 minutes | Last ping: 88 minutes ago

Step 8: Public status page

Go to Status Pages → New Status Page in Vigilmon. Add:

  • ColossalAI Training (health endpoint)
  • ColossalAI Chat Server
  • Checkpoint and throughput heartbeats

Share the URL with your ML team so anyone can check training status without SSH access.

Summary

| Monitor | What it catches | |---|---| | HTTP /health on rank-0 (1 min) | Training stall, Gemini OOM, kernel failures | | HTTP/TCP on Chat server port 8080 | Inference server downtime | | Checkpoint heartbeat | Worker crash, checkpoint stall | | Step heartbeat | Throughput drop to zero | | Disk health in response | Pre-flight checkpoint failure prevention |

Get started free at vigilmon.online →

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles