tutorial

Monitoring Determined AI with Vigilmon

Determined AI is the open source deep learning training platform for distributed GPU workloads — here's how to monitor its master server, PostgreSQL backend, agent nodes, experiment API, and checkpoint storage with Vigilmon.

Determined AI is the open source deep learning training platform that handles distributed training orchestration, hyperparameter search (AutoML/NAS), experiment management, and cluster resource management for GPU workloads. Teams running large-scale model training on GPU clusters — whether on Kubernetes or on-premises hardware — rely on Determined to allocate GPUs, schedule trials, and coordinate multi-node training. An unmonitored Determined cluster can silently stop scheduling experiments, waste expensive GPU hours on idle or crashed agents, or lose checkpoint data. Vigilmon lets you track every critical layer of the Determined stack from a single dashboard.

What You'll Set Up

  • Determined master server availability monitor (port 8080)
  • PostgreSQL database connectivity check
  • Web UI health check
  • Experiment scheduling API health check
  • Per-agent connectivity monitoring
  • Checkpoint storage health check
  • GPU resource utilization heartbeat
  • Trial log streaming endpoint check
  • SSL/TLS certificate expiry monitoring

Prerequisites

  • Determined master server running on port 8080
  • PostgreSQL running as the primary data store
  • At least one Determined agent running on a compute node
  • A free Vigilmon account

Step 1: Monitor the Determined Master Server

The Go master process at port 8080 orchestrates all experiments, allocates GPU resources, and serves both the web UI and the API. If the master goes down, no new experiments can be submitted, running trials lose their orchestrator, and the dashboard becomes inaccessible.

  1. Log in to vigilmon.online and click Add Monitor.
  2. Set Type to HTTP / HTTPS.
  3. Enter the master info endpoint: http://your-determined-host:8080/info
  4. Set Check interval to 1 minute.
  5. Set Expected HTTP status to 200.
  6. Click Save.

The /info endpoint returns cluster metadata including the master version and is a reliable, unauthenticated liveness probe.

Step 2: Monitor PostgreSQL Database Connectivity

PostgreSQL stores all experiment metadata, hyperparameter configurations, checkpoints, trial results, and cluster state. It is critical for all Determined operations — without it, the master cannot read experiment configs, write trial results, or track checkpoint references.

  1. Click Add Monitor → TCP Port.
  2. Enter your PostgreSQL host and port: your-pg-host:5432.
  3. Set Check interval to 1 minute.
  4. Click Save.

For a deeper check, add a monitor for the Determined master's database-backed API:

  1. Click Add Monitor → HTTP / HTTPS.
  2. Enter: http://your-determined-host:8080/api/v1/experiments?limit=1
  3. Under Request Headers, add Authorization: Bearer YOUR_API_KEY.
  4. Set Expected HTTP status to 200.
  5. Click Save.

A successful experiments query proves both the master API and the PostgreSQL connection are healthy.

Step 3: Monitor Determined Agent Connectivity

Each compute node runs a Determined agent that communicates with the master to receive task assignments and report status. A dead agent means those GPU slots are silently removed from the schedulable pool — experiments may queue indefinitely without error.

Use the Determined master's agent listing API to check agent health. Because the agent list changes as nodes scale in and out, monitor the master API rather than individual agent endpoints:

  1. Click Add Monitor → HTTP / HTTPS.
  2. Enter: http://your-determined-host:8080/api/v1/agents
  3. Under Request Headers, add Authorization: Bearer YOUR_API_KEY.
  4. Set Expected HTTP status to 200.
  5. Add a response body contains assertion for the known agent ID of a critical node (e.g., "id":"agent-gpu-01").
  6. Set Check interval to 2 minutes.
  7. Click Save.

For clusters with many agents, use a Vigilmon webhook integration to call this endpoint and alert when the agent count drops below a minimum threshold.

Step 4: Monitor the Web UI

The Determined web UI (/det/login) is the React dashboard for managing experiments, trials, and clusters. A UI outage does not stop running trials but blocks your team from submitting new jobs and monitoring progress.

  1. Click Add Monitor → HTTP / HTTPS.
  2. Enter: http://your-determined-host:8080/det/login
  3. Set Expected HTTP status to 200.
  4. Set Check interval to 2 minutes.
  5. Click Save.

Step 5: Monitor the Experiment Scheduling API

The experiment submission endpoint (POST /api/v1/experiments) accepts new training jobs and validates GPU slot allocation. If this endpoint fails, no new experiments can be started — your team's training queue silently stops accepting work.

Because POST endpoints cannot be probed without a valid experiment config body, use a GET probe on the experiments list:

  1. Click Add Monitor → HTTP / HTTPS.
  2. Enter: http://your-determined-host:8080/api/v1/experiments?limit=1
  3. Under Request Headers, add Authorization: Bearer YOUR_API_KEY.
  4. Set Expected HTTP status to 200.
  5. Set Check interval to 1 minute.
  6. Click Save.

Step 6: Monitor Checkpoint Storage Health

Determined saves model checkpoints to shared storage — NFS, S3, GCS, or a configured checkpoint directory. If the checkpoint path is unmounted or write permissions are lost, trials complete but their checkpoints are not saved, making the entire training run unrecoverable.

Use a Vigilmon cron heartbeat to validate checkpoint storage write access:

  1. Click Add Monitor → Cron Heartbeat.
  2. Set the expected ping interval to 15 minutes.
  3. Copy the heartbeat URL: https://vigilmon.online/heartbeat/abc123.
  4. On the master host or a shared compute node, schedule a check:
#!/bin/bash
# /etc/cron.d/determined-checkpoint-check — runs every 15 minutes
CHECKPOINT_DIR="/mnt/checkpoints"  # adjust to your configured path
TEST_FILE="$CHECKPOINT_DIR/.vigilmon-write-test"

if touch "$TEST_FILE" 2>/dev/null && rm -f "$TEST_FILE"; then
  curl -s https://vigilmon.online/heartbeat/abc123
fi

If the checkpoint directory is unmounted or read-only, the write test fails, the heartbeat is not sent, and Vigilmon alerts after 15 minutes.

For S3/GCS checkpoint storage, replace the file write test with an aws s3 cp or gsutil cp to a test object in your checkpoint bucket.

Step 7: Monitor GPU Resource Utilization

Idle GPU hours indicate scheduling inefficiency — agents connected but no experiments queued. Full saturation with a long queue indicates capacity need. Use the Determined slots API to monitor resource utilization:

  1. Click Add Monitor → HTTP / HTTPS.
  2. Enter: http://your-determined-host:8080/api/v1/resourcePools
  3. Under Request Headers, add Authorization: Bearer YOUR_API_KEY.
  4. Set Expected HTTP status to 200.
  5. Set Check interval to 5 minutes.
  6. Click Save.

For actionable GPU utilization alerts, configure a Vigilmon webhook that polls this endpoint on each check and triggers if slotsOccupied drops to 0 during business hours (unexpected idle state) or if the pending queue exceeds a threshold.

Step 8: Monitor Trial Log Streaming

The WebSocket endpoint for live training logs (GET /api/v1/trials/:id/logs) lets your team watch training progress in real time. A failure here does not stop training but means engineers cannot debug in-progress runs without SSH access to compute nodes.

Check that the logs API is reachable:

  1. Click Add Monitor → HTTP / HTTPS.
  2. Enter: http://your-determined-host:8080/api/v1/trials
  3. Under Request Headers, add Authorization: Bearer YOUR_API_KEY.
  4. Set Expected HTTP status to 200.
  5. Set Check interval to 5 minutes.
  6. Click Save.

Step 9: SSL/TLS Certificate Expiry Monitoring

If Determined is served over HTTPS (recommended for any cluster accessible beyond localhost), add a certificate expiry check:

  1. Open the HTTP monitor you created in Step 1.
  2. Update the URL to https://your-determined-host/info.
  3. Enable Monitor SSL certificate.
  4. Set Alert when certificate expires in less than 21 days.
  5. Click Save.

A TLS error on the master API blocks all CLI and SDK clients from submitting experiments.

Step 10: Configure Alert Channels

  1. Go to Alert Channels in Vigilmon and add Slack, email, or a webhook.
  2. Set Consecutive failures before alert to 1 on the master server — any master outage is an emergency for a training cluster.
  3. Set alert threshold to 2 on the web UI monitor — the UI can briefly restart during deployments.
  4. For agent connectivity, configure a webhook alert that fires if the agent count drops below your cluster minimum.

For overnight training jobs, configure alert escalation so master failures and checkpoint storage outages page the on-call engineer immediately, even outside business hours.

Summary

| Monitor | Target | What It Catches | |---|---|---| | Master server | :8080/info | Master process crash or restart | | PostgreSQL TCP | :5432 | Database unreachable | | Experiments API | /api/v1/experiments | API + DB layer failure | | Agent list | /api/v1/agents | Agent node disconnected | | Web UI | /det/login | Dashboard unavailable | | Resource pools | /api/v1/resourcePools | GPU utilization anomaly | | Checkpoint storage | Cron heartbeat | Checkpoint dir unmounted or full | | Trial logs | /api/v1/trials | Log streaming API degraded | | SSL certificate | HTTPS endpoint | Certificate near expiry |

Determined AI turns a cluster of GPUs into a managed training platform — but that platform needs its own monitoring layer. A silent master crash means queued experiments never start. An unmounted NFS checkpoint share means hours of GPU time produce no recoverable weights. With Vigilmon watching the master, the agents, the database, and the storage, you can trust that every training run your team submits actually completes and persists its results.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles