tutorial

Monitoring Weights & Biases (wandb) with Vigilmon: API Health, Dashboard Uptime & SSL Certificates

How to monitor Weights & Biases with Vigilmon — HTTP health checks on the wandb API, dashboard availability, SSL certificate monitoring, and alerting for your MLOps experiment tracking infrastructure.

Weights & Biases (wandb) is the MLOps platform for experiment tracking, training metrics visualization, model registry management, and hyperparameter sweep orchestration. Data science and ML engineering teams rely on wandb to log training runs, compare model versions, and coordinate collaborative experiments. When wandb's API endpoint goes down during a training run, your GPU instances silently lose their logging connection — runs complete but metrics are never recorded, sweeps stall, and you lose the experiment data you needed to reproduce results. When the wandb dashboard is unavailable, your team cannot review running experiments or access model registry history. Vigilmon gives you external uptime monitoring for wandb — API health, dashboard availability, SSL certificate validity, and alerting before a wandb outage silently discards your training metrics.

What You'll Build

  • An HTTP monitor on the wandb API endpoint to detect logging failures
  • A monitor on the wandb dashboard for team-facing availability
  • SSL certificate monitoring for your wandb instance domain
  • Alerting rules that notify the team before a wandb outage causes experiment data loss

Prerequisites

  • A wandb instance — self-hosted (https://wandb.example.com) or wandb Cloud (https://api.wandb.ai)
  • The wandb API endpoint network-reachable from Vigilmon probes
  • A free account at vigilmon.online

Step 1: Understand wandb's Architecture

Weights & Biases exposes several endpoints that your training code and team members interact with:

| Component | URL pattern | Role | |---|---|---| | API endpoint | https://api.wandb.ai or /api | Receives metric logs from training scripts via wandb SDK | | Dashboard / web app | https://wandb.ai or https://wandb.example.com | Browser UI for experiment comparison and model registry | | File storage | https://storage.wandb.ai or /files | Artifacts, model checkpoints, media files | | GraphQL API | /graphql | Underlying API layer for the dashboard |

For self-hosted deployments (wandb Server / W&B Local), all components run under a single domain. For wandb Cloud, the API and dashboard share wandb.ai infrastructure.

Step 2: Monitor the wandb API Health Endpoint

The wandb API is the critical ingestion path for all training metrics. A health check on the API confirms the server is alive and accepting SDK connections:

curl https://api.wandb.ai/healthz
# Returns {"status":"ok"} on a healthy instance
# For self-hosted: curl https://wandb.example.com/healthz
  1. Log in to Vigilmon → Add Monitor → HTTP.
  2. URL: https://api.wandb.ai/healthz (Cloud) or https://wandb.example.com/healthz (self-hosted).
  3. Check interval: 60 seconds.
  4. Response timeout: 15 seconds.
  5. Expected status: 200.
  6. Keyword: ok (present in the health response body).
  7. Label: wandb API health.
  8. Click Save.

This monitor catches:

  • wandb server process crashes
  • Database connection failures preventing metric ingestion
  • Container OOM kills on self-hosted wandb deployments
  • Load balancer failures that drop SDK connections during training runs

Alert sensitivity: Set to trigger after 1 consecutive failure. A single API outage during a long training run causes all subsequent metric logs to fail silently.

Step 3: Monitor the wandb Dashboard

The wandb web dashboard is where data scientists compare experiments, review sweep results, and access the model registry. When the dashboard is unavailable, teams cannot review active training runs or coordinate on experiment results:

curl -I https://wandb.ai
# Returns 200 with HTML content
# For self-hosted: curl -I https://wandb.example.com
  1. Add Monitor → HTTP.
  2. URL: https://wandb.ai (Cloud) or https://wandb.example.com (self-hosted).
  3. Check interval: 60 seconds.
  4. Response timeout: 15 seconds.
  5. Expected status: 200.
  6. Keyword: Weights & Biases (appears in the page title).
  7. Label: wandb dashboard.
  8. Click Save.

Self-hosted deployments: On W&B Local, the dashboard and API typically share the same container. A dashboard failure often co-occurs with API degradation — monitor both to distinguish a frontend-only issue from a full service outage.

Step 4: Monitor the wandb File Storage Endpoint

wandb stores model checkpoints, artifact files, and media (images, audio, video logged during training) in its file storage backend. When file storage is unavailable, wandb.log({"model": artifact}) calls fail, and artifact versioning in the model registry breaks:

curl -I https://storage.wandb.ai
# Returns 200 or 403 (storage endpoint alive but requires auth)
  1. Add Monitor → HTTP.
  2. URL: https://storage.wandb.ai (Cloud) or your self-hosted artifact storage endpoint.
  3. Check interval: 5 minutes.
  4. Response timeout: 15 seconds.
  5. Expected status: 200 or 403 (403 confirms the endpoint is reachable; authentication is required for actual artifact access).
  6. Label: wandb artifact storage.
  7. Click Save.

For self-hosted deployments using S3 or GCS as the artifact backend, monitor the underlying storage endpoint as well (see the DVC monitoring tutorial for S3/GCS endpoint monitoring patterns).

Step 5: Monitor the GraphQL API

The wandb dashboard is powered by a GraphQL API layer. If the GraphQL endpoint fails, the dashboard renders empty or shows errors even when the metric ingestion API is healthy:

curl -I https://api.wandb.ai/graphql
# Returns 200 or 400 (GET to a POST-only GraphQL endpoint)
  1. Add Monitor → HTTP.
  2. URL: https://api.wandb.ai/graphql (Cloud) or https://wandb.example.com/graphql (self-hosted).
  3. Check interval: 2 minutes.
  4. Response timeout: 10 seconds.
  5. Expected status: 200 or 405.
  6. Label: wandb GraphQL API.
  7. Click Save.

Step 6: Monitor SSL Certificates

wandb SDK clients validate TLS certificates when connecting to the API. An expired certificate causes wandb.init() to raise an SSL error in your training scripts, and breaks browser access to the dashboard. For long-running training jobs (multi-day fine-tuning runs), certificate expiry mid-run is a real failure mode:

openssl s_client -connect wandb.example.com:443 2>/dev/null | openssl x509 -noout -dates
  1. Add Monitor → SSL Certificate.
  2. Domain: wandb.example.com (self-hosted) or wandb.ai (Cloud — Vigilmon monitors the certificate regardless of who manages it).
  3. Alert when expiry is within: 30 days.
  4. Alert again: 14 days, 7 days, 3 days, 1 day.
  5. Click Save.

Multi-day training runs: If a GPU cluster is running a 72-hour fine-tuning job and the wandb SSL certificate expires mid-run, the SDK will fail silently or crash the logging thread. A 30-day alert window gives you time to renew the certificate long before any active runs are at risk.

Step 7: Configure Alerting

In Vigilmon under Settings → Notifications, configure your alert channels:

| Monitor | Trigger | Action | |---|---|---| | wandb API health | Non-200 or keyword missing | Check wandb server process; inspect database health; review container logs | | Dashboard | Non-200 or keyword missing | wandb web frontend degraded; check web container health | | Artifact storage | Unexpected status | Artifact saves failing; model checkpoints at risk | | GraphQL API | Unexpected status | Dashboard rendering broken; check API container health | | SSL certificate | < 30 days to expiry | Renew certificate; test SDK connection and browser access |

Alert after: 1 consecutive failure for the API health endpoint. 2 consecutive failures for dashboard and GraphQL monitors.

Common wandb Failure Modes and What Vigilmon Catches

| Scenario | Vigilmon monitor | |---|---| | wandb server OOM killed during large experiment | API health monitor fires; training logs silently fail | | Database connection pool exhausted | API returns 500; health monitor fires within 60 s | | Artifact storage bucket access revoked | Storage monitor fires; model checkpoint saves fail | | SSL certificate expired on self-hosted instance | SSL monitor alerts at 30-day threshold; SDK raises SSL errors | | Kubernetes pod eviction during training | API health monitor fires; active runs lose logging | | Container registry pull failure blocks upgrade | Dashboard and API monitors fire simultaneously post-deploy | | Hyperparameter sweep coordinator crash | API health monitor fires; sweep agents stall waiting for next parameters | | Dashboard frontend deploy failure | Dashboard monitor fires; API health may still be OK | | Network partition isolates wandb from GPU cluster | API health monitor fires from external probe; team alerted before jobs complete | | DNS misconfiguration after server migration | All wandb monitors fire simultaneously |

Monitoring Training Run Health vs. wandb Service Health

Vigilmon monitors whether wandb's endpoints are reachable — it does not monitor the health of individual training runs. For run-level observability:

  • wandb Alerts: wandb has a built-in alerting feature (wandb.alert()) for run-specific conditions like NaN losses or accuracy thresholds — use these for training-logic alerts.
  • Sweep status: Monitor active sweep agent counts and pending run counts via the wandb API to detect stalled sweep orchestration.
  • GPU utilization: Use cloud provider metrics or nvidia-smi exporter for Prometheus to monitor GPU health separately from wandb availability.
  • Run recovery: Configure wandb.init(resume="allow") in training scripts so runs can automatically resume after a brief wandb API outage without losing progress.

Vigilmon catches infrastructure-level failures — API unreachability, dashboard downtime, certificate expiry. Per-run training health requires wandb's native alerting and SDK configuration.

Weights & Biases is where your team tracks every experiment, but when the wandb API goes down during a GPU training run, metrics are silently dropped — you get a completed job with no data and no way to reproduce the results. Vigilmon monitors wandb's API health endpoint, dashboard, artifact storage, and SSL certificate on a 60-second cycle, so you know within a minute when your experiment tracking infrastructure is unavailable and can act before active training runs complete without their metrics.

Start monitoring Weights & Biases in under 5 minutes — register free at vigilmon.online.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles