tutorial

Monitoring Comet ML with Vigilmon: Experiment Dashboard, Model Registry API, SDK Connectivity & SSL

How to monitor Comet ML with Vigilmon — experiment dashboard uptime, model registry API availability, SDK connectivity, and SSL certificate monitoring for ML teams.

Comet ML is the experiment tracking and model management platform that ML teams rely on to log training runs, compare hyperparameters, version datasets, and manage the full model lifecycle. When Comet's experiment dashboard goes down, engineers lose visibility into live training runs and reproducibility metadata. When the REST API is unavailable, automated pipelines that push experiment metrics or pull model artifacts break silently. When the model registry API fails, CI/CD workflows that promote models to production stall. Vigilmon gives you external visibility into Comet ML's availability: the dashboard, REST API, model registry, and SSL certificate health.

What You'll Build

  • An HTTP monitor on the Comet ML dashboard to detect platform-level outages
  • An API availability check on the Comet REST API endpoint
  • A model registry API monitor to confirm artifact storage and retrieval
  • SSL certificate monitoring for your Comet workspace domain
  • An alerting setup that distinguishes dashboard failures from API-level degradations

Prerequisites

  • A Comet ML workspace (cloud-hosted at comet.com or self-hosted Comet Enterprise)
  • Your Comet workspace URL (e.g., https://www.comet.com/<workspace>)
  • The Comet REST API base URL (https://www.comet.com/api/rest/v2)
  • A free account at vigilmon.online

Step 1: Understand Comet ML's External-Facing Endpoints

Comet ML exposes several endpoints relevant to external health monitoring:

| Endpoint | Role | |---|---| | https://www.comet.com | Main dashboard — experiment list, project views, metric charts | | https://www.comet.com/api/rest/v2 | REST API — used by SDKs, CI pipelines, and automation scripts | | https://www.comet.com/api/rest/v2/projects | Project list API — confirms API connectivity and auth layer health | | Model Registry | Artifact storage and promotion workflows |

For self-hosted Comet Enterprise, replace www.comet.com with your internal domain (e.g., https://comet.internal.example.com).

Step 2: Create a Vigilmon HTTP Monitor for the Dashboard

The Comet ML dashboard is the primary interface for reviewing experiment results, comparing runs, and tracking model performance over time. Downtime means engineers can't inspect live or historical training runs:

curl -I https://www.comet.com
# Returns HTTP 200 with Comet dashboard HTML
  1. Log in to Vigilmon → Add Monitor → HTTP.
  2. URL: https://www.comet.com (or your self-hosted Comet URL).
  3. Check interval: 60 seconds.
  4. Response timeout: 15 seconds.
  5. Expected status: 200.
  6. Keyword: Comet (appears in the page title and dashboard content).
  7. Click Save.

This monitor catches:

  • Comet ML cloud platform outages
  • Frontend container or CDN failures for self-hosted deployments
  • Deployment failures after Comet Enterprise upgrades
  • Network-level routing failures blocking dashboard access

Alert sensitivity: Set to trigger after 1 consecutive failure. When the dashboard is down, every ML engineer and data scientist on the team loses visibility into ongoing experiments.

Step 3: Monitor the Comet REST API

Comet's Python SDK, the comet_ml library, communicates with the REST API to log experiment parameters, metrics, system info, and artifacts in real time. Pipeline scripts that query experiment results or download models also hit this API. A degraded API means training runs log nothing and automation pipelines fail silently:

# Check API reachability (no auth required for this probe)
curl -I https://www.comet.com/api/rest/v2
# Returns HTTP 401 (unauthorized) — which confirms the API is reachable
  1. Add Monitor → HTTP.
  2. URL: https://www.comet.com/api/rest/v2.
  3. Check interval: 60 seconds.
  4. Response timeout: 10 seconds.
  5. Expected status: 401 (the API returns 401 for unauthenticated probes — this confirms the API layer is alive).
  6. Label: Comet REST API.
  7. Click Save.

Why check for 401? An unauthenticated request to a protected API endpoint returning 401 Unauthorized is a healthy signal — it means the API gateway, auth layer, and routing are all working correctly. A 502 or timeout means the API backend is down, which is what you need to catch.

Step 4: Monitor the Model Registry API

The Comet model registry stores versioned model artifacts and manages the promotion workflow from staging to production. CI/CD pipelines that auto-promote models query the registry API to fetch artifact metadata and download model files:

# Test model registry API accessibility
curl -I https://www.comet.com/api/rest/v2/registry/model
# Returns HTTP 401 — confirms registry API is reachable
  1. Add Monitor → HTTP.
  2. URL: https://www.comet.com/api/rest/v2/registry/model.
  3. Check interval: 2 minutes.
  4. Response timeout: 10 seconds.
  5. Expected status: 401.
  6. Label: Comet Model Registry API.
  7. Click Save.

For self-hosted Comet Enterprise, the model registry may be backed by S3, GCS, or Azure Blob Storage. If your deployment exposes a dedicated artifact storage endpoint, add a separate HTTP monitor for it.

Step 5: Monitor SSL Certificates

Comet ML SDKs and all browser connections rely on valid TLS certificates. An expired certificate causes the Python SDK to fail with SSL: CERTIFICATE_VERIFY_FAILED errors during experiment logging, breaking every active training run:

openssl s_client -connect www.comet.com:443 2>/dev/null | openssl x509 -noout -dates
  1. Add Monitor → SSL Certificate.
  2. Domain: www.comet.com (or your self-hosted Comet domain).
  3. Alert when expiry is within: 30 days.
  4. Alert again: 14 days, 7 days, 3 days, 1 day.
  5. Click Save.

Self-hosted Comet Enterprise: If you're managing your own TLS certificates (Let's Encrypt, internal CA), set the alert threshold to 30 days. Automated renewal tools like certbot can fail silently; Vigilmon's SSL monitor catches expiry before it disrupts SDK connections.

Step 6: Configure Alerting

In Vigilmon under Settings → Notifications, configure your alert channels:

| Monitor | Trigger | Action | |---|---|---| | Dashboard | Non-200 or keyword missing | Check Comet status page; alert ML engineering team; notify active experiment owners | | REST API | Non-401/non-200 | SDK logging broken; pipeline queries failing; check Comet API status | | Model Registry API | Non-401/non-200 | Model promotion pipelines stalled; check registry service health | | SSL certificate | < 30 days to expiry | Renew certificate; test SDK connectivity and browser access |

Alert after: 1 consecutive failure for the dashboard and REST API monitors. 2 consecutive failures for the model registry monitor.

Common Comet ML Failure Modes and What Vigilmon Catches

| Scenario | Vigilmon monitor | |---|---| | Comet cloud platform outage | Dashboard monitor fires; REST API monitor fires | | API gateway overloaded (5xx errors) | REST API monitor fires; SDK logging silently drops metrics | | Model registry storage backend failure | Registry API monitor fires; model promotion pipelines stall | | Self-hosted Comet container crash | Dashboard and API monitors fire simultaneously | | SSL certificate expiry | SSL monitor alerts at 30-day threshold; SDK connections fail | | CDN failure (assets returning 502) | Dashboard monitor catches via keyword check | | Auth service degradation | API monitors catch 5xx instead of expected 401 | | Database backend failure | Dashboard loads but API returns errors; REST API monitor catches | | Network partition to artifact storage | Registry API degrades; download pipelines time out |

Self-Hosted Comet Enterprise Considerations

For teams running Comet Enterprise on-premises or in a private cloud:

  • Add an internal health endpoint: Comet Enterprise exposes /health or /status endpoints. Add an HTTP monitor for https://comet.internal.example.com/health with keyword healthy or ok.
  • Monitor component services separately: If your deployment splits the API server, web frontend, and artifact storage, add individual monitors for each component.
  • PostgreSQL/MySQL backend: Comet Enterprise uses a relational database for experiment metadata. Database connectivity issues cause API errors without visible pod crashes — monitor the API endpoint to catch these indirectly.

Comet ML is the system of record for your model development history — experiment parameters, metric curves, dataset versions, and the artifacts that became production models. When Comet goes down or degrades, training runs log nothing, reproducibility is broken, and model promotion pipelines stall. Vigilmon gives you external visibility into Comet's availability from outside your ML infrastructure: dashboard uptime, REST API health, model registry access, and SSL certificate expiry, so you know the moment Comet becomes unreachable and can restore experiment tracking before the next training run completes silently unlogged.

Start monitoring Comet ML in under 5 minutes — register free at vigilmon.online.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles