Monitoring ClearML with Vigilmon: Open-Source MLOps Platform Health Checks

ClearML is the open-source end-to-end MLOps platform providing experiment tracking, dataset versioning, pipeline orchestration, model serving, and remote execution on on-prem or cloud compute — designed as a self-hostable alternative to MLflow, Kubeflow, and DVC combined. When ClearML's API server goes down, your training scripts cannot log experiments, pipeline tasks queue indefinitely, and dataset version metadata becomes inaccessible. When the ClearML file server becomes unreachable, artifact uploads and model checkpoint storage fail silently across your entire ML infrastructure. Vigilmon gives you external uptime monitoring for ClearML — API server availability, web UI health, file server reachability, SSL certificate validity, and alerting before your MLOps backbone fails during critical training runs.

What You'll Build

• An HTTP monitor on ClearML's API server to detect experiment logging failures
• A monitor on the ClearML web UI to catch dashboard outages
• A monitor on the ClearML file server for artifact storage availability
• An SSL certificate monitor for your ClearML instance domain
• Alerting rules that notify your team when MLOps infrastructure fails

Prerequisites

• A running ClearML server — self-hosted (https://clearml.example.com) or ClearML Cloud (https://app.clear.ml)
• The ClearML API server, web UI, and file server network-reachable from Vigilmon probes
• A free account at vigilmon.online

Step 1: Understand ClearML's Three-Service Architecture

ClearML's self-hosted server runs three distinct services, each with a different role and network port:

| Service | Default URL | Role | |---|---|---| | API server | https://api.clearml.example.com or :8008 | REST API for task creation, experiment logging, dataset versioning | | Web server | https://app.clearml.example.com or :8080 | Browser-accessible experiment UI and pipeline dashboard | | File server | https://files.clearml.example.com or :8081 | Artifact, model checkpoint, and dataset storage |

For self-hosted ClearML, each service can run on its own subdomain or port on the same host. For ClearML Cloud, these are managed by Allegro AI. Monitor all three independently — a file server failure does not fail the API server, so a single monitor misses half the failure modes.

Step 2: Monitor the ClearML API Server

The API server is the most critical component — it handles all task creation, experiment logging, dataset versioning, and pipeline orchestration requests from the ClearML Python SDK. A liveness check on the API server confirms the backend is accepting connections:

curl -I https://api.clearml.example.com/v2.23/info
# Returns 200 with JSON server info (no authentication required for /info)

1. Log in to Vigilmon → Add Monitor → HTTP.
2. URL: https://api.clearml.example.com/v2.23/info (adjust API version to match your ClearML installation; the /info endpoint requires no authentication).
3. Check interval: 60 seconds.
4. Response timeout: 15 seconds.
5. Expected status: 200.
6. Keyword: version (appears in the JSON info response).
7. Label: ClearML API server.
8. Click Save.

This monitor catches:

• ClearML API container crashes or OOM kills
• MongoDB connection failures preventing task persistence
• Elasticsearch degradation causing task query failures
• Redis failures disrupting queue management

Alert sensitivity: Set to trigger after 1 consecutive failure. A single API outage prevents all experiment logging, pipeline task creation, and dataset version registration across your ML infrastructure.

Step 3: Monitor the ClearML Web UI

The ClearML web frontend serves the experiment comparison UI, pipeline DAG visualization, and model management dashboard. An HTTP check on the web server confirms the UI layer is serving requests:

curl -I https://app.clearml.example.com
# Returns 200 with text/html content

1. Add Monitor → HTTP.
2. URL: https://app.clearml.example.com (or your ClearML web server address).
3. Check interval: 60 seconds.
4. Response timeout: 15 seconds.
5. Expected status: 200.
6. Keyword: ClearML (appears in the page title).
7. Label: ClearML web UI.
8. Click Save.

Port-based deployments: If your ClearML web server runs on port 8080 rather than a subdomain, use http://clearml.example.com:8080 as the URL. Configure your Vigilmon monitors to match your actual deployment topology.

Step 4: Monitor the ClearML File Server

The file server stores model artifacts, dataset files, debug images, and other large binary objects logged by ClearML tasks. File server failures do not fail the API server — tasks appear to run normally but artifact uploads silently fail, leading to incomplete experiment records:

curl -I https://files.clearml.example.com
# Returns 200; the file server serves a minimal health page

1. Add Monitor → HTTP.
2. URL: https://files.clearml.example.com (or http://clearml.example.com:8081 for port-based deployments).
3. Check interval: 2 minutes.
4. Response timeout: 15 seconds.
5. Expected status: 200.
6. Label: ClearML file server.
7. Click Save.

Why file server monitoring matters: When the file server is down, task.upload_artifact() calls in training scripts raise errors or silently fail depending on ClearML SDK configuration. Model checkpoints, debug samples, and dataset snapshots logged during the outage window are permanently lost — they are not buffered and retried by the SDK.

Step 5: Monitor SSL Certificates

ClearML's TLS certificates protect the API server, web UI, and file server. An expired API certificate causes Python SDKs to raise SSL errors across all training scripts — every ClearML task fails to connect and experiment tracking stops entirely:

openssl s_client -connect api.clearml.example.com:443 2>/dev/null | openssl x509 -noout -dates

Monitor all three ClearML domains separately if they use different certificates:

1. Add Monitor → SSL Certificate for api.clearml.example.com — alert 30 days before expiry.
2. Add Monitor → SSL Certificate for app.clearml.example.com — alert 30 days before expiry.
3. Add Monitor → SSL Certificate for files.clearml.example.com — alert 30 days before expiry.

Step 6: Configure Alerting

In Vigilmon under Settings → Notifications, configure your alert channels:

| Monitor | Trigger | Action | |---|---|---| | API server | Non-200 response or timeout | Check MongoDB and Elasticsearch health; review API container logs; inspect Redis queue | | Web UI | Non-200 or keyword missing | ClearML frontend degraded; check web container; API may still be healthy | | File server | Non-200 response or timeout | Artifact uploads failing; check storage backend (S3/NFS/local disk); review file server logs | | SSL certificate | < 30 days to expiry | Renew certificate; verify Python SDKs can reconnect; check all three ClearML services |

Alert after: 1 consecutive failure for the API server and file server. 2 consecutive failures for the web UI.

Common ClearML Failure Modes and What Vigilmon Catches

| Scenario | Vigilmon monitor | |---|---| | API container OOM killed | API monitor fires; all task creation and logging stops | | MongoDB disk full | API returns 500 on task write; API monitor fires | | Elasticsearch index corruption | API returns errors for task queries; API monitor fires | | File server disk full | File server returns 500 on artifact uploads; file server monitor fires | | SSL certificate expired on API | SSL monitor alerts at 30-day threshold; SDK raises SSL errors | | File server network partition | File server monitor fires; artifacts lost silently without API alert | | Kubernetes pod restart mid-pipeline | API monitor fires; in-progress pipeline tasks may fail or stall | | Redis failure disrupts task queue | API behavior degrades; queue-dependent pipeline steps stall | | DNS cutover failure after migration | All monitors fire; training clusters cannot reach ClearML | | NFS mount failure for artifact storage | File server returns 503; file server monitor fires |

Monitoring Task Execution vs. Service Availability

Vigilmon monitors whether ClearML's services are reachable — it does not verify that tasks are executing successfully within your ClearML agents. For deeper operational monitoring:

• Agent health: ClearML agents poll the API server for queued tasks. If an agent has been idle longer than expected, check API server health and the agent's own process status on the worker machine.
• Queue depth: ClearML's web UI shows pending task counts per queue. Unusual queue depth growth indicates that agents are offline even if the API server itself is healthy.
• Artifact completeness: After a file server recovery, verify whether in-progress tasks that failed to upload artifacts have been re-run or marked incomplete.

Vigilmon catches infrastructure-level failures — endpoint unreachability, certificate expiry, service crashes. End-to-end task execution monitoring requires ClearML's internal agent and queue metrics alongside external uptime monitoring.

ClearML orchestrates your entire ML lifecycle — from experiment tracking to pipeline execution to model versioning — but your MLOps platform itself needs monitoring. An API server outage silently stops all experiment logging and pipeline task scheduling. A file server failure causes artifact uploads to fail across every running task. Vigilmon checks ClearML's API server, web UI, file server, and SSL certificates on a 60-second cycle, so you know within a minute when your MLOps backbone goes down.

Start monitoring ClearML in under 5 minutes — register free at vigilmon.online.