tutorial

Monitoring ZenML with Vigilmon: Pipeline Server Health, Dashboard Uptime, Artifact Store & Metadata Store Availability

How to monitor ZenML MLOps pipeline infrastructure with Vigilmon — pipeline server health, dashboard uptime, artifact store connectivity, and metadata store availability.

ZenML is the MLOps framework that standardizes machine learning pipelines across cloud environments, artifact stores, and experiment trackers. When ZenML's pipeline server goes down, pipeline runs fail to register, artifacts cannot be tracked, and the dashboard that teams use to inspect pipeline history becomes unavailable. When the metadata store is degraded, ZenML loses the ability to associate pipeline runs with their artifacts, making reproducibility impossible. When the artifact store is unreachable, model checkpoints, datasets, and evaluation results cannot be persisted or retrieved. Vigilmon gives you external visibility into ZenML's infrastructure availability: the pipeline server, dashboard, and the HTTP endpoints that report store connectivity health.

What You'll Build

  • An HTTP monitor on the ZenML dashboard to catch server-level failures
  • A health-check monitor on the ZenML REST API to detect pipeline server degradation
  • An SSL certificate monitor for your ZenML domain
  • Alerting that distinguishes server crashes from store connectivity failures

Prerequisites

  • A self-hosted ZenML server (deployed via Docker, Kubernetes, or cloud VM) with a network-reachable domain
  • The ZenML dashboard exposed via HTTPS (e.g., https://zenml.example.com)
  • A free account at vigilmon.online

Step 1: Understand ZenML's Service Architecture

ZenML's self-hosted deployment centers on the ZenML Server, which exposes both the REST API and the web dashboard from a single process:

| Component | Default port | Role | |---|---|---| | ZenML Server | 8080 (HTTP/HTTPS) | REST API for pipeline registration, artifact tracking, and the web dashboard | | Metadata store | Internal DB (MySQL/SQLite) | Stores pipeline run metadata, artifact metadata, and stack configurations | | Artifact store | Remote storage (S3, GCS, Azure Blob) | Stores pipeline artifacts: model checkpoints, datasets, evaluation outputs |

External monitoring focuses on the ZenML Server HTTP endpoints — these surface the health of the entire ZenML deployment, including connectivity to the backing metadata and artifact stores.

Step 2: Monitor the ZenML Dashboard with HTTP Check

The ZenML dashboard is the primary interface for inspecting pipeline runs, artifact lineage, and stack configurations. It is served directly by the ZenML server process:

curl https://zenml.example.com
# Returns HTML containing the ZenML dashboard application
  1. Log in to Vigilmon → Add Monitor → HTTP.
  2. URL: https://zenml.example.com.
  3. Check interval: 60 seconds.
  4. Response timeout: 15 seconds.
  5. Expected status: 200.
  6. Keyword: ZenML (appears in the dashboard page title and content).
  7. Click Save.

This monitor catches:

  • ZenML server process crashes or OOM kills
  • Kubernetes pod failures for the zenml-server deployment
  • Reverse proxy or ingress misconfigurations blocking dashboard access
  • Failed ZenML upgrades that leave the server in a broken state

Alert sensitivity: Set to trigger after 1 consecutive failure. When the dashboard is unavailable, teams lose visibility into pipeline run history, artifact lineage, and active stack configurations.

Step 3: Monitor the ZenML REST API Health Endpoint

ZenML exposes a health endpoint that reports the server's status, including database connectivity:

curl https://zenml.example.com/health
# Returns: {"status": "ok"} when the server and database are healthy
  1. Add Monitor → HTTP.
  2. URL: https://zenml.example.com/health.
  3. Check interval: 60 seconds.
  4. Response timeout: 10 seconds.
  5. Expected status: 200.
  6. Keyword: ok (present in the health response when the server and metadata store are reachable).
  7. Label: ZenML server health.
  8. Click Save.

Why monitor /health separately? The dashboard may appear to load (returning cached HTML) even when the underlying REST API is degraded. A failing /health endpoint means the ZenML server cannot reach its metadata store — pipeline runs will fail to register even if the dashboard HTML loads. Catching this early prevents silent pipeline failures.

Step 4: Monitor the ZenML API Version Endpoint

ZenML exposes /api/v1/version which returns server version metadata and confirms that the REST API routing layer is functioning:

curl https://zenml.example.com/api/v1/version
# Returns: {"version": "0.x.x", ...}
  1. Add Monitor → HTTP.
  2. URL: https://zenml.example.com/api/v1/version.
  3. Check interval: 2 minutes.
  4. Response timeout: 10 seconds.
  5. Expected status: 200.
  6. Keyword: version (present in the version response JSON).
  7. Label: ZenML API version check.
  8. Click Save.

This catches API routing failures — situations where the server process is alive (so the health endpoint might respond) but the API layer has crashed or is misconfigured after an upgrade.

Step 5: Monitor SSL Certificates

ZenML's web dashboard and REST API are accessed via HTTPS by both human users and automated pipeline clients (the ZenML Python SDK). An expired or invalid certificate causes SDK connections to fail, blocking pipeline registration and artifact tracking:

openssl s_client -connect zenml.example.com:443 2>/dev/null | openssl x509 -noout -dates
  1. Add Monitor → SSL Certificate.
  2. Domain: zenml.example.com.
  3. Alert when expiry is within: 30 days.
  4. Alert again: 14 days, 7 days, 3 days, 1 day.
  5. Click Save.

SDK connectivity: The ZenML Python SDK connects to the server URL configured in ~/.config/zenml/config.yaml. When the certificate expires, zenml pipeline run commands and automated CI/CD pipeline triggers will fail with TLS verification errors. Certificate monitoring prevents these silent failures.

Step 6: Configure Alerting

In Vigilmon under Settings → Notifications, configure your alert channels:

| Monitor | Trigger | Action | |---|---|---| | Dashboard | Non-200 or ZenML missing | Check zenml-server pod; inspect ingress configuration; review container logs | | Health endpoint | Non-200 or ok missing | Metadata store unreachable; check database connectivity from the ZenML pod | | API version | Non-200 or version missing | REST API layer failed; check ZenML server logs for startup errors | | SSL certificate | < 30 days to expiry | Renew certificate; verify SDK connections succeed after renewal |

Alert after: 1 consecutive failure for the dashboard and health monitors. 2 consecutive failures for the API version monitor to avoid noise from transient load spikes during pipeline runs.

Common ZenML Failure Modes and What Vigilmon Catches

| Scenario | Vigilmon monitor | |---|---| | ZenML server pod OOM killed | Dashboard monitor fires; health endpoint fires; alert within 60 s | | Metadata database unreachable | Health monitor fires (ok missing); pipeline runs fail to register | | ZenML upgrade leaves server broken | Dashboard or API version monitor fires during botched deployment | | TLS certificate expired | SSL monitor alerts at 30-day threshold; SDK pipeline triggers fail | | Ingress controller misconfiguration | Dashboard monitor fires; API monitors fire; all HTTP checks fail | | Database connection pool exhausted | Health endpoint returns non-200; pipeline run registration starts timing out | | DNS misconfiguration | All monitors fire simultaneously | | ZenML server running out of disk | API may degrade; dashboard may return errors; health endpoint will catch DB write failures |

Monitoring Pipeline Run Success (Beyond Infrastructure)

Vigilmon monitors ZenML's infrastructure endpoints — the server, API, and SSL layer. Pipeline execution health is a separate concern that requires application-level observability:

  • ZenML's built-in pipeline run history: Use the dashboard's pipeline run view to inspect failed runs. Failed steps are shown with error tracebacks.
  • Artifact store monitoring: For S3/GCS/Azure Blob artifact stores, monitor the storage bucket availability via cloud provider health dashboards or cloud-native monitoring.
  • Alerting on pipeline failures: ZenML supports callbacks and alerters (Slack, email) that can be wired into pipeline steps to notify on step failures — configure these at the pipeline level for application-layer monitoring.

Vigilmon catches infrastructure-level failures. Pipeline step failures require ZenML's built-in alerter integrations.

ZenML is the glue layer for reproducible ML pipelines — when the server goes down, pipeline runs cannot be registered, artifacts lose their lineage tracking, and the audit trail that makes ML reproducible disappears. Vigilmon gives you external visibility into ZenML's availability that doesn't depend on your pipelines self-reporting: server uptime, REST API health, metadata store connectivity, and SSL certificate validity, so you know the moment the platform becomes unavailable and can restore pipeline execution before the next scheduled training run fails silently.

Start monitoring ZenML in under 5 minutes — register free at vigilmon.online.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles