tutorial

Monitoring Evidently with Vigilmon: ML Model Monitoring Service Uptime & Collector Health

How to monitor Evidently with Vigilmon — HTTP health checks on the Evidently service, collector endpoint availability, SSL certificate monitoring, and alerting for your ML model drift detection infrastructure.

Evidently is the open-source ML model monitoring library and platform that detects data drift, prediction drift, and data quality issues in production ML systems. Teams use Evidently to generate HTML reports for ad-hoc analysis and JSON metrics for live dashboards in Grafana, Prometheus, and other monitoring stacks. When Evidently's collector service goes down, your production ML models stop receiving drift signals — data distribution shifts go undetected, model degradation accumulates silently, and you lose the early warning system that triggers retraining. Vigilmon gives you external uptime monitoring for Evidently's service layer — collector health, the Evidently UI dashboard, SSL certificate validity, and alerting before your ML model monitoring infrastructure fails to detect the drift it was built to catch.

What You'll Build

  • An HTTP monitor on the Evidently service health endpoint to detect collector failures
  • A monitor on the Evidently UI dashboard for analyst-facing availability
  • SSL certificate monitoring for your Evidently deployment domain
  • Alerting rules that notify the team before drift detection goes offline

Prerequisites

  • A running Evidently service deployment — self-hosted Evidently Cloud (https://evidently.example.com) or the hosted Evidently Cloud (https://app.evidently.cloud)
  • The Evidently API or collector endpoint network-reachable from Vigilmon probes
  • A free account at vigilmon.online

Step 1: Understand Evidently's Service Architecture

Evidently has two deployment modes: a Python library used locally or in CI/CD, and the Evidently service that runs as a persistent server for live monitoring:

| Component | URL pattern | Role | |---|---|---| | Evidently service / API | https://evidently.example.com/api | Receives metric snapshots from production ML pipelines | | Evidently dashboard | https://evidently.example.com | Browser UI for viewing drift reports and model health | | Prometheus metrics endpoint | :8080/metrics | Exposes Evidently metrics for Prometheus scraping | | JSON reports endpoint | /api/projects/{id}/snapshots | REST API for retrieving snapshot data |

For the Evidently Python library used without a service, there is no persistent server to monitor — monitoring applies when Evidently runs as a deployed service with a persistent API.

Step 2: Monitor the Evidently Service Health

The Evidently service exposes a health endpoint that confirms the API layer is running and can accept metric snapshots from your production ML pipelines:

curl https://evidently.example.com/api/health
# Returns {"status":"ok"} on a healthy Evidently service
# For Evidently Cloud: curl https://app.evidently.cloud/api/health
  1. Log in to Vigilmon → Add Monitor → HTTP.
  2. URL: https://evidently.example.com/api/health (self-hosted) or https://app.evidently.cloud/api/health (Cloud).
  3. Check interval: 60 seconds.
  4. Response timeout: 15 seconds.
  5. Expected status: 200.
  6. Keyword: ok (present in the health response body).
  7. Label: Evidently service health.
  8. Click Save.

This monitor catches:

  • Evidently service container crashes
  • Database or storage backend failures preventing snapshot ingestion
  • Kubernetes pod failures on the Evidently deployment
  • Out-of-memory kills when processing large dataset snapshots

Alert sensitivity: Set to trigger after 1 consecutive failure. When the Evidently service is down, production ML pipelines cannot push drift metrics, and model degradation may go undetected until the next manual run.

Step 3: Monitor the Evidently UI Dashboard

The Evidently UI is where ML engineers and data scientists review drift reports, compare model performance across time windows, and access pre-built test suites. When the UI is unavailable, teams lose visibility into model health:

curl -I https://evidently.example.com
# Returns 200 with HTML content
  1. Add Monitor → HTTP.
  2. URL: https://evidently.example.com (self-hosted) or https://app.evidently.cloud (Cloud).
  3. Check interval: 60 seconds.
  4. Response timeout: 15 seconds.
  5. Expected status: 200.
  6. Keyword: Evidently (appears in the dashboard page title).
  7. Label: Evidently UI dashboard.
  8. Click Save.

Self-hosted note: On self-hosted deployments, the UI and API typically run as the same service. A dashboard failure often indicates the API layer is also degraded — monitor both to distinguish a frontend rendering issue from a full service outage.

Step 4: Monitor the Evidently API Snapshot Endpoint

The /api/projects endpoint is the core REST API for managing Evidently projects and retrieving snapshots. A reachability check on this endpoint confirms the API layer is functional beyond just the health route:

curl -I https://evidently.example.com/api/projects
# Returns 200 (authenticated) or 401/403 (authentication required — endpoint is alive)
  1. Add Monitor → HTTP.
  2. URL: https://evidently.example.com/api/projects.
  3. Check interval: 2 minutes.
  4. Response timeout: 10 seconds.
  5. Expected status: 200 or 401 (401 confirms the API is alive and enforcing authentication).
  6. Label: Evidently projects API.
  7. Click Save.

Step 5: Monitor the Prometheus Metrics Endpoint

Evidently can expose a Prometheus-compatible /metrics endpoint that your Prometheus instance scrapes to build Grafana dashboards for drift monitoring. When this endpoint is down, all Prometheus-based drift dashboards and alerts go stale:

curl http://evidently.example.com:8080/metrics
# Returns Prometheus text format with evidently_* metric names
  1. Add Monitor → HTTP.
  2. URL: http://evidently.example.com:8080/metrics (adjust port to your Evidently Prometheus export configuration).
  3. Check interval: 2 minutes.
  4. Response timeout: 10 seconds.
  5. Expected status: 200.
  6. Keyword: evidently (present in Evidently's Prometheus metric names).
  7. Label: Evidently Prometheus metrics.
  8. Click Save.

Port note: The Prometheus metrics endpoint port varies by deployment configuration. Check your Evidently service configuration (EVIDENTLY_METRICS_PORT or equivalent) to confirm the correct port before setting up this monitor.

Step 6: Monitor SSL Certificates

Evidently service endpoints use TLS to protect metric data in transit. An expired certificate blocks production ML pipelines from pushing snapshots and prevents browser access to drift reports:

openssl s_client -connect evidently.example.com:443 2>/dev/null | openssl x509 -noout -dates
  1. Add Monitor → SSL Certificate.
  2. Domain: evidently.example.com.
  3. Alert when expiry is within: 30 days.
  4. Alert again: 14 days, 7 days, 3 days, 1 day.
  5. Click Save.

Automated ML pipelines: If your production ML inference pipeline sends Evidently snapshots on every prediction batch (e.g., hourly or daily), an SSL expiry blocks every batch submission. Unlike human-facing services where users notice the error, automated pipelines may fail silently without a monitoring alert.

Step 7: Configure Alerting

In Vigilmon under Settings → Notifications, configure your alert channels:

| Monitor | Trigger | Action | |---|---|---| | Evidently service health | Non-200 or keyword missing | Check service container; inspect database connectivity; review deployment logs | | UI dashboard | Non-200 or keyword missing | Evidently web frontend degraded; check container health | | Projects API | Unexpected status | API layer degraded; snapshot submissions from ML pipelines may be failing | | Prometheus metrics | Non-200 or keyword missing | Grafana drift dashboards going stale; Prometheus scrape failing | | SSL certificate | < 30 days to expiry | Renew certificate; test pipeline snapshot submission after renewal |

Alert after: 1 consecutive failure for the service health endpoint. 2 consecutive failures for dashboard, API, and Prometheus monitors.

Common Evidently Failure Modes and What Vigilmon Catches

| Scenario | Vigilmon monitor | |---|---| | Evidently container crash after snapshot overload | Service health monitor fires; drift detection goes dark | | Database write failure during large dataset snapshot | Service returns 500; health monitor fires within 60 s | | Disk full on self-hosted deployment (snapshot storage) | Service degraded; health monitor fires | | SSL certificate expired | SSL monitor alerts at 30-day threshold; pipeline submissions fail with TLS errors | | Prometheus metrics endpoint unavailable | Metrics monitor fires; Grafana drift dashboards go stale | | Kubernetes pod eviction during batch snapshot submission | Service health monitor fires; snapshot lost | | Database migration failure after upgrade | Service returns 500; health monitor fires; snapshots not persisted | | Evidently UI frontend deploy failure | Dashboard monitor fires; API health may remain OK | | Network partition between ML inference and Evidently | Service health fires from external probe; pipelines accumulate unsubmitted snapshots | | DNS misconfiguration after infrastructure migration | All Evidently monitors fire simultaneously |

Monitoring Drift Detection Results vs. Service Health

Vigilmon monitors whether Evidently's service is reachable — it does not monitor whether drift is actually being detected in your ML models. For drift-level observability:

  • Evidently Alerts: Configure Evidently test suites with TestSuite and DataDriftTable checks that output pass/fail results your CI/CD pipeline can act on.
  • Prometheus alerting rules: Define ALERT rules in Prometheus on evidently_data_drift_detected or similar metrics to trigger PagerDuty/Slack notifications when drift crosses thresholds.
  • Scheduled drift runs: Run Evidently reports on a schedule (cron job, Airflow DAG, GitHub Actions workflow) so drift detection happens regularly even without a persistent service.
  • Retraining triggers: Integrate Evidently's drift results with your MLflow or Weights & Biases pipelines to automatically trigger model retraining when drift exceeds defined thresholds.

Vigilmon catches infrastructure-level failures — Evidently service unreachability, dashboard downtime, certificate expiry, Prometheus endpoint failures. Model drift detection results require Evidently's native test suite configuration and alerting integration.

Evidently is your early warning system for model degradation — it catches the data drift and prediction shifts that silently reduce your model's accuracy in production. But when Evidently's own service goes down, your ML pipeline loses its watchdog. Vigilmon monitors Evidently's health endpoint, dashboard, API, and Prometheus metrics on a 60-second cycle, so you know within a minute when your drift detection infrastructure goes offline and can restore monitoring before model degradation accumulates undetected.

Start monitoring Evidently in under 5 minutes — register free at vigilmon.online.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles