tutorial

Monitoring MLflow: Tracking Server, Artifact Store, and Model Registry Uptime

"A practical guide to monitoring MLflow in production — tracking server availability, artifact store health, model registry API uptime, and setting up Vigilmon alerts on MLflow degradation."

Monitoring MLflow in Production

MLflow is the de-facto standard for ML experiment tracking, but "MLflow is down" in a production MLOps environment means your model training pipelines are silently failing, artifact uploads are being dropped, and your model registry is inaccessible to serving infrastructure. This guide covers how to monitor every critical MLflow component and wire up external uptime checks with Vigilmon.

MLflow's Critical Components

MLflow in production typically has three separate services:

  1. Tracking server — the HTTP API that your training jobs call to log metrics, parameters, and tags
  2. Artifact store — S3, GCS, Azure Blob, or a local filesystem path where models and files are stored
  3. Model registry — a subset of the tracking server API for managing model versions and stages

Each can fail independently. Training jobs can appear to run while silently swallowing connection errors. Your model serving infra can go stale because the registry is unreachable. The artifact store can become a bottleneck when large model files are being uploaded.

1. Tracking Server Health

The MLflow tracking server exposes a health endpoint:

curl -s http://localhost:5000/health
# "OK"

This confirms the Flask/Gunicorn process is alive and the backend database (SQLite, Postgres, or MySQL) is reachable.

For a more detailed check, hit the experiments list API:

curl -s http://localhost:5000/api/2.0/mlflow/experiments/search \
  -H "Content-Type: application/json" \
  -d '{"max_results": 1}'

A 200 response with a JSON body means the tracking server is fully functional — Flask is running, the database is reachable, and the API layer is serving requests.

Database backend check

If you're using Postgres as the MLflow backend store:

# Quick connectivity check
psql -h localhost -U mlflow -d mlflow -c "SELECT COUNT(*) FROM experiments;"

Slow results here (>500ms) indicate database saturation from concurrent training jobs.

2. Artifact Store Health

The artifact store is often the silent point of failure. Training jobs upload checkpoints and model files during training — if the store is down or throttled, jobs fail at save time, not at the start.

S3 artifact store

# Check bucket accessibility
aws s3 ls s3://your-mlflow-artifacts/ --max-items 1

# Check upload latency with a small test object
echo "health-check" | aws s3 cp - s3://your-mlflow-artifacts/.health
aws s3 rm s3://your-mlflow-artifacts/.health

Local filesystem artifact store

#!/bin/bash
ARTIFACT_ROOT="/opt/mlflow/artifacts"
# Check writable and accessible
if [ -d "$ARTIFACT_ROOT" ] && touch "$ARTIFACT_ROOT/.health_check" 2>/dev/null; then
  rm -f "$ARTIFACT_ROOT/.health_check"
  echo "ok"
  exit 0
else
  echo "artifact store not writable"
  exit 1
fi

Wrap this in a simple HTTP endpoint (a 5-line FastAPI app works well) and expose it at /artifact-health for external monitoring.

3. Model Registry API

The model registry is accessed via MLflow's REST API. Check it directly:

# List registered models
curl -s "http://localhost:5000/api/2.0/mlflow/registered-models/list?max_results=1"

A 200 with a valid JSON body confirms the model registry is operational. This endpoint exercises the same database backend as the tracking server but tests a different code path that's essential for model serving infrastructure.

Python health check

For Python-based health probes that you can run as a sidecar or cron job:

import mlflow
import sys

def check_mlflow_health(tracking_uri: str) -> bool:
    try:
        client = mlflow.tracking.MlflowClient(tracking_uri=tracking_uri)
        # Try to list experiments — exercises DB + API
        experiments = client.search_experiments(max_results=1)
        return True
    except Exception as e:
        print(f"MLflow health check failed: {e}", file=sys.stderr)
        return False

if __name__ == "__main__":
    healthy = check_mlflow_health("http://localhost:5000")
    sys.exit(0 if healthy else 1)

Run this from a cron job every minute, or wire it into a /health HTTP endpoint in your MLflow deployment.

4. Metrics to Track

Beyond simple up/down status, instrument these metrics for capacity planning and anomaly detection:

# Add to your training pipeline to track experiment creation latency
import time
import mlflow

start = time.time()
with mlflow.start_run():
    mlflow.log_metric("latency_probe", 1.0)
latency_ms = (time.time() - start) * 1000

if latency_ms > 5000:  # 5 second threshold
    # Alert or log warning
    print(f"WARNING: MLflow run creation took {latency_ms:.0f}ms")

Key metrics to monitor:

| Metric | Normal | Alert threshold | |---|---|---| | /health response time | <100ms | >1000ms | | Run creation latency | <500ms | >5000ms | | Artifact upload throughput | Baseline × 0.5 | Baseline × 0.1 | | Model registry list latency | <200ms | >2000ms | | Active experiment count | Stable | Sudden drop |

5. Log Analysis for Silent Failures

MLflow's default logging doesn't surface artifact upload failures well. Enable structured logging:

# mlflow_config.py
import logging
logging.basicConfig(
    level=logging.INFO,
    format='{"time": "%(asctime)s", "level": "%(levelname)s", "message": "%(message)s"}'
)

Then watch for patterns that indicate silent failures:

# Watch for artifact upload failures in real time
tail -f /var/log/mlflow/mlflow.log | grep -E "(ERROR|artifact|upload|store)"

# Count errors per minute
grep "ERROR" /var/log/mlflow/mlflow.log | grep "$(date '+%Y-%m-%d %H:%M')" | wc -l

6. Integrating Vigilmon for External Uptime Checks

Internal checks miss network-level failures, DNS issues, and load balancer problems. Vigilmon provides external uptime monitoring that checks your MLflow endpoints from outside your infrastructure.

Step 1: Set up the tracking server monitor

In Vigilmon, create an HTTP monitor:

  • URL: https://mlflow.yourdomain.com/health
  • Method: GET
  • Expected status: 200
  • Expected body contains: OK
  • Check interval: 60 seconds
  • Locations: Select 3+ geographic regions for global coverage

Step 2: Monitor the model registry API

Create a second monitor targeting the registry endpoint:

  • URL: https://mlflow.yourdomain.com/api/2.0/mlflow/registered-models/list?max_results=1
  • Method: GET
  • Expected status: 200
  • Check interval: 2 minutes

Step 3: Artifact store proxy health check

Expose a proxy health check endpoint in your MLflow deployment (a custom Flask route or sidecar):

from flask import Flask, jsonify
import boto3

app = Flask(__name__)

@app.route("/artifact-health")
def artifact_health():
    try:
        s3 = boto3.client("s3")
        s3.head_bucket(Bucket="your-mlflow-artifacts")
        return jsonify({"status": "ok"}), 200
    except Exception as e:
        return jsonify({"status": "error", "detail": str(e)}), 503

Then add a Vigilmon monitor:

  • URL: https://mlflow.yourdomain.com/artifact-health
  • Method: GET
  • Expected status: 200
  • Check interval: 5 minutes

Step 4: Configure alert escalation

# Vigilmon alert configuration
monitors:
  - name: "MLflow Tracking Server"
    interval: 60
    alerts:
      - channel: pagerduty
        severity: critical
        after_failures: 2

  - name: "MLflow Model Registry"
    interval: 120
    alerts:
      - channel: slack
        severity: high
        after_failures: 3

  - name: "MLflow Artifact Store"
    interval: 300
    alerts:
      - channel: email
        severity: medium
        after_failures: 2

7. Setting Up a Status Page

If you have internal stakeholders (data scientists, ML engineers, product teams) who depend on MLflow, a status page prevents repeat "is MLflow down?" Slack messages. In Vigilmon, group your MLflow monitors into a single status page and share the URL with your organization.

Alerting on Degradation, Not Just Downtime

MLflow can degrade without going fully down. Watch for these degradation signals:

  • Response time creep: P95 latency on /health rising from 50ms to 800ms indicates DB saturation
  • Artifact upload errors: 5xx responses on artifact upload endpoints during peak training hours
  • Registry version drift: Model versions not updating in the registry while training jobs succeed

With Vigilmon's response time monitoring enabled, you can set threshold alerts at 2× and 5× baseline to catch these before they become full outages. Your ML pipelines will be more reliable, and your on-call team will spend less time fire-fighting MLflow issues.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles