tutorial

How to Monitor MLRun with Vigilmon

MLRun is an open-source MLOps platform providing end-to-end ML pipeline orchestration, feature store, model serving, and monitoring. Here's how to monitor your MLRun services, track pipeline health, and get alerts when model endpoints or workflow runs fail.

MLRun is an open-source MLOps platform that orchestrates end-to-end ML pipelines — from data ingestion through feature engineering, model training, and real-time serving. It integrates Kubeflow for pipeline orchestration, Nuclio for serverless model deployment, and Spark for distributed data processing, and provides a feature store, model registry, and experiment tracker in one platform. In production, MLRun's API server, model endpoints, and pipeline schedulers all need uptime monitoring. Vigilmon gives you HTTP health checks for your MLRun API and model endpoints, heartbeat monitoring for scheduled pipeline runs, SSL alerts for your serving layer, and instant notifications when any part of your MLOps infrastructure goes offline.

What You'll Set Up

  • HTTP uptime monitor for the MLRun API server
  • HTTP monitor for deployed Nuclio model endpoints
  • Cron heartbeat for scheduled MLRun pipeline runs
  • SSL certificate expiry alerts for your MLRun services
  • Alert channels for immediate notification on pipeline and serving failures

Prerequisites

  • MLRun deployed (local, Kubernetes, or the hosted Iguazio platform)
  • At least one project with a model endpoint or scheduled pipeline
  • A free Vigilmon account

Step 1: Verify the MLRun API Server Health Endpoint

MLRun's API server exposes a health endpoint. Confirm it's reachable before setting up monitoring:

# Check the MLRun API server health
curl http://localhost:8080/api/healthz
# {"status": "ok"}

For a Kubernetes deployment, the API server is typically exposed via a NodePort or Ingress:

curl https://mlrun-api.yourdomain.com/api/healthz

If you're adding a custom application layer on top of MLRun (e.g., a model gateway), include a health endpoint that checks MLRun connectivity:

from fastapi import FastAPI
from fastapi.responses import JSONResponse
import mlrun

app = FastAPI()
project = mlrun.get_or_create_project("my-project", context="./")

@app.get("/health")
async def health():
    try:
        # Verify MLRun API is reachable
        runs = project.list_runs(last=1)
        return {"status": "ok", "mlrun_api": "connected", "project": project.name}
    except Exception as e:
        return JSONResponse(
            status_code=503,
            content={"status": "error", "detail": str(e)},
        )

Step 2: Monitor the MLRun API Server

Add a Vigilmon monitor for your MLRun API:

  1. Log in to vigilmon.online and click Add Monitor.
  2. Set Type to HTTP / HTTPS.
  3. Enter your MLRun API health URL: https://mlrun-api.yourdomain.com/api/healthz
  4. Set Check interval to 1 minute.
  5. Set Expected HTTP status to 200.
  6. Click Save.

The MLRun API server is the control plane for all pipeline submissions, feature store queries, and model registry operations — downtime here blocks every ML workflow in your organization.

Step 3: Monitor Nuclio Model Endpoints

MLRun deploys models as Nuclio serverless functions, each with its own HTTP endpoint. Add monitors for your critical model serving endpoints:

MLRun model functions expose a /ready endpoint that returns 200 when the function is initialized and ready to serve predictions:

# Monitor a deployed model function
curl https://model-server.yourdomain.com:31001/ready

To add a Vigilmon monitor for a Nuclio model endpoint:

  1. Click Add Monitor → HTTP / HTTPS.
  2. Enter the model endpoint URL: https://model-server.yourdomain.com:31001/ready
  3. Set Check interval to 1 minute.
  4. Set Expected HTTP status to 200.
  5. Click Save.

You can also expose a richer health endpoint from within your MLRun function:

import mlrun

def init_context(context):
    # Load model at function initialization
    model_path = context.get_secret("MODEL_PATH")
    context.model = mlrun.artifacts.get_model(model_path)
    context.logger.info("Model loaded successfully")

def handler(context, event):
    if event.path == "/health":
        return context.Response(body='{"status": "ok"}', content_type="application/json")

    # Run inference
    input_data = event.body
    prediction = context.model.predict(input_data)
    return context.Response(body=str(prediction))

Add a Vigilmon monitor for each production model endpoint separately so you can isolate which model is down when alerts fire.

Step 4: Heartbeat Monitoring for Scheduled Pipeline Runs

MLRun supports scheduled pipeline runs via Kubernetes cron jobs or Kubeflow recurring runs. These background pipelines — nightly retraining, feature store refresh, drift detection — don't serve HTTP traffic. Use a Vigilmon cron heartbeat to confirm each run completes on schedule:

  1. In Vigilmon, click Add Monitor → Cron Heartbeat.
  2. Set the expected ping interval to match your pipeline schedule (e.g. 1440 minutes for daily retraining).
  3. Copy the heartbeat URL (e.g. https://vigilmon.online/heartbeat/abc123).

Add the Vigilmon ping as a final step in your MLRun pipeline:

import mlrun
import httpx

project = mlrun.get_or_create_project("churn-prediction", context="./")

def ping_vigilmon(context):
    """Final pipeline step — signal successful completion to Vigilmon."""
    httpx.get("https://vigilmon.online/heartbeat/abc123", timeout=5)
    context.logger.info("Vigilmon heartbeat sent")

# Define pipeline steps
@mlrun.handler()
def train_model(context, dataset_path: str):
    df = mlrun.get_dataitem(dataset_path).as_df()
    # ... training logic ...
    context.log_model("churn_model", body=model, framework="sklearn")

@mlrun.handler()
def evaluate_model(context, model_path: str):
    # ... evaluation logic ...
    context.log_result("accuracy", accuracy)

# Register the heartbeat step in the pipeline
project.set_function(ping_vigilmon, name="ping-vigilmon", kind="job")

For Kubeflow pipelines, add the ping as the final step so it only fires after all upstream steps complete successfully.

Step 5: Monitor the MLRun Feature Store

MLRun's feature store serves real-time features to model endpoints. Add a monitor for the online feature store serving layer:

@app.get("/health/feature-store")
async def feature_store_health():
    try:
        import mlrun.feature_store as fstore
        # List feature sets to confirm the feature store is reachable
        feature_sets = fstore.list_feature_sets(project="my-project")
        return {"status": "ok", "feature_sets": len(feature_sets)}
    except Exception as e:
        return JSONResponse(
            status_code=503,
            content={"status": "error", "detail": str(e)},
        )

Add this as a separate Vigilmon HTTP monitor. A feature store outage can silently degrade model predictions if features fall back to stale values.

Step 6: SSL Certificate Alerts

MLRun's API server, model endpoints, and UI dashboard all need valid SSL certificates for secure access:

  1. Open the HTTP monitor for your MLRun API (created in Step 2).
  2. Enable Monitor SSL certificate under the SSL section.
  3. Set Alert when certificate expires in less than 21 days.
  4. Click Save.

Repeat for each model endpoint monitor. Nuclio function URLs often use wildcard certificates tied to the cluster ingress — monitor them individually since each endpoint URL is distinct.

Step 7: Configure Alert Channels

  1. Go to Alert Channels in Vigilmon and add Slack, email, or a webhook endpoint.
  2. Set Consecutive failures before alert to 2 on each monitor — Nuclio function cold starts after inactivity can cause a single-probe timeout.
  3. Use Maintenance windows during MLRun upgrades or pipeline redeployments:
# Before upgrading MLRun or redeploying models
curl -X POST https://vigilmon.online/api/maintenance \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -d '{"monitor_id": "abc123", "duration_minutes": 15}'

# Perform the upgrade
helm upgrade mlrun mlrun/mlrun --version 1.7.0

# Maintenance window expires automatically

Summary

| Monitor | Target | What It Catches | |---|---|---| | HTTP endpoint | /api/healthz on MLRun API server | API server down, database unreachable | | Model endpoint | /ready on Nuclio function | Model serving crash, OOM, cold-start failure | | Feature store | Feature store health endpoint | Online feature serving outage | | Cron heartbeat | Heartbeat URL | Pipeline crash, missed retraining run | | SSL certificate | MLRun API and model domains | Certificate expiry |

MLRun ties together the entire ML lifecycle from experiment to production — but end-to-end ML infrastructure is only reliable with end-to-end monitoring. With Vigilmon watching your MLRun API, model endpoints, pipeline heartbeats, and SSL certificates, you catch failures before they affect predictions, retraining cycles, or data freshness across your production ML systems.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles