Monitoring Phoenix (Arize) ML Applications with Vigilmon: External Uptime, Model API Health & Synthetic Probes

Phoenix by Arize is an open-source ML observability platform that gives you deep, inside-out visibility into your machine learning and LLM applications: trace-level debugging, dataset analysis, embedding visualizations, retrieval quality evaluation, and model performance monitoring. What Phoenix doesn't provide is an independent external health check — a synthetic probe that measures whether your model serving endpoints are reachable and responding correctly from the outside world. Vigilmon fills that gap. It sits entirely outside your infrastructure and probes your services from multiple external locations, alerting you independently of whether your Phoenix instance is healthy.

This tutorial covers adding external uptime monitoring to ML model APIs already observed with Phoenix.

What You'll Build

• A /health endpoint that reflects the real state of your model serving dependencies
• A Vigilmon HTTP monitor with response-body assertions
• A heartbeat monitor for model retraining and batch inference jobs
• An alert routing strategy that keeps Phoenix ML alerts and Vigilmon uptime alerts separate but correlated

Prerequisites

• A Phoenix (Arize) instance — cloud or self-hosted — with your ML application sending traces or spans
• At least one model inference API endpoint reachable via a public HTTPS URL
• A free account at vigilmon.online

Step 1: Add a Health Endpoint to Your Model Serving Layer

Phoenix observes your model from inside: tracing predictions, capturing embeddings, and evaluating output quality. Vigilmon needs an HTTP endpoint it can reach from the outside. Make the health check verify real dependencies — model readiness, feature store connectivity, and backing database — not just a static 200 that would pass even when the model is unloaded.

Python (FastAPI serving a scikit-learn model)

# app/health.py
import os
import numpy as np
from fastapi import FastAPI
from fastapi.responses import JSONResponse

app = FastAPI()
model = None

@app.on_event("startup")
async def load_model():
    global model
    import joblib
    model = joblib.load(os.environ["MODEL_PATH"])

@app.get("/health")
async def health():
    checks = {}
    ok = True

    # Model readiness check — run a dummy inference
    try:
        dummy = np.zeros((1, model.n_features_in_))
        model.predict(dummy)
        checks["model"] = "ok"
    except Exception as exc:
        checks["model"] = f"error: {exc}"
        ok = False

    # Feature store connectivity
    try:
        import redis
        r = redis.Redis.from_url(os.environ["FEATURE_STORE_URL"], socket_timeout=2)
        r.ping()
        checks["feature_store"] = "ok"
    except Exception as exc:
        checks["feature_store"] = f"error: {exc}"
        ok = False

    return JSONResponse(
        status_code=200 if ok else 503,
        content={"status": "ok" if ok else "degraded", "checks": checks},
    )

Python (FastAPI with a Hugging Face transformer)

# app/health.py
import os
from fastapi import FastAPI
from fastapi.responses import JSONResponse

app = FastAPI()
pipeline_model = None

@app.on_event("startup")
async def load_model():
    global pipeline_model
    from transformers import pipeline
    pipeline_model = pipeline("text-classification", model=os.environ["HF_MODEL_PATH"])

@app.get("/health")
async def health():
    checks = {}
    ok = True

    # Sanity inference
    try:
        result = pipeline_model("health check probe")
        checks["model"] = "ok" if result else "empty_response"
        if not result:
            ok = False
    except Exception as exc:
        checks["model"] = f"error: {exc}"
        ok = False

    # Database probe
    try:
        import sqlalchemy
        engine = sqlalchemy.create_engine(os.environ["DATABASE_URL"])
        with engine.connect() as conn:
            conn.execute(sqlalchemy.text("SELECT 1"))
        checks["database"] = "ok"
    except Exception as exc:
        checks["database"] = f"error: {exc}"
        ok = False

    return JSONResponse(
        status_code=200 if ok else 503,
        content={"status": "ok" if ok else "degraded", "checks": checks},
    )

Verify the endpoint is reachable before configuring Vigilmon:

curl -s https://your-ml-api.example.com/health | jq .
# {"status":"ok","checks":{"model":"ok","feature_store":"ok"}}

Step 2: Configure Vigilmon HTTP Monitor

1. Log in to Vigilmon → Add Monitor → HTTP.
2. URL: https://your-ml-api.example.com/health
3. Check interval: 60 seconds — appropriate for production model serving.
4. Response timeout: 20 seconds — model warm-up checks can take longer.
5. Expected status code: 200.
6. JSON body assertion:
   • Path: status
   • Expected value: ok
7. Click Save.

Vigilmon immediately begins probing from multiple external probe locations. A non-200 response or a failed JSON assertion triggers an alert to your configured channels, independently of Phoenix's observability pipeline.

Multiple model endpoints

Production ML platforms often serve multiple models. Create a separate Vigilmon monitor for each:

| Model endpoint | Vigilmon monitor URL | Interval | |---|---|---| | Primary classification model | https://classify.example.com/health | 60 s | | Embeddings service | https://embed.example.com/health | 60 s | | LLM inference API | https://llm.example.com/health | 60 s | | Shadow / canary model | https://canary.example.com/health | 120 s |

Group all monitors under a Monitor group to track overall ML platform availability at a glance.

Step 3: Heartbeat Monitor for Retraining and Batch Inference Jobs

Phoenix observes real-time predictions, but model retraining pipelines, nightly batch inference runs, and offline evaluation jobs execute asynchronously. A failed retraining job that Phoenix can't detect — because it never receives spans — will silently leave a stale model in production. Use Vigilmon heartbeats to enforce execution guarantees.

# jobs/retrain_model.py
import os
import requests

def retrain():
    """Pull new training data, retrain, evaluate, and promote if metrics pass."""
    data = fetch_training_data()
    new_model = train(data)
    metrics = evaluate(new_model)
    if metrics["accuracy"] >= float(os.environ.get("MIN_ACCURACY", "0.85")):
        promote_model(new_model)
    else:
        raise ValueError(f"Model accuracy {metrics['accuracy']:.3f} below threshold")

if __name__ == "__main__":
    try:
        retrain()
        requests.post(os.environ["VIGILMON_HEARTBEAT_URL"], timeout=5)
        print("Retraining complete — heartbeat sent")
    except Exception as exc:
        # Deliberate silence — missed heartbeat fires Vigilmon alert
        print(f"Retraining failed: {exc}")
        raise

In Vigilmon, create a Heartbeat monitor and set the grace period slightly above your pipeline cadence. For a weekly retraining job, use 8 days. Store the heartbeat URL in your secrets manager and inject it as VIGILMON_HEARTBEAT_URL.

Airflow DAG integration

# dags/retrain_dag.py
from airflow import DAG
from airflow.operators.python import PythonOperator
from airflow.utils.dates import days_ago
import os, requests

def ping_vigilmon():
    requests.post(os.environ["VIGILMON_HEARTBEAT_URL"], timeout=5)

with DAG("model_retrain", schedule_interval="@weekly", start_date=days_ago(1)) as dag:
    fetch = PythonOperator(task_id="fetch_data", python_callable=fetch_training_data)
    train = PythonOperator(task_id="train", python_callable=train_model)
    promote = PythonOperator(task_id="promote", python_callable=promote_model)
    heartbeat = PythonOperator(task_id="heartbeat", python_callable=ping_vigilmon)

    fetch >> train >> promote >> heartbeat

The heartbeat only fires if the entire DAG completes successfully. A failed or skipped DAG leaves Vigilmon without a ping, which triggers an alert after the grace period.

Step 4: Alert Routing Strategy

Phoenix alerts on model quality signals: data drift, prediction quality degradation, embedding distribution shift, evaluation score drops. Vigilmon alerts on availability: the inference API is unreachable, a health check failed, a retraining job missed its window. These are distinct failure modes.

| Failure mode | Source | Route to | |---|---|---| | Data drift detected | Phoenix | ML team Slack #model-health | | Prediction quality degradation | Phoenix | ML team + product lead | | Inference API endpoint down | Vigilmon | PagerDuty on-call + Slack #prod-alerts | | Retraining job missed | Vigilmon | ML team Slack #ml-ops-critical | | TLS certificate expiring | Vigilmon | DevOps team |

Configure Vigilmon alert channels under Alerts → Add channel:

• Email: your on-call distribution list.
• PagerDuty webhook: wire to your primary on-call rotation for user-facing outages.
• Slack webhook: a dedicated #vigilmon-alerts channel separate from Phoenix dashboards.

Keeping alert channels separate helps distinguish "our model quality is degrading" from "our model API is completely unreachable" — two very different incident responses.

Step 5: Correlating Phoenix Observations and Vigilmon Alerts During Incidents

When Vigilmon fires a downtime alert on a model endpoint, use this checklist to determine root cause:

1. Check Vigilmon probe regions — all failing points to the model serving infrastructure. One failing suggests network routing.
2. Open Phoenix → check the trace error rate and latency for the same time window. Rising errors in Phoenix traces often precede availability failures.
3. Check model deployment status — was a new model version promoted recently? A bad promotion can cause the model loading step to fail, making the health endpoint return 503.
4. Check GPU / resource exhaustion — models under heavy load can exhaust GPU memory, causing new inference requests to fail while existing long-running requests continue.
5. Compare timestamps — did Phoenix show quality degradation before Vigilmon fired? Quality drops can cascade to availability failures if models start throwing exceptions.

Step 6: Public Status Page

Phoenix dashboards are private ML operations views. Give customers and API consumers a public-facing status page using Vigilmon.

1. In Vigilmon, go to Status Pages → Create.
2. Add your production inference API monitor and any critical heartbeat monitors.
3. Configure a custom domain (e.g., status.example.com) or use the Vigilmon subdomain.
4. Embed the status badge in your developer documentation:

<img src="https://vigilmon.online/badge/<monitor-id>.svg" alt="Model API Status" />

Your API consumers get real-time uptime visibility without requiring access to Phoenix or your internal infrastructure.

What Vigilmon Adds to a Phoenix Stack

| Capability | Phoenix (Arize) | Vigilmon | |---|---|---| | Prediction traces and spans | Yes | No | | Data drift and distribution shift | Yes | No | | Embedding visualization | Yes | No | | Retrieval quality evaluation | Yes | No | | External synthetic HTTP probing | No | Yes | | DNS resolution failure detection | No | Yes | | TLS/SSL certificate expiry alerts | No | Yes | | Retraining job heartbeat monitoring | No | Yes | | Public status page | No | Yes | | Independent of model serving layer | No | Yes |

Phoenix gives you deep visibility into the quality and behavior of your ML models from the inside. Vigilmon gives you the external, synthetic view that tells you what your users actually experience when they call your inference API. Together they cover every dimension of ML service reliability: from model quality metrics to customer-facing uptime.

Add external uptime monitoring to your Phoenix-instrumented ML application today — register free at vigilmon.online.