Monitoring WhyLabs-Observed AI Models with Vigilmon: External Uptime, API Health & Heartbeat Checks

WhyLabs is an AI observability platform built around whylogs, the open-source data logging library. It monitors your ML models in production by tracking data quality, feature drift, model performance metrics, and data pipeline health. WhyLabs gives you a continuous statistical view of what your model is seeing and how well it is performing. What it doesn't provide is an independent external health check — a synthetic probe that measures whether your model serving endpoints are reachable and responding correctly from the outside. Vigilmon fills that gap. It sits entirely outside your infrastructure and probes your services from multiple external locations, alerting you independently of whether WhyLabs' logging pipeline is healthy.

This tutorial covers adding external uptime monitoring to AI model APIs already observed with WhyLabs.

What You'll Build

• A /health endpoint that reflects the real state of your model serving dependencies
• A Vigilmon HTTP monitor with response-body assertions
• A heartbeat monitor for data ingestion and batch scoring jobs
• An alert routing strategy that keeps WhyLabs model quality alerts and Vigilmon uptime alerts separate but correlated

Prerequisites

• A WhyLabs account with your ML application sending whylogs profiles via the whylogs Python or Java library
• At least one model inference endpoint reachable via a public HTTPS URL
• A free account at vigilmon.online

Step 1: Add a Health Endpoint to Your Model Serving Layer

WhyLabs observes your model by analyzing statistical profiles of inputs and outputs. Vigilmon needs an HTTP endpoint it can reach from the outside. Make the health check verify real dependencies — model readiness, feature store connectivity, and backing infrastructure — not just a static 200 that would pass even when the model fails to load.

Python (Flask)

# app/health.py
import os
import numpy as np
from flask import Flask, jsonify

app = Flask(__name__)

# Loaded at startup via your model management code
model = None

def get_model():
    global model
    if model is None:
        import joblib
        model = joblib.load(os.environ["MODEL_PATH"])
    return model

@app.route("/health")
def health():
    checks = {}
    ok = True

    # Model readiness check
    try:
        m = get_model()
        dummy = np.zeros((1, m.n_features_in_))
        m.predict(dummy)
        checks["model"] = "ok"
    except Exception as exc:
        checks["model"] = f"error: {exc}"
        ok = False

    # Feature store connectivity
    try:
        import psycopg2
        conn = psycopg2.connect(os.environ["FEATURE_DB_URL"], connect_timeout=3)
        conn.close()
        checks["feature_store"] = "ok"
    except Exception as exc:
        checks["feature_store"] = f"error: {exc}"
        ok = False

    # WhyLabs writer connectivity (optional but useful)
    try:
        import whylogs as why
        # A lightweight profile flush verifies the writer is reachable
        from whylogs.api.writer.whylabs import WhyLabsWriter
        writer = WhyLabsWriter()
        checks["whylabs_writer"] = "ok"
    except Exception as exc:
        checks["whylabs_writer"] = f"warning: {exc}"
        # Writer unavailability is non-fatal for serving

    return jsonify({"status": "ok" if ok else "degraded", "checks": checks}), \
           200 if ok else 503

Java (Spring Boot)

// HealthController.java
import org.springframework.http.ResponseEntity;
import org.springframework.jdbc.core.JdbcTemplate;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.RestController;

import java.util.HashMap;
import java.util.Map;

@RestController
public class HealthController {

    private final JdbcTemplate jdbcTemplate;
    private final ModelRegistry modelRegistry;

    public HealthController(JdbcTemplate jdbcTemplate, ModelRegistry modelRegistry) {
        this.jdbcTemplate = jdbcTemplate;
        this.modelRegistry = modelRegistry;
    }

    @GetMapping("/health")
    public ResponseEntity<Map<String, Object>> health() {
        Map<String, String> checks = new HashMap<>();
        boolean ok = true;

        // Model registry check
        try {
            modelRegistry.getLatestVersion();
            checks.put("model_registry", "ok");
        } catch (Exception e) {
            checks.put("model_registry", "error: " + e.getMessage());
            ok = false;
        }

        // Feature database check
        try {
            jdbcTemplate.queryForObject("SELECT 1", Integer.class);
            checks.put("feature_store", "ok");
        } catch (Exception e) {
            checks.put("feature_store", "error: " + e.getMessage());
            ok = false;
        }

        Map<String, Object> response = new HashMap<>();
        response.put("status", ok ? "ok" : "degraded");
        response.put("checks", checks);

        return ok
            ? ResponseEntity.ok(response)
            : ResponseEntity.status(503).body(response);
    }
}

Verify the endpoint is reachable before configuring Vigilmon:

curl -s https://your-model-api.example.com/health | jq .
# {"status":"ok","checks":{"model":"ok","feature_store":"ok","whylabs_writer":"ok"}}

Step 2: Configure Vigilmon HTTP Monitor

1. Log in to Vigilmon → Add Monitor → HTTP.
2. URL: https://your-model-api.example.com/health
3. Check interval: 60 seconds — appropriate for production model serving.
4. Response timeout: 15 seconds — model loading checks can be slower.
5. Expected status code: 200.
6. JSON body assertion:
   • Path: status
   • Expected value: ok
7. Click Save.

Vigilmon immediately begins probing from multiple external locations. A non-200 response or a failed JSON assertion triggers an alert to your configured channels, independently of WhyLabs' statistical monitoring pipeline.

Multiple model deployments

Large ML platforms deploy many models simultaneously. Create a Vigilmon monitor for each production endpoint:

| Model | Vigilmon monitor URL | Interval | |---|---|---| | Fraud detection API | https://fraud.example.com/health | 60 s | | Recommendation model | https://recommend.example.com/health | 60 s | | NLP classification API | https://nlp.example.com/health | 60 s | | Shadow / A-B test model | https://shadow.example.com/health | 120 s |

Group all monitors under a Monitor group in Vigilmon to see overall ML platform availability at a glance.

Step 3: Heartbeat Monitor for Data Ingestion and Batch Scoring Jobs

WhyLabs monitors statistical profiles that your application logs via whylogs. If a data ingestion pipeline fails silently, WhyLabs stops receiving profiles — and you lose visibility without knowing why. Use Vigilmon heartbeats to enforce execution guarantees on the pipelines that feed WhyLabs and run batch scoring jobs.

# jobs/daily_batch_scoring.py
import os
import requests

def batch_score():
    """Score today's customer records and write results to the data warehouse."""
    records = fetch_todays_records()
    scores = model.predict(records)
    write_scores_to_warehouse(scores)

    # Log a whylogs profile for WhyLabs
    import whylogs as why
    import pandas as pd
    profile = why.log(pd.DataFrame({"score": scores}))
    profile.writer("whylabs").write()

if __name__ == "__main__":
    try:
        batch_score()
        requests.post(os.environ["VIGILMON_HEARTBEAT_URL"], timeout=5)
        print("Batch scoring complete — heartbeat sent")
    except Exception as exc:
        # Deliberate silence — missed heartbeat fires Vigilmon alert
        print(f"Batch scoring failed: {exc}")
        raise

In Vigilmon, create a Heartbeat monitor with a grace period slightly above your job cadence. For a daily scoring job, use 25 hours. Store the heartbeat URL in your secrets manager and inject it as VIGILMON_HEARTBEAT_URL.

AWS Lambda scheduled job

# lambda_function.py
import os
import boto3
import requests

def lambda_handler(event, context):
    try:
        run_daily_scoring()
        requests.post(os.environ["VIGILMON_HEARTBEAT_URL"], timeout=5)
        return {"statusCode": 200, "body": "scoring complete"}
    except Exception as exc:
        # Let Lambda retry; if retries exhaust, heartbeat is never sent
        raise exc

Step 4: Alert Routing Strategy

WhyLabs alerts on ML quality signals: feature drift above threshold, data quality violations, model performance degradation, missing data in ingestion pipelines. Vigilmon alerts on infrastructure availability: the inference endpoint is unreachable, a health check failed, a batch job missed its window. These are distinct failure modes.

| Failure mode | Source | Route to | |---|---|---| | Feature drift detected | WhyLabs | ML team Slack #model-monitoring | | Data quality violation | WhyLabs | Data engineering team | | Model API endpoint down | Vigilmon | PagerDuty on-call + Slack #prod-alerts | | Batch scoring job missed | Vigilmon | ML team Slack #ml-ops-critical | | TLS certificate expiring | Vigilmon | DevOps team |

Configure Vigilmon alert channels under Alerts → Add channel:

• Email: your on-call distribution list.
• PagerDuty webhook: wire to your primary on-call rotation for endpoint outages.
• Slack webhook: a dedicated #vigilmon-alerts channel separate from WhyLabs dashboards.

Keeping channels separate prevents alert fatigue — a data drift notification and an API outage notification require very different responses.

Step 5: Correlating WhyLabs Observations and Vigilmon Alerts During Incidents

When Vigilmon fires a downtime alert on a model endpoint, use this checklist to determine root cause:

1. Check Vigilmon probe regions — all failing points to application or infrastructure. One failing suggests network routing.
2. Open WhyLabs → check if profiles stopped arriving in the same time window. Absent profiles often mean the ingestion pipeline died around the same time as the outage.
3. Check model version deployment history — a bad model promotion can cause the health check to fail if the new model fails to initialize.
4. Check data pipeline status — if the feature store or database feeding your model went offline, both WhyLabs profiles and serving health can fail together.
5. Compare timestamps — did WhyLabs show data quality violations just before the availability failure? Corrupt input data can cascade to serving errors.

Step 6: Public Status Page

WhyLabs dashboards are private AI observability views. Give your API consumers a public-facing status page using Vigilmon.

1. In Vigilmon, go to Status Pages → Create.
2. Add your production model serving monitors and any critical heartbeat monitors.
3. Configure a custom domain (e.g., status.example.com) or use the Vigilmon subdomain.
4. Embed the badge in your developer documentation:

<img src="https://vigilmon.online/badge/<monitor-id>.svg" alt="Model API Status" />

What Vigilmon Adds to a WhyLabs Stack

| Capability | WhyLabs | Vigilmon | |---|---|---| | Feature drift detection | Yes | No | | Data quality monitoring | Yes | No | | Model performance tracking | Yes | No | | whylogs profile ingestion | Yes | No | | External synthetic HTTP probing | No | Yes | | DNS resolution failure detection | No | Yes | | TLS/SSL certificate expiry alerts | No | Yes | | Batch job heartbeat monitoring | No | Yes | | Public status page | No | Yes | | Independent of ML infrastructure | No | Yes |

WhyLabs gives you continuous statistical visibility into the health of your AI models and the data flowing through them. Vigilmon gives you the external, synthetic view that tells you what your users actually experience when they call your inference API. Together they cover every dimension of AI service reliability: from feature drift detection to customer-facing uptime.

Add external uptime monitoring to your WhyLabs-observed AI platform today — register free at vigilmon.online.