Monitoring Monte Carlo Data Observability Integrations with Vigilmon

Monte Carlo is a leading data observability platform. It monitors your data warehouse for anomalies, detects data freshness issues, tracks schema changes, and opens incidents when data quality degrades. When Monte Carlo is healthy, your data team has visibility into every corner of the data pipeline.

But here's the blind spot: Monte Carlo needs to be connected to your data sources to observe them. If the Monte Carlo collector loses access to your warehouse, if your Monte Carlo API keys expire, or if your data sources stop sending metadata — Monte Carlo goes quiet. And a quiet observability platform looks exactly like a healthy one.

This tutorial shows you how to build a safety net around Monte Carlo: a lightweight monitoring layer that checks whether Monte Carlo is actively receiving data, alert you when ingestion jobs stall, and route those alerts through Vigilmon so your team always knows whether their data observability tool is actually working.

Understanding the Monte Carlo architecture

Monte Carlo uses a collector agent deployed in your environment to periodically pull metadata from your data sources (BigQuery, Snowflake, Databricks, etc.). The collector runs ingestion jobs on a schedule and pushes results to Monte Carlo's cloud platform.

The components that can fail:

The collector agent — the agent process can crash, get OOM-killed, or lose network connectivity to Monte Carlo's API. If the agent is down, Monte Carlo stops receiving updates from that environment.

Ingestion job scheduling — the collector runs individual jobs for each data source and job type (metadata, query logs, table stats). A specific job type can fail while others succeed, giving you partial visibility.

API key rotation — if your organization rotates API keys or the Monte Carlo API key expires, the collector can no longer authenticate. Ingestion stops.

Data source credentials — if your warehouse credentials change (password rotation, IAM role change), the collector fails to query the source, but the collector process itself remains healthy.

None of these failures produce alerts from Monte Carlo itself — because Monte Carlo isn't receiving data, it has nothing to analyze.

Step 1: Build a collector health checker

Monte Carlo provides a REST API and Python SDK for checking collector status and recent ingestion activity. Install the SDK:

pip install pycarlo httpx

Write a health checker that queries Monte Carlo for recent ingestion activity:

# mc_health.py
from pycarlo.core import Client, Session
import httpx
import os
import logging
from datetime import datetime, timedelta, timezone

logger = logging.getLogger(__name__)


def get_mc_client() -> Client:
    return Client(session=Session(
        mcd_id=os.environ["MCD_ID"],
        mcd_token=os.environ["MCD_TOKEN"],
    ))


def check_collector_health(
    max_staleness_minutes: int = 60,
) -> dict:
    """
    Check Monte Carlo collector health by verifying recent ingestion activity.
    Returns a health summary dict.
    """
    client = get_mc_client()

    try:
        # Query for recent events from the Monte Carlo API
        # This checks that the collector is actively sending data
        query = """
        query GetIngestionStatus {
          getUser {
            email
          }
          getWarehouseConnections {
            edges {
              node {
                name
                connectionType
                createdTime
                lastUpdateTime
              }
            }
          }
        }
        """
        result = client(query)
        connections = result.get_warehouse_connections.edges

        now = datetime.now(timezone.utc)
        stale_connections = []
        healthy_connections = []

        for edge in connections:
            conn = edge.node
            last_update = conn.last_update_time
            if last_update is None:
                stale_connections.append({"name": conn.name, "reason": "never_updated"})
                continue

            age_minutes = (now - last_update).total_seconds() / 60
            if age_minutes > max_staleness_minutes:
                stale_connections.append({
                    "name": conn.name,
                    "last_update": last_update.isoformat(),
                    "age_minutes": round(age_minutes, 1),
                })
            else:
                healthy_connections.append({
                    "name": conn.name,
                    "last_update": last_update.isoformat(),
                    "age_minutes": round(age_minutes, 1),
                })

        return {
            "status": "ok" if not stale_connections else "degraded",
            "healthy_connections": len(healthy_connections),
            "stale_connections": stale_connections,
            "timestamp": now.isoformat(),
        }

    except Exception as e:
        logger.error(f"Monte Carlo health check failed: {e}")
        return {
            "status": "error",
            "error": str(e),
            "timestamp": datetime.utcnow().isoformat(),
        }


def ping_heartbeat_if_healthy(health: dict, heartbeat_url: str | None):
    """Ping Vigilmon heartbeat only if Monte Carlo is healthy."""
    if health["status"] == "ok" and heartbeat_url:
        try:
            with httpx.Client(timeout=10) as client:
                client.get(heartbeat_url)
            logger.info("Monte Carlo heartbeat pinged")
        except Exception as e:
            logger.warning(f"Heartbeat ping failed: {e}")

Step 2: Build a health endpoint

Expose the Monte Carlo health check through an HTTP endpoint your monitoring system can poll:

# app.py
from fastapi import FastAPI
from fastapi.responses import JSONResponse
import os
from mc_health import check_collector_health

app = FastAPI()


@app.get("/health/monte-carlo")
def mc_health():
    """
    Checks Monte Carlo collector health.
    Returns 503 when connections are stale or API is unreachable.
    """
    health = check_collector_health(
        max_staleness_minutes=int(os.environ.get("MC_MAX_STALENESS_MINUTES", "90")),
    )

    http_status = 200 if health["status"] == "ok" else 503
    return JSONResponse(status_code=http_status, content=health)


@app.get("/health")
def overall_health():
    """Overall data platform health."""
    mc = check_collector_health()
    # Add checks for other systems here

    all_ok = mc["status"] == "ok"
    return JSONResponse(
        status_code=200 if all_ok else 503,
        content={
            "status": "ok" if all_ok else "degraded",
            "monte_carlo": mc,
        }
    )

Step 3: Set up a heartbeat runner

For environments where Monte Carlo runs scheduled ingestion jobs, add a heartbeat runner that executes on the same schedule as your ingestion:

# mc_heartbeat_runner.py
import os
import logging
from mc_health import check_collector_health, ping_heartbeat_if_healthy

logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)


def run_health_check_and_heartbeat():
    """
    Run Monte Carlo health check and ping heartbeat if healthy.
    Designed to run on a schedule matching your ingestion window.
    """
    heartbeat_url = os.environ.get("MC_COLLECTOR_HEARTBEAT_URL")

    logger.info("Running Monte Carlo health check...")
    health = check_collector_health(max_staleness_minutes=90)

    logger.info(f"Health status: {health['status']}")
    if health.get("stale_connections"):
        logger.warning(f"Stale connections: {health['stale_connections']}")

    ping_heartbeat_if_healthy(health, heartbeat_url)

    if health["status"] != "ok":
        logger.error(f"Monte Carlo health check failed: {health}")
        exit(1)

    logger.info(
        f"Monte Carlo healthy: {health['healthy_connections']} connections active"
    )


if __name__ == "__main__":
    run_health_check_and_heartbeat()

Schedule this script to run every hour via cron:

# crontab -e
0 * * * * /usr/local/bin/python /opt/mc_heartbeat_runner.py >> /var/log/mc_heartbeat.log 2>&1

Or as a Kubernetes CronJob:

apiVersion: batch/v1
kind: CronJob
metadata:
  name: mc-health-check
spec:
  schedule: "0 * * * *"
  jobTemplate:
    spec:
      template:
        spec:
          containers:
            - name: mc-health-check
              image: your-org/mc-health-check:latest
              env:
                - name: MCD_ID
                  valueFrom:
                    secretKeyRef:
                      name: monte-carlo-creds
                      key: mcd_id
                - name: MCD_TOKEN
                  valueFrom:
                    secretKeyRef:
                      name: monte-carlo-creds
                      key: mcd_token
                - name: MC_COLLECTOR_HEARTBEAT_URL
                  valueFrom:
                    secretKeyRef:
                      name: vigilmon-heartbeats
                      key: mc_collector_url
          restartPolicy: OnFailure

Step 4: Connect to Vigilmon

Set up monitoring in Vigilmon:

HTTP monitor for collector health

1. Sign up at vigilmon.online — free tier, no credit card
2. Click New Monitor → HTTP
3. Enter https://your-data-platform.internal/health/monte-carlo
4. Set interval: 10 minutes
5. Save

Heartbeat monitor for ingestion health

1. Click New Monitor → Heartbeat
2. Name it "monte_carlo_collector_health"
3. Set expected interval: 90 minutes (matching your ingestion schedule with a buffer)
4. Copy the unique URL

Add to your environment:

# .env
MCD_ID=your-mcd-id
MCD_TOKEN=your-mcd-token
MC_COLLECTOR_HEARTBEAT_URL=https://vigilmon.online/api/heartbeat/your-unique-token
MC_MAX_STALENESS_MINUTES=90

Now if your collector agent crashes, your API keys expire, or your CronJob fails to schedule, Vigilmon alerts you within 90 minutes — long before your data team notices that Monte Carlo isn't surfacing new incidents.

Step 5: Monitor Monte Carlo circuit breakers

Monte Carlo exposes API endpoints for incidents and circuit breakers. Build monitoring around the incident API to catch when Monte Carlo itself is opening incidents at an unusual rate (which can indicate your data environment is broadly unhealthy):

# mc_incident_check.py
from pycarlo.core import Client, Session
import os
from datetime import datetime, timedelta, timezone


def check_open_incidents(threshold: int = 10) -> dict:
    """
    Returns count of open Monte Carlo incidents.
    High counts may indicate broad data quality degradation.
    """
    client = Client(session=Session(
        mcd_id=os.environ["MCD_ID"],
        mcd_token=os.environ["MCD_TOKEN"],
    ))

    query = """
    query GetIncidents {
      getIncidents(first: 50, filter: {status: OPEN}) {
        edges {
          node {
            id
            status
            incidentType
            createdTime
          }
        }
      }
    }
    """

    result = client(query)
    incidents = result.get_incidents.edges
    open_count = len(incidents)

    return {
        "open_incidents": open_count,
        "status": "ok" if open_count < threshold else "alert",
        "threshold": threshold,
        "timestamp": datetime.now(timezone.utc).isoformat(),
    }

Expose this via a dedicated endpoint:

@app.get("/health/monte-carlo/incidents")
def mc_incidents():
    status = check_open_incidents(
        threshold=int(os.environ.get("MC_INCIDENT_ALERT_THRESHOLD", "10"))
    )
    http_status = 200 if status["status"] == "ok" else 503
    return JSONResponse(status_code=http_status, content=status)

Add a second Vigilmon HTTP monitor pointing at /health/monte-carlo/incidents. If Monte Carlo suddenly has 10+ open incidents, Vigilmon alerts you — useful for detecting broad pipeline health degradation during warehouse migrations, major deploys, or data source outages.

Step 6: Slack alerts and escalation

Configure Vigilmon to route alerts to your data platform team:

1. Create a Slack incoming webhook for your #data-platform-alerts channel
2. In Vigilmon: Notifications → New Channel → Slack
3. Paste the webhook URL
4. Enable on all Monte Carlo monitors

Set up escalation for critical monitors:

• Primary: Slack alert immediately
• Escalation after 30 minutes: page the on-call data engineer via PagerDuty webhook

Vigilmon supports webhook-based escalation, so you can chain alerts to any incident management system.

Step 7: Status page for data consumers

When Monte Carlo is down or ingestion has stalled, business stakeholders need to understand why their dashboards show stale data.

In Vigilmon:

1. Status Pages → New Status Page
2. Name it "Data Observability Platform"
3. Include your Monte Carlo health monitors
4. Share the URL with your BI team and data analysts

When they see stale dashboards and the status page shows "Monte Carlo Collector: Degraded," they understand the situation without paging the data engineering team.

Monitoring coverage map

| What you're monitoring | Monitor type | What failure it catches | |---|---|---| | Monte Carlo collector health endpoint | HTTP | Stale connections, API errors | | Hourly ingestion health check | Heartbeat | Scheduler failure, collector crash, key expiry | | Monte Carlo incident count | HTTP | Broad data quality degradation events | | Data platform overall health | HTTP | Cross-system rollup |

Summary

Monte Carlo observes your data. Vigilmon observes Monte Carlo.

The combination closes the observability loop: when Monte Carlo is functioning, it catches data quality issues. When Monte Carlo itself has a problem, Vigilmon catches that too.

The setup takes under an hour and ensures your data observability platform is itself observable.

Get started free at vigilmon.online — your first monitor is running in under a minute.