tutorial

Monitoring Mage.ai Data Pipelines with Vigilmon: External Uptime, Health Checks & Heartbeat Monitoring

Practical guide to complementing Mage.ai's data pipeline and ML orchestration with Vigilmon's external uptime checks — health endpoints, API monitors, pipeline heartbeat monitors, and independent alerting for your Mage.ai-orchestrated workflows.

Mage.ai is an open-source data pipeline and ML orchestration platform that lets data engineers and ML practitioners build, run, and monitor pipelines using interactive notebooks and a clean visual interface. It handles scheduling, backfills, dependency graphs, and integrated transformations. What Mage.ai's internal scheduler doesn't provide is an independent, external view of whether its HTTP API is reachable and whether its scheduled pipelines are actually completing on time. Vigilmon fills that gap. It probes your Mage.ai server's endpoints from multiple external locations on a continuous basis, and its heartbeat monitors detect when scheduled pipelines silently stop completing — giving your data platform team an independent early warning system that doesn't share failure modes with the pipeline engine itself.

This tutorial covers adding external uptime monitoring to Mage.ai servers and the pipelines they run.

What You'll Build

  • A pipeline block that pings a Vigilmon heartbeat on successful completion
  • A Vigilmon HTTP monitor for the Mage.ai server health endpoint
  • Heartbeat monitors for each critical scheduled pipeline
  • Alert routing that keeps Mage.ai internal scheduler alerts and external uptime alerts distinct

Prerequisites

  • A running Mage.ai server (self-hosted or Mage.ai Cloud) with a public HTTPS endpoint
  • At least one scheduled pipeline in production
  • A free account at vigilmon.online

Step 1: Add a Health Check to Your Mage.ai Server

Mage.ai exposes an internal status endpoint. For Vigilmon to monitor externally, you need a URL that's reachable from the internet and returns a machine-readable status.

Option A: Use the built-in Mage.ai status endpoint

Mage.ai's API server exposes /api/status. Verify it's accessible:

curl -s https://mage.example.com/api/status | jq .
# {"status": 200, "version": "0.9.x", ...}

If your Mage.ai instance is behind authentication, create a dedicated unauthenticated health endpoint using a custom API endpoint block.

Option B: Add a custom /health endpoint via Mage.ai's custom API

Create a file at mage_data/custom_api/health.py in your Mage project:

# mage_data/custom_api/health.py
import os
import json
from mage_ai.server.api.base import BaseHandler

class HealthHandler(BaseHandler):
    def get(self):
        checks = {}
        ok = True

        # Check database connectivity
        try:
            from mage_ai.data_preparation.repo_manager import get_repo_config
            get_repo_config()
            checks["repo_config"] = "ok"
        except Exception as exc:
            checks["repo_config"] = f"error: {exc}"
            ok = False

        # Check scheduler is running
        try:
            from mage_ai.orchestration.db.models.schedules import PipelineSchedule
            count = PipelineSchedule.select_all(status='active').count()
            checks["active_schedules"] = str(count)
        except Exception as exc:
            checks["active_schedules"] = f"error: {exc}"
            ok = False

        self.set_status(200 if ok else 503)
        self.write(json.dumps({
            "status": "ok" if ok else "degraded",
            "checks": checks,
        }))

Verify the endpoint:

curl -s https://mage.example.com/health | jq .
# {"status":"ok","checks":{"repo_config":"ok","active_schedules":"5"}}

Step 2: Configure Vigilmon HTTP Monitor for the Mage.ai Server

  1. Log in to Vigilmon → Add Monitor → HTTP.
  2. URL: https://mage.example.com/api/status (or your custom /health endpoint)
  3. Check interval: 60 seconds.
  4. Response timeout: 10 seconds.
  5. Expected status code: 200.
  6. JSON body assertion (for the custom endpoint):
    • Path: status
    • Expected value: ok
  7. Click Save.

Vigilmon probes from multiple external locations concurrently and requires a quorum of probe agents to agree before generating an alert, eliminating false positives from transient network issues that don't represent real server unavailability.

Step 3: Heartbeat Monitors for Scheduled Pipelines

This is the highest-value monitoring pattern for Mage.ai deployments. Mage.ai's internal scheduler will show a pipeline as "scheduled" and "active" even if the schedule trigger is broken or the pipeline is silently failing in a way that doesn't mark the run as errored. Vigilmon heartbeat monitors catch these silent failures independently.

Add a heartbeat ping block at the end of each critical pipeline.

Python block: heartbeat ping on success

Create a Data Exporter or Custom block at the end of each pipeline:

# blocks/send_vigilmon_heartbeat.py
import os
import urllib.request

if 'data_exporter' not in globals():
    from mage_ai.data_preparation.decorators import data_exporter

@data_exporter
def export_data(data, *args, **kwargs):
    """
    Send a heartbeat ping to Vigilmon.
    Add this block as the final step in critical pipelines.
    Set VIGILMON_HEARTBEAT_URL as an environment variable or secret.
    """
    heartbeat_url = os.environ.get('VIGILMON_HEARTBEAT_URL')
    if not heartbeat_url:
        print("VIGILMON_HEARTBEAT_URL not set — skipping heartbeat")
        return data

    try:
        req = urllib.request.Request(
            heartbeat_url,
            method='POST',
            headers={'User-Agent': 'mage-ai-heartbeat/1.0'},
        )
        with urllib.request.urlopen(req, timeout=5) as resp:
            print(f"Heartbeat sent — HTTP {resp.status}")
    except Exception as exc:
        print(f"Heartbeat send failed (non-fatal): {exc}")

    return data

Set the heartbeat URL as a project secret in Mage.ai:

  1. In Mage.ai → Settings → Secrets → Add secret.
  2. Name: VIGILMON_HEARTBEAT_URL_{PIPELINE_NAME} (one per pipeline).
  3. Paste the Vigilmon heartbeat URL for that pipeline's heartbeat monitor.

Configure Vigilmon heartbeat monitors

In Vigilmon → Add Monitor → Heartbeat, create one monitor per critical pipeline:

| Pipeline | Expected interval | Vigilmon grace period | |---|---|---| | daily_etl_pipeline | 24 h | 26 h | | hourly_feature_refresh | 60 min | 90 min | | ml_model_retrain | 7 days | 8 days | | data_quality_checks | 4 h | 5 h |

Use a grace period 20–30% longer than the pipeline interval to avoid false alerts from minor scheduling delays.

Trigger heartbeat from pipeline metadata (alternative approach)

If you prefer to send the heartbeat from outside the pipeline (e.g., from a CI job that monitors pipeline status), poll the Mage.ai API and send the heartbeat only on confirmed success:

#!/usr/bin/env python3
# scripts/check_pipeline_and_heartbeat.py
import os
import sys
import time
import urllib.request
import json

MAGE_API = os.environ["MAGE_API_URL"]
MAGE_TOKEN = os.environ["MAGE_API_TOKEN"]
PIPELINE_UUID = os.environ["PIPELINE_UUID"]
HEARTBEAT_URL = os.environ["VIGILMON_HEARTBEAT_URL"]

def get_latest_run_status():
    url = f"{MAGE_API}/api/pipeline_runs?pipeline_uuid={PIPELINE_UUID}&status=completed&limit=1"
    req = urllib.request.Request(url, headers={
        "Authorization": f"Bearer {MAGE_TOKEN}",
        "Content-Type": "application/json",
    })
    with urllib.request.urlopen(req, timeout=10) as resp:
        data = json.loads(resp.read())
    runs = data.get("pipeline_runs", [])
    if not runs:
        return None, None
    latest = runs[0]
    return latest.get("status"), latest.get("completed_at")

status, completed_at = get_latest_run_status()
print(f"Latest run status: {status}, completed: {completed_at}")

if status == "completed":
    req = urllib.request.Request(HEARTBEAT_URL, method="POST")
    with urllib.request.urlopen(req, timeout=5) as resp:
        print(f"Heartbeat sent — HTTP {resp.status}")
else:
    print(f"Pipeline not in completed state — no heartbeat sent")
    sys.exit(1)

Step 4: Alert Routing Strategy

Mage.ai generates internal alerts for pipeline failures, block errors, and scheduler issues — typically surfaced through Mage.ai's built-in notification integrations (Slack, email, PagerDuty). Vigilmon generates external uptime and heartbeat alerts. These two streams are complementary and should route to different channels to avoid noise overlap.

| Failure mode | Source | Route to | |---|---|---| | Pipeline block error | Mage.ai internal | Slack #data-pipeline-failures + data team | | Pipeline SLA breach | Mage.ai internal | Data team lead | | Mage.ai server unreachable | Vigilmon | PagerDuty + Slack #infra-critical | | Critical pipeline heartbeat missed | Vigilmon | Data platform on-call + Slack #data-ops-critical | | TLS certificate expiring | Vigilmon | DevOps team |

Configure Vigilmon alert channels under Alerts → Add channel:

  • Email: data platform on-call distribution list.
  • PagerDuty webhook: production outage and critical heartbeat alerts.
  • Slack webhook: a #vigilmon-data-alerts channel for the data engineering team.

Step 5: TLS Certificate Monitoring

Mage.ai servers are typically deployed behind an Nginx or Traefik reverse proxy that terminates TLS. If the certificate expires, the Mage.ai web interface and API both become unreachable. Add a Vigilmon TLS monitor as a backstop:

In Vigilmon → Add Monitor → SSL/TLS certificate:

  • Host: mage.example.com
  • Port: 443
  • Alert before expiry: 30 days

This provides an external confirmation that the certificate in front of your Mage.ai server is valid and renewing correctly.

Step 6: Public Status Page for Data Platform

Data pipeline availability matters to downstream teams, business stakeholders, and ML model consumers who depend on fresh features and up-to-date model predictions. A public status page gives them transparency without requiring access to Mage.ai's internal run history.

  1. In Vigilmon → Status Pages → Create.
  2. Add your Mage.ai server monitor and critical pipeline heartbeat monitors.
  3. Configure a custom domain (data-status.example.com) or use the Vigilmon subdomain.
  4. Share the status page URL with downstream teams and stakeholders.

What Vigilmon Adds to a Mage.ai Deployment

| Capability | Mage.ai | Vigilmon | |---|---|---| | Pipeline authoring and scheduling | Yes | No | | Block-level dependency graphs | Yes | No | | Built-in data transformations | Yes | No | | Internal run history and logs | Yes | No | | Internal Slack/email notifications | Yes | No | | External synthetic HTTP probing | No | Yes | | Multi-location quorum-based checks | No | Yes | | Pipeline completion heartbeat monitoring | No | Yes | | TLS/SSL certificate expiry alerts | No | Yes | | Public status page | No | Yes | | Independent of scheduler health | No | Yes |

Mage.ai gives your data team a powerful orchestration environment for building and running pipelines. Vigilmon gives you the independent, external signal that confirms those pipelines are completing on schedule and the server that runs them is reachable — without relying on the orchestrator to report its own failures accurately.

Add external uptime monitoring to your Mage.ai deployment today — register free at vigilmon.online.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles