tutorial

Monitoring Meltano with Vigilmon: ELT Pipeline Health, Airflow Webserver Uptime & Job Heartbeats

How to monitor Meltano ELT data pipelines with Vigilmon — pipeline job heartbeats, Airflow webserver uptime, scheduler health, and SSL certificate monitoring.

Meltano is the open-source ELT platform built on the Singer ecosystem — orchestrating data pipelines with 600+ taps and targets, supporting dbt for transformation, and integrating with Airflow and Prefect for scheduling. When a Meltano pipeline fails silently, tables stop refreshing, dashboards show stale data, and downstream teams work from yesterday's numbers without knowing it. When the Airflow webserver is down, engineers can't inspect DAG runs, trigger manual backfills, or debug failing tasks. Vigilmon gives you external visibility into Meltano's operational health: pipeline execution via heartbeats, Airflow webserver uptime, scheduler health, and SSL certificate expiry.

What You'll Build

  • A heartbeat monitor to detect when Meltano pipeline jobs stop running
  • An HTTP monitor on the Airflow webserver to catch UI-level failures
  • An HTTP monitor on Airflow's health endpoint to catch scheduler failures
  • An SSL certificate monitor for your Airflow or Meltano UI domain
  • An alerting setup that separates pipeline execution failures from orchestrator failures

Prerequisites

  • A Meltano project with at least one pipeline (meltano run or scheduled via Airflow)
  • Airflow webserver reachable via HTTPS (if using Meltano's Airflow integration)
  • A free account at vigilmon.online

Step 1: Understand Meltano's Observable Surfaces

Meltano operates as a CLI-first tool; it doesn't expose a native web server. What's observable externally depends on your orchestrator:

| Surface | Observable via | |---|---| | Pipeline job execution | Heartbeat monitoring (job emits ping on completion) | | Airflow webserver (if used) | HTTP uptime monitoring | | Airflow scheduler health | HTTP health endpoint monitoring | | Airflow API | Authenticated HTTP monitoring | | Meltano UI (if self-hosted) | HTTP uptime monitoring | | Custom domain SSL | SSL certificate monitoring |

The most critical monitoring for Meltano is at the pipeline execution level — ensuring jobs run on schedule — rather than at the UI level.

Step 2: Monitor Pipeline Execution with Heartbeats

Meltano pipelines are CLI invocations (meltano run tap-postgres target-snowflake). They don't expose HTTP endpoints. The correct monitoring pattern is a heartbeat: the pipeline emits a signal to Vigilmon on successful completion, and Vigilmon alerts if that signal doesn't arrive within the expected interval.

Add a heartbeat call to your Meltano pipeline wrapper or post-run hook:

Shell script wrapper:

#!/bin/bash
set -e

HEARTBEAT_URL="https://vigilmon.online/api/heartbeat/YOUR_HEARTBEAT_TOKEN"

# Run the Meltano pipeline
meltano run tap-postgres target-snowflake

# Signal success (only reached if meltano run exits 0)
curl -s "$HEARTBEAT_URL" > /dev/null

Python post-run hook (meltano.yml):

schedules:
  - name: postgres-to-snowflake
    interval: "@daily"
    job: postgres-to-snowflake

jobs:
  - name: postgres-to-snowflake
    tasks:
      - tap-postgres target-snowflake

plugins:
  utilities:
    - name: airflow
      pip_url: apache-airflow

For Airflow-orchestrated pipelines, add a success callback to your DAG:

import httpx
from airflow import DAG
from airflow.operators.bash import BashOperator
from airflow.operators.python import PythonOperator
from datetime import datetime

HEARTBEAT_URL = "https://vigilmon.online/api/heartbeat/YOUR_HEARTBEAT_TOKEN"

def ping_heartbeat():
    httpx.get(HEARTBEAT_URL, timeout=5)

with DAG("meltano_postgres_to_snowflake", schedule="@daily", start_date=datetime(2024, 1, 1)) as dag:
    run_pipeline = BashOperator(
        task_id="run_meltano",
        bash_command="meltano run tap-postgres target-snowflake"
    )
    
    notify_success = PythonOperator(
        task_id="heartbeat",
        python_callable=ping_heartbeat,
        trigger_rule="all_success"
    )
    
    run_pipeline >> notify_success

In Vigilmon:

  1. Add Monitor → Heartbeat.
  2. Expected interval: 24 hours (match your pipeline's schedule).
  3. Grace period: 30 minutes.
  4. Label: Meltano postgres→snowflake pipeline.
  5. Copy the heartbeat URL into your wrapper script or DAG.
  6. Click Save.

Create a separate heartbeat monitor for each critical pipeline. A heartbeat per pipeline gives you granular failure attribution — you'll know which pipeline failed, not just that "something is wrong with Meltano."

Step 3: Monitor the Airflow Webserver

If you use Meltano's Airflow integration to schedule DAG runs, the Airflow webserver is the primary operational interface for engineers. Monitor it to detect UI-level failures:

curl https://airflow.example.com
# Returns Airflow login page HTML
  1. Log in to Vigilmon → Add Monitor → HTTP.
  2. URL: https://airflow.example.com.
  3. Check interval: 60 seconds.
  4. Response timeout: 15 seconds.
  5. Expected status: 200.
  6. Keyword: Airflow (present in the login page title).
  7. Label: Airflow webserver.
  8. Click Save.

This monitor catches:

  • Airflow webserver container crashes or pod OOM kills
  • Database connectivity failures blocking webserver startup
  • Ingress or reverse proxy misconfigurations blocking web access
  • Failed Airflow upgrades leaving the webserver in a broken state

Alert sensitivity: Set to trigger after 1 consecutive failure. When the webserver is down, engineers can't inspect DAG run history, trigger manual backfills, or diagnose failing tasks.

Step 4: Monitor the Airflow Health Endpoint

Airflow exposes a /health endpoint that checks both the webserver and the scheduler's liveness:

curl https://airflow.example.com/health
# Returns:
# {
#   "metadatabase": {"status": "healthy"},
#   "scheduler": {"status": "healthy", "latest_scheduler_heartbeat": "..."},
#   "triggerer": {"status": "healthy"},
#   "dag_processor": {"status": "healthy"}
# }
  1. Add Monitor → HTTP.
  2. URL: https://airflow.example.com/health.
  3. Check interval: 2 minutes.
  4. Response timeout: 10 seconds.
  5. Expected status: 200.
  6. Keyword: "healthy" (present in all component status fields when Airflow is fully operational).
  7. Label: Airflow health endpoint.
  8. Click Save.

Why monitor the health endpoint separately from the webserver? The webserver can return 200 on its login page even when the scheduler has died. A dead scheduler means no DAGs are being executed — pipelines stop running silently while the UI appears healthy. The health endpoint catches this.

Step 5: Monitor the Airflow REST API

For teams that trigger Meltano runs via the Airflow REST API (e.g., from CI/CD pipelines or event-driven triggers), monitor the API availability separately:

curl -H "Authorization: Basic <base64(user:password)>" \
  https://airflow.example.com/api/v1/dags
# Returns JSON list of DAGs
  1. Add Monitor → HTTP.
  2. URL: https://airflow.example.com/api/v1/dags.
  3. Method: GET.
  4. Headers: Authorization: Basic <your-base64-credentials>.
  5. Check interval: 2 minutes.
  6. Response timeout: 15 seconds.
  7. Expected status: 200.
  8. Keyword: "dag_id" (present in all DAG list responses).
  9. Label: Airflow REST API.
  10. Click Save.

Step 6: Monitor SSL Certificates

An expired SSL certificate blocks browser access to the Airflow webserver and API calls from automated pipelines:

openssl s_client -connect airflow.example.com:443 2>/dev/null | openssl x509 -noout -dates
  1. Add Monitor → SSL Certificate.
  2. Domain: airflow.example.com.
  3. Alert when expiry is within: 30 days.
  4. Alert again: 14 days, 7 days, 3 days, 1 day.
  5. Click Save.

Step 7: Configure Alerting

In Vigilmon under Settings → Notifications, configure your alert channels:

| Monitor | Trigger | Action | |---|---|---| | Pipeline heartbeat | Silence beyond interval + grace | Check Airflow DAG run history; inspect Meltano run logs; verify tap/target connectivity | | Airflow webserver | Non-200 or Airflow missing | Check Airflow pod logs; verify Postgres metadata DB connectivity; inspect ingress | | Airflow health endpoint | Non-200 or healthy missing | Scheduler may be dead; check scheduler pod; verify Airflow metadata DB; inspect celery workers | | Airflow REST API | Non-200 or keyword missing | API degraded; CI/CD triggers will fail; check auth config and API enablement | | SSL certificate | < 30 days to expiry | Renew certificate; verify Airflow webserver access after renewal |

Alert after: 1 consecutive failure for the Airflow webserver and health monitors. 1 missed heartbeat for pipeline monitors.

Common Meltano Failure Modes and What Vigilmon Catches

| Scenario | Vigilmon monitor | |---|---| | Meltano tap fails to connect to source database | Heartbeat not emitted; heartbeat monitor fires after interval | | Target (e.g., Snowflake) credentials expired | Pipeline exits non-zero; heartbeat not emitted; monitor fires | | Airflow scheduler process dies | Health endpoint returns unhealthy for scheduler; health monitor fires | | Airflow webserver container OOM killed | Webserver monitor fires; health monitor also fires | | DAG paused accidentally in Airflow UI | No runs triggered; heartbeat silent; monitor fires after interval | | dbt transformation fails mid-pipeline | Pipeline exits non-zero; heartbeat not emitted; monitor fires | | SSL certificate expires on Airflow domain | All HTTPS monitors fire simultaneously | | Singer tap breaks on source schema change | Pipeline exits non-zero; heartbeat not emitted; check tap logs for schema mismatch | | Meltano upgrade breaks plugin compatibility | First run fails; heartbeat not emitted; monitor fires | | Celery worker pool exhausted | DAG tasks queue but don't run; heartbeat delayed; monitor fires after grace period |

Monitoring Data Quality

Vigilmon monitors Meltano's operational health — whether pipelines run and orchestrators are available — not the quality or completeness of the data they produce. For data quality monitoring:

  • Row count checks: After each pipeline run, query the target table for row counts and compare to expected ranges. Alert if counts fall outside expected bounds (e.g., daily orders table has fewer than 100 rows when it usually has 10,000).
  • Freshness checks: Use dbt's freshness tests or a custom query to verify that the updated_at column in key tables is within the expected freshness window.
  • Schema validation: Use dbt's schema.yml tests to validate column types, nullability, and value constraints after each transformation run.
  • dbt test results: Have your CI/CD or Airflow DAG post dbt test results to a monitoring dashboard. Alert on any failing test in the critical path.

Vigilmon catches operational failures (pipeline didn't run, orchestrator is down). Data quality monitoring requires dbt tests and application-level checks on your target data.

Meltano and its Singer-ecosystem pipelines are the foundation of reproducible data infrastructure — but "reproducible" only holds when pipelines actually run on schedule. Vigilmon gives you external visibility into the operational health of your Meltano setup: per-pipeline heartbeats that fire when jobs miss their schedule, Airflow webserver and scheduler health, REST API availability, and SSL certificate expiry, so data engineers know immediately when a pipeline stalls rather than discovering stale dashboards at the morning standup.

Start monitoring Meltano in under 5 minutes — register free at vigilmon.online.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles