Monitoring Google Cloud Composer with Vigilmon: Airflow Health Checks, DAG Heartbeats & Pipeline Uptime

Google Cloud Composer manages your Apache Airflow environment so you don't have to run the scheduler yourself — but that doesn't mean your pipelines can't fail silently. A DAG that stops scheduling, a Composer environment that enters an unhealthy state, or a task that retries indefinitely without alerting anyone: these are real production failures that Cloud Monitoring won't catch by default. Vigilmon provides an external, independent health check that fires alerts even when the GCP control plane itself has a problem.

This tutorial shows you how to wire Cloud Composer into Vigilmon for environment availability monitoring, DAG heartbeats, and smart alert thresholds.

What You'll Build

• An Airflow health endpoint accessible through Cloud Composer's web server URL
• A Vigilmon HTTP monitor for Composer environment availability
• A DAG that pings a Vigilmon heartbeat on successful completion
• An alerting strategy for data engineering teams

Prerequisites

• A Google Cloud Composer 2 environment (Composer 1 also works with minor adjustments)
• A GCP service account with roles/composer.user for API access
• A free account at vigilmon.online

Step 1: Use the Airflow Health Endpoint

Cloud Composer exposes Airflow's built-in /health endpoint through its web server. This endpoint checks whether the Airflow scheduler, metadatabase, and triggerer are healthy — making it an ideal Vigilmon probe target.

Find your Composer web server URL in the GCP Console → Cloud Composer → your environment → Environment Configuration → Airflow URI.

The health endpoint is:

https://<your-composer-web-server-url>/health

A healthy response looks like:

{
  "metadatabase": {"status": "healthy"},
  "scheduler": {"status": "healthy", "latest_scheduler_heartbeat": "2026-07-06T02:00:00+00:00"},
  "triggerer": {"status": "healthy"},
  "dag_processor": {"status": "healthy"}
}

The endpoint returns HTTP 200 when all components are healthy and a non-200 status when any component is unhealthy.

Note: The Airflow web server requires authentication by default. See Step 2 for how to configure Vigilmon to pass credentials.

Step 2: Configure Vigilmon with Airflow Authentication

Cloud Composer's web server uses Google Cloud IAP (Identity-Aware Proxy) authentication. The easiest way to expose a probe-friendly health endpoint without IAP is to add a small Cloud Function or Cloud Run service that fetches the Airflow health status using a GCP service account.

Option A: Cloud Run health proxy (recommended)

# main.py — deploy as Cloud Run service
import os
import google.auth.transport.requests
import google.oauth2.id_token
import requests
from flask import Flask, jsonify

app = Flask(__name__)

COMPOSER_URL = os.environ["COMPOSER_AIRFLOW_URL"]  # e.g. https://xxxx.composer.googleusercontent.com

def get_iap_token():
    """Get an IAP-scoped ID token for the Composer web server."""
    auth_req = google.auth.transport.requests.Request()
    return google.oauth2.id_token.fetch_id_token(auth_req, COMPOSER_URL)

@app.route("/health")
def health():
    checks = {}
    ok = True

    try:
        token = get_iap_token()
        resp = requests.get(
            f"{COMPOSER_URL}/health",
            headers={"Authorization": f"Bearer {token}"},
            timeout=10,
        )
        resp.raise_for_status()
        data = resp.json()

        scheduler_status = data.get("scheduler", {}).get("status")
        metadb_status = data.get("metadatabase", {}).get("status")
        checks["scheduler"] = scheduler_status
        checks["metadatabase"] = metadb_status

        if scheduler_status != "healthy" or metadb_status != "healthy":
            ok = False

    except Exception as e:
        checks["error"] = str(e)
        ok = False

    return jsonify({"status": "ok" if ok else "degraded", "checks": checks}), (200 if ok else 503)

if __name__ == "__main__":
    app.run(host="0.0.0.0", port=8080)

Deploy to Cloud Run:

gcloud run deploy composer-health \
  --source . \
  --region us-central1 \
  --allow-unauthenticated \
  --set-env-vars COMPOSER_AIRFLOW_URL=https://xxxx.composer.googleusercontent.com

Option B: Public Airflow health endpoint (Composer 2 with no IAP)

If you've disabled IAP, Vigilmon can probe the Airflow health endpoint directly:

1. Log in to Vigilmon → Add Monitor → HTTP.
2. URL: https://<your-composer-web-server-url>/health
3. Check interval: 60 seconds.
4. Response timeout: 15 seconds.
5. Expected status: 200.
6. JSON assertion: path scheduler.status, expected value healthy.

Step 3: Heartbeat DAG for Pipeline Completion

Create a DAG that pings Vigilmon after each successful run. This catches cases where the scheduler stops executing DAGs even though the environment appears healthy.

# dags/vigilmon_heartbeat.py
from datetime import datetime, timedelta
from airflow import DAG
from airflow.operators.python import PythonOperator
from airflow.models import Variable

def ping_heartbeat(**context):
    """POST to Vigilmon heartbeat URL on successful DAG completion."""
    import urllib.request
    heartbeat_url = Variable.get("vigilmon_heartbeat_url")
    req = urllib.request.Request(heartbeat_url, method="POST")
    with urllib.request.urlopen(req, timeout=5) as resp:
        print(f"Heartbeat sent: HTTP {resp.status}")

with DAG(
    dag_id="vigilmon_heartbeat",
    start_date=datetime(2026, 1, 1),
    schedule="*/30 * * * *",  # every 30 minutes
    catchup=False,
    default_args={"retries": 0},
    tags=["monitoring"],
) as dag:
    heartbeat = PythonOperator(
        task_id="ping_vigilmon",
        python_callable=ping_heartbeat,
    )

Store the heartbeat URL as an Airflow Variable via the UI or CLI:

gcloud composer environments run YOUR_ENVIRONMENT \
  --location YOUR_REGION \
  variables set -- vigilmon_heartbeat_url "https://vigilmon.online/api/heartbeat/<your-id>"

For a production pipeline, add the heartbeat task at the end of your real DAG rather than in a separate DAG:

# At the end of your ETL DAG
heartbeat = PythonOperator(
    task_id="vigilmon_heartbeat",
    python_callable=ping_heartbeat,
    trigger_rule="all_success",  # only ping if all upstream tasks succeeded
)

last_task >> heartbeat

In Vigilmon, create a Heartbeat monitor with a grace period that matches your DAG schedule:

| DAG schedule | Recommended grace period | |---|---| | Every 30 minutes | 35 minutes | | Hourly | 75 minutes | | Daily at midnight | 26 hours |

Step 4: Alerting Strategy for Data Teams

| Alert type | Recommended channel | |---|---| | Composer environment unhealthy | PagerDuty or on-call email | | Scheduler heartbeat missed | Slack data-engineering channel | | DAG pipeline heartbeat missed | Slack data-engineering channel | | Status page | Public Vigilmon status page for stakeholders |

Configure the Composer health monitor to re-notify every 5 minutes while the environment remains unhealthy. This is aggressive — Composer environment restarts can take 10–20 minutes — but it ensures the on-call engineer stays aware rather than assuming the alert resolved itself.

What Vigilmon Catches That Cloud Monitoring Misses

| Scenario | Cloud Monitoring | Vigilmon | |---|---|---| | Airflow scheduler stopped | Needs custom metric alert | HTTP monitor + scheduler.status assertion | | DAG stopped scheduling silently | No coverage | Heartbeat grace period fires alert | | Composer environment in error state | Needs uptime check config | HTTP monitor catches immediately | | GCP control plane outage | Alerts may not fire | External check — unaffected | | Airflow web server unreachable | Needs uptime check | HTTP monitor catches immediately |

Composer manages the infrastructure, but it doesn't monitor your pipelines for you. External monitoring from Vigilmon gives your data engineering team an independent signal that catches failures before analysts notice stale data.

Start monitoring your Cloud Composer environment in under 5 minutes — register free at vigilmon.online.