tutorial

Monitoring Apache Airflow with Vigilmon: Webserver Health, DAG Scheduler, Worker Availability & SSL Alerts

How to monitor Apache Airflow with Vigilmon — webserver uptime, scheduler health, Celery worker availability, TCP port checks, and SSL certificate alerts for your data pipelines.

Apache Airflow is the backbone of data pipeline operations at thousands of companies — orchestrating ETL jobs, ML training runs, dbt transformations, and batch processing through a dependency graph of scheduled tasks. When Airflow goes down, pipelines stop silently: DAGs miss their schedule, downstream jobs block on missing data, and SLAs breach before anyone notices. When the scheduler crashes, DAG runs queue up but never execute. When a worker pool shrinks, tasks pile up in the queue indefinitely.

Vigilmon gives you external visibility into Airflow before your data team notices: webserver availability, scheduler liveness, Celery flower worker dashboard, and SSL certificate expiry.

What You'll Build

  • An HTTP monitor on the Airflow webserver health endpoint
  • A monitor on the Airflow login page for UI availability
  • A TCP port check for the Airflow scheduler's health port
  • A monitor on Celery Flower for worker pool visibility
  • SSL certificate monitoring for your Airflow domain
  • Alerting that distinguishes UI failures from scheduler failures

Prerequisites

  • A running Apache Airflow installation (2.x) accessible over HTTPS
  • A domain pointing to Airflow (e.g., https://airflow.example.com)
  • A free account at vigilmon.online

Why Monitor Apache Airflow?

Airflow has several independent processes that can fail independently:

  • Webserver — the Flask-based UI and REST API; if it crashes, no one can trigger DAGs, inspect logs, or view pipeline status
  • Scheduler — the brain of Airflow; if it crashes, no DAG runs are created, tasks never queue, and pipelines silently stop
  • Workers — Celery or Kubernetes executors that actually run tasks; if workers die, queued tasks wait indefinitely
  • Metadata database — if the PostgreSQL/MySQL database is unreachable, every Airflow component fails

External monitoring from Vigilmon catches failures that Airflow's own alerting system can't: if Airflow itself is down, it can't send an alert. Vigilmon checks from outside your infrastructure and alerts your team even if every Airflow process is dead.

Key Metrics to Monitor

| Metric | What it reveals | |---|---| | Webserver HTTP response | UI and API availability | | /health endpoint status | Scheduler and metadatabase connectivity | | Scheduler heartbeat via API | Whether DAG runs are being created | | Celery Flower availability | Worker pool health | | TCP port reachability | Network-level connectivity | | SSL certificate expiry | Webhook and API client connectivity |

Step 1: Verify Airflow's Health Endpoint

Airflow 2.x ships with a built-in health endpoint at /health:

curl https://airflow.example.com/health

A healthy response looks like:

{
  "metadatabase": {"status": "healthy"},
  "scheduler": {"status": "healthy", "latest_scheduler_heartbeat": "2024-01-15T10:30:00+00:00"}
}

This endpoint is unauthenticated by default and designed for load balancers and monitoring tools. It checks both the metadatabase connection and the scheduler's last heartbeat — a stale heartbeat means the scheduler has crashed even if the webserver is still responding.

# Check health endpoint directly
curl -s https://airflow.example.com/health | python3 -m json.tool

# Verify login page
curl -I https://airflow.example.com/login

Authentication note: If you have auth_backend = airflow.api.auth.backend.deny_all or similar, the /health endpoint may still be reachable — it's exempt from API authentication in Airflow 2.x. Verify with curl -I before setting up your Vigilmon monitor.

Step 2: Create an HTTP Monitor for the Health Endpoint

  1. Log in to Vigilmon → Add Monitor → HTTP.
  2. URL: https://airflow.example.com/health.
  3. Check interval: 60 seconds.
  4. Response timeout: 15 seconds.
  5. Expected status: 200.
  6. Keyword: healthy (appears in both the metadatabase and scheduler fields of a healthy response).
  7. Label: Airflow health endpoint.
  8. Click Save.

This monitor catches:

  • Airflow webserver process crashes
  • Metadata database connection failures
  • Scheduler crashes (the scheduler field shows unhealthy before going fully down)
  • Airflow startup failures after deployments

Alert sensitivity: Trigger after 2 consecutive failures. A single failure can be a transient network blip; two consecutive failures indicate a real issue.

Step 3: Create an HTTP Monitor for the Webserver Login Page

The login page is the most user-facing check — if it fails, no operator can manually trigger or inspect DAGs:

  1. Add Monitor → HTTP.
  2. URL: https://airflow.example.com/login.
  3. Check interval: 2 minutes.
  4. Response timeout: 15 seconds.
  5. Expected status: 200.
  6. Keyword: Airflow (appears in the page title of every Airflow login page).
  7. Label: Airflow webserver login.
  8. Click Save.

When this monitor fires but the /health endpoint still returns 200, you have a routing or reverse proxy issue — the health endpoint is lightweight and may pass while the full UI fails.

Step 4: Create a TCP Monitor for the Webserver Port

The TCP monitor catches network-level failures before HTTP-level checks fire. Airflow's webserver typically runs on port 8080 (or 443 behind a reverse proxy):

# Verify the port is reachable
nc -zv airflow.example.com 8080
# Or if behind HTTPS reverse proxy
nc -zv airflow.example.com 443
  1. Add Monitor → TCP.
  2. Host: airflow.example.com.
  3. Port: 8080 (or 443 if you terminate TLS at a reverse proxy).
  4. Check interval: 2 minutes.
  5. Response timeout: 10 seconds.
  6. Label: Airflow webserver TCP.
  7. Click Save.

When the TCP monitor fires but previously HTTP monitors were green, you have a new firewall rule or network routing change.

Step 5: Monitor Celery Flower for Worker Health

If you run the Celery executor, Flower provides a dashboard of active workers at port 5555. A dead Flower means you can't inspect worker status, but more importantly, a Flower that shows zero workers means tasks won't execute:

# Verify Flower is accessible
curl -I https://airflow.example.com/flower
# Or if Flower runs on a separate port
curl -I http://airflow.example.com:5555
  1. Add Monitor → HTTP.
  2. URL: https://airflow.example.com/flower (or http://airflow.example.com:5555 if direct).
  3. Check interval: 2 minutes.
  4. Response timeout: 10 seconds.
  5. Expected status: 200.
  6. Keyword: Flower (appears in Celery Flower's page title).
  7. Label: Airflow Celery Flower.
  8. Click Save.

KubernetesExecutor users: If you use the KubernetesExecutor instead of Celery, skip the Flower monitor — you don't run a Celery broker. Instead, consider monitoring your Kubernetes API server health if Airflow's executor depends on it.

Step 6: Monitor SSL Certificates

Airflow's REST API is used by dbt Cloud, data orchestration tools, and custom pipeline triggers. An expired certificate causes:

  • All external API clients to fail with TLS errors
  • Webhook triggers from orchestration tools to stop
  • Browser warnings that block operators from accessing the UI
  1. Add Monitor → SSL Certificate.
  2. Domain: airflow.example.com.
  3. Alert when expiry is within: 30 days.
  4. Alert again: 14 days, 7 days, 3 days, 1 day.
  5. Click Save.

MWAA and Cloud Composer users: Managed Airflow services handle TLS automatically. If you use AWS MWAA or Google Cloud Composer, focus your SSL monitoring on any custom domains you've configured rather than the managed endpoint.

Step 7: Configure Alerting

In Vigilmon under Settings → Notifications, configure your alert channels:

| Monitor | Trigger | Runbook action | |---|---|---| | Health endpoint | Non-200 or healthy missing | Check airflow health; inspect scheduler logs; verify DB connection | | Login page | Non-200 or Airflow missing | Check airflow webserver process; inspect gunicorn/nginx logs | | TCP port | Connection refused or timeout | Check firewall rules; verify webserver process is bound to the port | | Celery Flower | Non-200 or timeout | Check celery -A airflow.executors.celery_executor worker processes; inspect broker connectivity | | SSL certificate | < 30 days to expiry | Renew certificate; check certbot/ACME renewal automation |

Recommended thresholds:

  • HTTP monitors: alert after 2 consecutive failures
  • TCP monitors: alert after 1 failure (TCP timeouts are rarely transient)
  • SSL certificates: alert at 30 days, escalate at 7 days

Common Airflow Failure Modes and What Vigilmon Catches

| Scenario | Vigilmon monitor that fires | |---|---| | Webserver process crash | Login page unreachable; health endpoint fails; alert within 60 s | | Scheduler crash | Health endpoint shows scheduler unhealthy; DAGs stop running | | Celery broker (Redis/RabbitMQ) unreachable | Flower shows zero workers; tasks queue indefinitely | | Metadata DB connection failure | Health endpoint shows metadatabase unhealthy | | Airflow upgrade failure | Login page returns 500; health endpoint fails | | Disk full on scheduler host | Scheduler stops creating DAG runs; health heartbeat goes stale | | Firewall change blocks webserver port | TCP monitor fires; may precede HTTP monitor failure | | SSL certificate expires | SSL monitor fires at 30-day threshold; API clients break | | Reverse proxy misconfiguration | Login and health monitors return 502/503; TCP port may still respond | | DNS misconfiguration | All monitors fire simultaneously |

Monitoring Airflow in Production: Architecture Notes

For production Airflow deployments, consider layering your monitoring:

  1. Vigilmon external checks — catches outages visible to your users and API clients; fires even if Airflow's internal alerting is broken
  2. Airflow SLA miss callbacks — fire when individual DAG runs miss their SLA; configure sla_miss_callback in your DAG definitions
  3. Scheduler logs and metrics — Airflow exports StatsD metrics; use these for scheduler queue depth and task duration trends
  4. Worker resource monitoring — monitor CPU, memory, and disk on Celery worker machines separately

Vigilmon covers the first layer: independent external visibility that can alert you when Airflow itself is too broken to send its own alerts.

Pipeline downtime is silent by nature — missed DAG runs don't produce errors, they just produce absence. By the time a data consumer notices missing data, the pipeline may have been broken for hours. Vigilmon gives you external uptime monitoring that catches Airflow failures in seconds, before they cascade into downstream SLA breaches.

Start monitoring Airflow in under 5 minutes — register free at vigilmon.online.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles