Monitoring Luigi Pipelines with Vigilmon

Luigi is Spotify's Python framework for building complex data pipeline dependency graphs. It handles task dependency resolution, retries, and a central scheduler that tracks which tasks have run. But Luigi's central scheduler has no built-in uptime alerting — if the scheduler process crashes, your pipelines silently stop executing, and you won't know until someone notices stale data. Vigilmon monitors the Luigi scheduler web UI, the task list API endpoint, task history, and pipeline heartbeats so you catch failures the moment they happen.

What You'll Set Up

• HTTP monitor for the Luigi central scheduler web UI (port 8082)
• HTTP monitor for the /api/task_list endpoint
• HTTP monitor for the task history API
• Heartbeat monitoring for scheduled Luigi pipelines
• SSL certificate alerts for deployed Luigi scheduler instances

Prerequisites

• Luigi installed and the central scheduler (luigid) running
• A Vigilmon free account
• Python 3.8+

Step 1: Monitor the Luigi Central Scheduler Web UI

Luigi's luigid daemon runs a web UI on port 8082 by default. This UI shows task dependency graphs, running tasks, and failure history. If the scheduler is down, no new tasks can be registered or coordinated.

1. Log in to vigilmon.online and click Add Monitor.
2. Set Type to HTTP / HTTPS.
3. Enter the scheduler URL: http://your-luigi-host:8082/
4. Set Check interval to 1 minute.
5. Set Expected HTTP status to 200.
6. Click Save.

Verify the scheduler is running and accessible:

curl http://your-luigi-host:8082/
# Should return the Luigi web UI HTML

If you've configured luigid behind a reverse proxy with a domain, use the proxied URL instead:

curl https://luigi.yourdomain.com/

A scheduler failure means tasks submitted with --scheduler-host in their invocation will fail immediately with a connection error, halting all coordinated pipeline execution.

Step 2: Monitor the /api/task_list Endpoint

Luigi's scheduler exposes a REST API. The /api/task_list endpoint returns the current state of all known tasks — pending, running, done, and failed. This is both useful for monitoring and a good health signal: if the API stops responding, the scheduler's internal state tracking has broken.

1. Click Add Monitor → HTTP / HTTPS.
2. Enter: http://your-luigi-host:8082/api/task_list
3. Set Check interval to 2 minutes.
4. Set Expected HTTP status to 200.
5. Click Save.

You can inspect the endpoint manually:

curl "http://your-luigi-host:8082/api/task_list?status=FAILED&upstream_status=&limit=100"
# Returns JSON: {"response": {"FAILED": {"TaskName": {...}, ...}}}

This endpoint is also useful for building your own alerting on top of Vigilmon — if the task list consistently shows tasks in FAILED state, you want to know.

Step 3: Monitor the Task History API

Luigi's task history API (requires record_task_history = True in luigi.cfg) tracks historical task runs. Monitor it as a secondary health signal:

1. Click Add Monitor → HTTP / HTTPS.
2. Enter: http://your-luigi-host:8082/api/task_history
3. Set Check interval to 5 minutes.
4. Set Expected HTTP status to 200.
5. Click Save.

Enable task history in your luigi.cfg:

[scheduler]
record_task_history = True

[task_history]
db_connection = sqlite:////var/lib/luigi/task_history.db

The task history database grows over time; add a housekeeping entry to your crontab to prune old records if you're using SQLite:

# Keep 90 days of task history
find /var/lib/luigi/ -name "*.db" -exec sqlite3 {} \
  "DELETE FROM task_events WHERE ts < datetime('now', '-90 days');" \;

Step 4: Heartbeat Monitoring for Scheduled Luigi Pipelines

Luigi tasks that run on a schedule (via cron, Airflow handoff, or a custom scheduler) will silently miss their runs if the trigger fails or a task crashes mid-dependency-chain. Use Vigilmon heartbeats to confirm each pipeline completes successfully.

1. In Vigilmon, click Add Monitor → Cron Heartbeat.
2. Set the expected ping interval to match your pipeline schedule (e.g. 60 minutes for hourly pipelines, 1440 minutes for daily).
3. Copy the heartbeat URL (e.g. https://vigilmon.online/heartbeat/abc123).

Add the heartbeat ping to the end of your Luigi pipeline entry point:

import luigi
import requests

class FinalTask(luigi.Task):
    """Terminal task in the pipeline DAG."""

    def requires(self):
        return [TransformTask(), LoadTask()]

    def run(self):
        # Signal pipeline completion to Vigilmon
        try:
            requests.get(
                'https://vigilmon.online/heartbeat/abc123',
                timeout=5
            )
        except requests.RequestException:
            pass  # Don't fail the pipeline if the heartbeat call fails

    def output(self):
        return luigi.LocalTarget('/tmp/pipeline-complete.flag')


if __name__ == '__main__':
    luigi.build([FinalTask()], workers=4, local_scheduler=False)

For pipelines triggered by cron, wrap the Luigi invocation in a shell script:

#!/bin/bash
# /usr/local/bin/run-daily-pipeline.sh

LUIGI_HOST=your-luigi-host
PIPELINE_MODULE=your_pipeline

python -m luigi --module $PIPELINE_MODULE FinalTask \
    --scheduler-host $LUIGI_HOST \
    --scheduler-port 8082

if [ $? -eq 0 ]; then
    curl -s --max-time 10 https://vigilmon.online/heartbeat/abc123
    echo "Pipeline completed successfully at $(date)"
else
    echo "Pipeline FAILED at $(date)" >&2
    exit 1
fi

Add to crontab:

0 6 * * * /usr/local/bin/run-daily-pipeline.sh >> /var/log/luigi-pipeline.log 2>&1

If any task in the dependency graph fails, Luigi returns a non-zero exit code, the if block is skipped, and Vigilmon alerts after the expected interval passes without a ping.

Step 5: SSL Certificate Alerts for Deployed Scheduler Instances

If your Luigi scheduler is exposed over HTTPS (via nginx or another reverse proxy), add SSL certificate monitoring:

1. Open the HTTP / HTTPS monitor for your scheduler (created in Step 1).
2. Enable Monitor SSL certificate.
3. Set Alert when certificate expires in less than 21 days.
4. Click Save.

Example nginx configuration for SSL-terminated Luigi scheduler:

server {
    listen 443 ssl;
    server_name luigi.yourdomain.com;

    ssl_certificate /etc/letsencrypt/live/luigi.yourdomain.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/luigi.yourdomain.com/privkey.pem;

    location / {
        proxy_pass http://127.0.0.1:8082;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
    }
}

Check the certificate expiry:

openssl s_client -connect luigi.yourdomain.com:443 -showcerts </dev/null 2>/dev/null | \
  openssl x509 -noout -enddate
# notAfter=Mar  1 00:00:00 2026 GMT

Step 6: Configure Alert Channels

1. Go to Alert Channels in Vigilmon and add email, Slack, or a webhook.
2. For the scheduler web UI and task list API monitors, set Consecutive failures before alert to 2 — the Luigi scheduler can briefly miss requests during GC pauses under heavy load.
3. For heartbeat monitors, no grace period is needed: a missed heartbeat means a missed pipeline run.

Summary

| Monitor | Target | What It Catches | |---|---|---| | HTTP (scheduler UI) | http://host:8082/ | luigid process crash, scheduler down | | HTTP (task list API) | /api/task_list | Scheduler API failure, state corruption | | HTTP (task history) | /api/task_history | History DB failure | | Cron heartbeat | Heartbeat URL | Failed or skipped pipeline run | | SSL certificate | Scheduler domain cert | Certificate expiry before renewal |

Luigi gives you battle-tested DAG-based pipeline orchestration with minimal operational overhead — but "minimal" doesn't mean "zero." With Vigilmon watching the scheduler process, the task API endpoints, and your pipeline heartbeats, you'll know the moment a task silently fails instead of finding out from stale dashboards the next morning.