tutorial

Monitoring Prefect: Orion Server, Flow Runs, Agent Heartbeats, and Work Queue Depth

"A practical guide to monitoring Prefect in production — Orion/API server health, flow run failure rates, agent heartbeats, work queue depth, and integrating Vigilmon for external health checks."

Monitoring Prefect in Production

Prefect is a modern workflow orchestration platform used for data pipelines, ML training jobs, and ETL workloads. In production, Prefect has multiple failure surfaces: the Prefect API/Orion server, the agents that execute flow runs, the work queues that route work to agents, and the individual flow runs themselves. This guide covers monitoring each layer, from basic uptime to nuanced queue-depth alerting, with Vigilmon integration for external checks.

Prefect Architecture: What Can Fail

A typical self-hosted Prefect deployment has:

  • Prefect API (Orion) — the central orchestration server; agents poll this to receive work
  • Prefect agents — worker processes that pull flows from work queues and execute them
  • Work queues — virtual queues that route flow runs to agents by type, priority, or tag
  • Flow runs — the actual executions; these can fail, crash, or get stuck in "pending" state

When Prefect fails silently, it usually looks like: flows stop running, but no one gets alerted because the failure happened before the flow code even started.

1. Prefect API Server Health

The Prefect API exposes a health endpoint:

curl -s http://localhost:4200/api/health
# {"status":"healthy"}

For Prefect Cloud or a remote Prefect server:

curl -s https://api.prefect.cloud/api/health

A 200 response confirms the API is available. But health alone doesn't tell you if agents can reach the server or if the database backend is functional.

Database connectivity check

# Test the full API stack by creating and immediately canceling a flow run
curl -s -X POST http://localhost:4200/api/flow-runs \
  -H "Content-Type: application/json" \
  -d '{"name": "health-probe", "flow_id": "00000000-0000-0000-0000-000000000000"}'

A 422 (Unprocessable Entity) response with a validation error means the API is healthy — you'd get a 503 or connection error if the server were down.

Python health probe

import httpx
import sys

def prefect_api_healthy(api_url: str = "http://localhost:4200") -> bool:
    try:
        resp = httpx.get(f"{api_url}/api/health", timeout=5)
        return resp.status_code == 200 and resp.json().get("status") == "healthy"
    except Exception as e:
        print(f"Prefect API unreachable: {e}", file=sys.stderr)
        return False

if __name__ == "__main__":
    sys.exit(0 if prefect_api_healthy() else 1)

2. Agent Heartbeat Monitoring

Prefect agents send heartbeats to the Orion server. If an agent dies, it stops sending heartbeats — and flows that were routed to it get stuck in "pending" forever.

Query the agents API to check last heartbeat time:

import httpx
from datetime import datetime, timezone, timedelta

def check_agent_heartbeats(api_url: str, max_stale_minutes: int = 5) -> list[str]:
    """Returns list of agent names that haven't heartbeat recently."""
    resp = httpx.get(f"{api_url}/api/workers/filter", timeout=10)
    workers = resp.json()
    stale = []
    now = datetime.now(timezone.utc)
    for worker in workers:
        last_heartbeat = datetime.fromisoformat(worker["last_heartbeat_time"].replace("Z", "+00:00"))
        age_minutes = (now - last_heartbeat).total_seconds() / 60
        if age_minutes > max_stale_minutes:
            stale.append(f"{worker['name']} (last heartbeat {age_minutes:.1f}m ago)")
    return stale

stale_agents = check_agent_heartbeats("http://localhost:4200")
if stale_agents:
    print(f"ALERT: Stale agents: {', '.join(stale_agents)}")

Wrap this in a /agent-health HTTP endpoint and expose it for external monitoring.

For Prefect 2.x using work pools:

# List work pools and their status
prefect work-pool ls

# Check specific pool status
prefect work-pool inspect my-pool

3. Work Queue Depth Monitoring

Work queue depth is the canary for agent capacity problems. If depth grows steadily, your agents are falling behind — either because they're down, under-resourced, or the flow run concurrency limit is too low.

import httpx
import sys

def get_work_queue_depths(api_url: str) -> dict[str, int]:
    """Return {queue_name: pending_count}."""
    resp = httpx.get(f"{api_url}/api/work_queues/filter", timeout=10)
    queues = resp.json()
    return {q["name"]: q.get("late_runs_count", 0) for q in queues}

depths = get_work_queue_depths("http://localhost:4200")
for queue, depth in depths.items():
    if depth > 10:  # alert threshold
        print(f"WARNING: Work queue '{queue}' has {depth} late runs")

For Prefect 2.x with work pools, check scheduled flow runs that are late:

# Via the CLI
prefect flow-run ls --state LATE --limit 50

# Count via API
curl -s -X POST http://localhost:4200/api/flow_runs/filter \
  -H "Content-Type: application/json" \
  -d '{"flow_runs": {"state": {"type": {"any_": ["LATE"]}}}}'

Alert if this count grows beyond your normal baseline.

4. Flow Run Failure Rate

Flow run failures are expected in data pipelines — transient network errors, upstream data issues, external API timeouts. What you want to catch is an elevated failure rate, not individual failures.

import httpx
from datetime import datetime, timezone, timedelta

def get_failure_rate(api_url: str, window_minutes: int = 60) -> float:
    """Returns failure rate (0.0–1.0) over the last N minutes."""
    since = (datetime.now(timezone.utc) - timedelta(minutes=window_minutes)).isoformat()
    
    # Count completed runs
    completed_resp = httpx.post(
        f"{api_url}/api/flow_runs/filter",
        json={"flow_runs": {"expected_start_time": {"after_": since},
                            "state": {"type": {"any_": ["COMPLETED", "FAILED", "CRASHED"]}}}},
        timeout=10
    )
    runs = completed_resp.json()
    
    if not runs:
        return 0.0
    
    failed = sum(1 for r in runs if r["state"]["type"] in ("FAILED", "CRASHED"))
    return failed / len(runs)

rate = get_failure_rate("http://localhost:4200", window_minutes=60)
if rate > 0.2:  # 20% failure rate threshold
    print(f"ALERT: Flow run failure rate is {rate:.0%} over last hour")

5. Detecting Stuck Flow Runs

Flow runs in "Running" state for longer than expected are often stuck — the agent process died while the run was in flight, leaving the state as "Running" forever.

import httpx
from datetime import datetime, timezone, timedelta

def find_stuck_runs(api_url: str, running_for_hours: int = 2) -> list[dict]:
    """Find flow runs stuck in Running state longer than expected."""
    cutoff = (datetime.now(timezone.utc) - timedelta(hours=running_for_hours)).isoformat()
    
    resp = httpx.post(
        f"{api_url}/api/flow_runs/filter",
        json={"flow_runs": {
            "state": {"type": {"any_": ["RUNNING"]}},
            "start_time": {"before_": cutoff}
        }},
        timeout=10
    )
    return resp.json()

stuck = find_stuck_runs("http://localhost:4200", running_for_hours=4)
for run in stuck:
    print(f"Stuck run: {run['name']} (id: {run['id']}, started: {run['start_time']})")

Schedule this check every 15 minutes and alert if any runs are found.

6. Integrating Vigilmon for External Health Checks

Internal monitoring catches code-level failures. Vigilmon adds external network-level checks — essential for catching DNS failures, load balancer issues, and full server outages.

Step 1: Add the Prefect API monitor

In Vigilmon, create an HTTP monitor:

  • URL: https://prefect.yourdomain.com/api/health
  • Method: GET
  • Expected status: 200
  • Expected body contains: healthy
  • Check interval: 60 seconds
  • Alert after: 2 consecutive failures

Step 2: Monitor the Prefect UI

The Prefect Orion UI at port 4200 serves a separate static asset path. Monitor it independently:

  • URL: https://prefect.yourdomain.com/
  • Method: GET
  • Expected status: 200
  • Check interval: 5 minutes

Step 3: Agent health endpoint

Expose the Python agent heartbeat checker from Step 2 as an HTTP endpoint:

from fastapi import FastAPI, Response
import httpx
from datetime import datetime, timezone, timedelta

app = FastAPI()

@app.get("/agent-health")
def agent_health():
    stale = check_agent_heartbeats("http://localhost:4200", max_stale_minutes=5)
    if stale:
        return Response(
            content=f"Stale agents: {', '.join(stale)}",
            status_code=503
        )
    return {"status": "ok"}

Add a Vigilmon monitor for this endpoint:

  • URL: https://prefect.yourdomain.com/agent-health
  • Method: GET
  • Expected status: 200
  • Check interval: 2 minutes

Step 4: Work queue depth via Vigilmon response check

monitors:
  - name: "Prefect API"
    url: https://prefect.yourdomain.com/api/health
    interval: 60
    alerts:
      - type: pagerduty
        after_failures: 2

  - name: "Prefect Agents"
    url: https://prefect.yourdomain.com/agent-health
    interval: 120
    alerts:
      - type: slack
        after_failures: 2

  - name: "Prefect UI"
    url: https://prefect.yourdomain.com/
    interval: 300
    alerts:
      - type: email
        after_failures: 3

Alert Severity Tiers

| Condition | Severity | Action | |---|---|---| | Prefect API unreachable | Critical | Page on-call immediately | | Agent heartbeat stale >5m | High | Slack alert + auto-restart | | Work queue depth >50 late | High | Slack alert | | Flow run failure rate >20% | Medium | Email alert | | Stuck runs >4 hours | Medium | Slack alert | | UI unreachable | Low | Email alert |

Putting It Together

A well-monitored Prefect deployment combines:

  1. Vigilmon external checks on the Prefect API and UI for network-level outage detection
  2. Agent heartbeat monitoring via a custom /agent-health endpoint checked every 2 minutes
  3. Work queue depth watched by a Python cron job with Slack alerts at configurable thresholds
  4. Flow run failure rate computed hourly and compared against your baseline
  5. Stuck run detection running every 15 minutes to catch orphaned executions

With this stack, you'll know about Prefect problems before your data pipeline SLAs are breached — and you'll have enough context to know exactly which layer to fix first.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles