tutorial

Monitoring Dagster: Dagit UI, Daemon Health, Asset Materialization, and Pipeline SLAs

"A practical guide to monitoring Dagster in production — Dagit availability, daemon process health, asset materialization lag, run queue monitoring, and Vigilmon integration for pipeline SLAs."

Monitoring Dagster in Production

Dagster's asset-based pipeline model is powerful, but monitoring it in production requires understanding several distinct processes: the Dagit web UI, the Dagster daemon that handles scheduling and sensors, the gRPC code server, and the run launcher that executes jobs. When any of these fail silently, your data assets go stale — and downstream consumers often don't find out until they're looking at bad numbers in a dashboard. This guide walks through monitoring every Dagster component and setting up Vigilmon for external SLA checks.

Dagster's Process Architecture

A production Dagster deployment has these moving parts:

  • Dagit (or dagster-webserver) — the UI and GraphQL API; operators and data engineers use this to inspect runs, assets, and schedules
  • Dagster daemon — a background process that handles schedule ticking, sensor evaluation, backfill coordination, and run monitoring
  • gRPC code server — serves your pipeline definitions; separate from Dagit so code changes don't restart the web server
  • Run launcher — executes jobs via Docker, Kubernetes, or subprocess; failures here leave runs stuck in "STARTING" state

Each component can fail independently, and Dagster's default behavior when a component is unhealthy is often to log a warning rather than surface a visible error.

1. Dagit / Webserver Health

Dagster's webserver exposes a health endpoint:

curl -s http://localhost:3000/dagit_info
# {"dagit_version": "1.x.x", ...}

For a more authoritative health check, query the GraphQL API:

curl -s -X POST http://localhost:3000/graphql \
  -H "Content-Type: application/json" \
  -d '{"query": "{ version }"}'
# {"data": {"version": "1.x.x"}}

A valid version response confirms the webserver and its GraphQL layer are functional.

Liveness vs readiness

Wrap these in a lightweight endpoint for your load balancer:

import httpx
from fastapi import FastAPI, Response

app = FastAPI()

@app.get("/health")
def health():
    try:
        resp = httpx.post(
            "http://localhost:3000/graphql",
            json={"query": "{ version }"},
            timeout=5
        )
        if resp.status_code == 200 and "version" in resp.json().get("data", {}):
            return {"status": "ok"}
    except Exception:
        pass
    return Response(content="unhealthy", status_code=503)

2. Daemon Health

The Dagster daemon is the most critical process you're least likely to monitor. It runs independently of Dagit, and if it crashes, schedules stop ticking, sensors stop firing, and backfills hang — all silently.

Dagster exposes daemon health via the GraphQL API:

curl -s -X POST http://localhost:3000/graphql \
  -H "Content-Type: application/json" \
  -d '{"query": "{ instance { daemonHealth { allDaemonStatuses { daemonType healthy id lastHeartbeatTime required } } } }"}'

Parse the response in Python to build a health check endpoint:

import httpx
import sys
from datetime import datetime, timezone, timedelta

GRAPHQL_URL = "http://localhost:3000/graphql"
DAEMON_HEALTH_QUERY = """
{
  instance {
    daemonHealth {
      allDaemonStatuses {
        daemonType
        healthy
        required
        lastHeartbeatTime
        lastHeartbeatErrors {
          message
        }
      }
    }
  }
}
"""

def check_daemon_health() -> tuple[bool, list[str]]:
    resp = httpx.post(GRAPHQL_URL, json={"query": DAEMON_HEALTH_QUERY}, timeout=10)
    data = resp.json()
    statuses = data["data"]["instance"]["daemonHealth"]["allDaemonStatuses"]
    
    issues = []
    for daemon in statuses:
        if daemon["required"] and not daemon["healthy"]:
            errors = [e["message"] for e in (daemon["lastHeartbeatErrors"] or [])]
            issues.append(f"{daemon['daemonType']}: {'; '.join(errors) or 'unhealthy'}")
    
    return len(issues) == 0, issues

healthy, issues = check_daemon_health()
if not healthy:
    print(f"ALERT: Daemon issues: {', '.join(issues)}")
    sys.exit(1)

Expose this as a /daemon-health endpoint for Vigilmon to probe.

3. Asset Materialization Lag

Asset-based pipelines are where Dagster shines — and where silent failures hurt the most. An asset that hasn't been materialized in 24 hours when it should update hourly is the definition of data staleness.

Query asset materialization recency via GraphQL:

import httpx
from datetime import datetime, timezone, timedelta

ASSET_FRESHNESS_QUERY = """
{
  assetsOrError {
    ... on AssetConnection {
      nodes {
        key {
          path
        }
        assetMaterializations(limit: 1) {
          timestamp
          runId
        }
      }
    }
  }
}
"""

def find_stale_assets(max_age_hours: int = 25) -> list[dict]:
    resp = httpx.post(
        "http://localhost:3000/graphql",
        json={"query": ASSET_FRESHNESS_QUERY},
        timeout=30
    )
    data = resp.json()
    assets = data["data"]["assetsOrError"]["nodes"]
    
    stale = []
    cutoff = datetime.now(timezone.utc) - timedelta(hours=max_age_hours)
    
    for asset in assets:
        materializations = asset.get("assetMaterializations", [])
        if not materializations:
            stale.append({"key": asset["key"]["path"], "last_materialized": "never"})
            continue
        
        last_ts = float(materializations[0]["timestamp"]) / 1000
        last_dt = datetime.fromtimestamp(last_ts, tz=timezone.utc)
        
        if last_dt < cutoff:
            stale.append({
                "key": asset["key"]["path"],
                "last_materialized": last_dt.isoformat(),
                "age_hours": (datetime.now(timezone.utc) - last_dt).total_seconds() / 3600
            })
    
    return stale

stale = find_stale_assets(max_age_hours=25)
for asset in stale:
    print(f"Stale: {'/'.join(asset['key'])} — last updated: {asset['last_materialized']}")

Run this check on a schedule (cron every hour works well) and alert when critical assets are stale.

4. Run Queue Monitoring

Runs piling up in "QUEUED" or "STARTING" state indicate the run launcher is unhealthy or at capacity.

import httpx

RUN_STATUS_QUERY = """
{
  runsOrError(filter: {statuses: [QUEUED, STARTING]}) {
    ... on Runs {
      results {
        runId
        status
        startTime
        jobName
      }
    }
  }
}
"""

def get_stuck_runs(max_queued_minutes: int = 10) -> list[dict]:
    resp = httpx.post(
        "http://localhost:3000/graphql",
        json={"query": RUN_STATUS_QUERY},
        timeout=10
    )
    data = resp.json()
    runs = data["data"]["runsOrError"].get("results", [])
    
    import time
    stuck = []
    for run in runs:
        if run["startTime"]:
            age_minutes = (time.time() - run["startTime"]) / 60
            if age_minutes > max_queued_minutes:
                stuck.append({**run, "age_minutes": age_minutes})
    
    return stuck

stuck = get_stuck_runs(max_queued_minutes=15)
if stuck:
    print(f"ALERT: {len(stuck)} runs stuck in queue/starting state")

A persistent queue backlog usually means the run launcher (Docker, K8s, or subprocess) is saturated or broken. Alert if more than 5 runs are stuck for more than 15 minutes.

5. gRPC Code Server Health

Dagster uses a gRPC code server to serve pipeline definitions. If this process dies, Dagit can't load jobs or assets, and you'll see "Repository unavailable" errors.

# Check if the code server gRPC port is responsive
# Dagster uses port 4244 by default for workspace gRPC
nc -z localhost 4244 && echo "gRPC server up" || echo "gRPC server down"

In Python:

import socket

def check_grpc_server(host: str = "localhost", port: int = 4244) -> bool:
    try:
        with socket.create_connection((host, port), timeout=3):
            return True
    except (socket.timeout, ConnectionRefusedError, OSError):
        return False

6. Integrating Vigilmon for Pipeline SLA Monitoring

External Vigilmon checks provide the view from outside your network — essential for catching load balancer failures, DNS issues, and full host outages.

Step 1: Dagit availability monitor

  • URL: https://dagster.yourdomain.com/dagit_info
  • Method: GET
  • Expected status: 200
  • Check interval: 60 seconds
  • Alert after: 2 consecutive failures → PagerDuty

Step 2: Daemon health monitor

Expose your daemon health check as an HTTP endpoint:

from fastapi import FastAPI, Response

app = FastAPI()

@app.get("/daemon-health")
def daemon_health():
    healthy, issues = check_daemon_health()
    if healthy:
        return {"status": "ok"}
    return Response(
        content=f"Daemon unhealthy: {'; '.join(issues)}",
        status_code=503
    )

Then add a Vigilmon monitor:

  • URL: https://dagster.yourdomain.com/daemon-health
  • Method: GET
  • Expected status: 200
  • Check interval: 2 minutes

Step 3: Asset freshness SLA check

For critical assets with explicit SLAs, expose a freshness check endpoint:

@app.get("/asset-sla")
def asset_sla():
    stale = find_stale_assets(max_age_hours=25)
    critical_stale = [a for a in stale if "critical" in "/".join(a["key"])]
    if critical_stale:
        names = ["/".join(a["key"]) for a in critical_stale]
        return Response(
            content=f"SLA breach: {', '.join(names)}",
            status_code=503
        )
    return {"status": "ok", "checked_at": datetime.now(timezone.utc).isoformat()}

Wire this into Vigilmon with a 5-minute interval and a Slack alert channel.

Step 4: Full monitor configuration

monitors:
  - name: "Dagster Dagit UI"
    url: https://dagster.yourdomain.com/dagit_info
    interval: 60
    alerts:
      - type: pagerduty
        after_failures: 2
        severity: critical

  - name: "Dagster Daemon Health"
    url: https://dagster.yourdomain.com/daemon-health
    interval: 120
    alerts:
      - type: slack
        after_failures: 2
        severity: high

  - name: "Dagster Asset SLA"
    url: https://dagster.yourdomain.com/asset-sla
    interval: 300
    alerts:
      - type: slack
        after_failures: 1
        severity: high
      - type: email
        after_failures: 3
        severity: medium

Alerting Strategy by Component

| Component | Failure mode | Alert channel | Severity | |---|---|---|---| | Dagit webserver | HTTP 5xx or unreachable | PagerDuty | Critical | | Dagster daemon | Heartbeat stale | Slack | High | | Asset materialization | SLA breach | Slack + email | High | | Run queue | >10 stuck runs | Slack | High | | gRPC code server | Port closed | PagerDuty | Critical | | Schedule tick failures | Missed 3+ ticks | Email | Medium |

Putting It All Together

A production-ready Dagster monitoring setup combines:

  1. Vigilmon external monitors on Dagit and your daemon health endpoint — catches network-level outages
  2. Daemon health GraphQL check every 2 minutes — catches the most common silent failure mode
  3. Asset materialization staleness check on a 1-hour cron, alerting on SLA-bound assets
  4. Run queue depth monitoring alerting on sustained backlogs (>15 minutes, >5 runs)
  5. gRPC code server port check ensuring pipeline definitions are always servable
  6. A Vigilmon status page aggregating all monitors for stakeholder visibility

With this stack, your data engineering team will know about Dagster problems at the infrastructure layer before data consumers notice stale reports — and you'll have enough diagnostic context to fix the right thing fast.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles