Monitoring Dagu with Vigilmon

Dagu is a self-hosted workflow orchestration engine that runs DAGs (Directed Acyclic Graphs) on a cron schedule. It runs on port 8080, exposes a REST API for workflow management, and provides a web UI for visualizing DAG execution history. Dagu is the kind of infrastructure that runs silently in the background — and fails silently too. If the scheduler stops firing or a step execution pipeline deadlocks, your workflows just quietly stop running. Vigilmon gives you the monitoring layer Dagu doesn't include: uptime checks, scheduler heartbeats, API response monitoring, and log storage availability alerts.

What You'll Set Up

• Web UI availability monitor
• DAG execution scheduler health check via cron heartbeat
• Step execution pipeline monitor
• Log storage service availability check
• REST API endpoint response time monitoring
• Scheduler cron health monitoring

Prerequisites

• Dagu running and accessible (default port 8080)
• DAGs configured and scheduled (at least one recurring DAG to use as a heartbeat)
• A free Vigilmon account

Step 1: Monitor the Dagu Web UI

Dagu's web interface runs on port 8080 and provides the dashboard for monitoring DAG execution history, logs, and status. If it goes down, you lose visibility into your workflow state:

1. Log in to vigilmon.online and click Add Monitor.
2. Set Type to HTTP / HTTPS.
3. Enter the URL: https://dagu.yourdomain.com (or http://your-server:8080 for direct access).
4. Set Check interval to 1 minute.
5. Set Expected HTTP status to 200.
6. Click Save.

Dagu serves its UI on the root path. A 200 response confirms the Go HTTP server is alive.

Verify your setup before saving:

curl -I http://your-server:8080
# HTTP/1.1 200 OK
# content-type: text/html

Step 2: Monitor the DAG Execution Scheduler Health

Dagu's scheduler fires DAG runs based on cron expressions defined in each DAG's YAML file. The scheduler runs as part of the Dagu process — there's no separate scheduler daemon — so if the Dagu process is up but internally deadlocked, scheduled runs stop while the UI continues to load.

Use a dedicated heartbeat DAG to confirm the scheduler is firing:

1. Create a minimal heartbeat DAG in your Dagu configuration directory:

# ~/.config/dagu/dags/vigilmon-heartbeat.yaml
name: vigilmon-heartbeat
schedule: "*/5 * * * *"   # every 5 minutes
steps:
  - name: ping
    command: curl -s https://vigilmon.online/heartbeat/YOUR_HEARTBEAT_ID

2. In Vigilmon, click Add Monitor → Cron Heartbeat.
3. Set the expected ping interval to 10 minutes (2× the schedule interval for tolerance).
4. Copy the heartbeat URL and update the DAG's command field with it.
5. In Dagu's web UI, enable the vigilmon-heartbeat DAG.

Now if the scheduler stops firing DAGs — for any reason — the heartbeat pings stop, and Vigilmon alerts you. This is the most reliable health check for a cron-based scheduler: the scheduler itself is the probe.

Step 3: Monitor the Step Execution Pipeline

Dagu executes DAG steps sequentially (or in parallel, if configured). If the step execution layer hangs — due to a subprocess deadlock, file descriptor exhaustion, or a zombie process accumulation — DAGs will appear to start but steps will never complete.

Monitor step execution health via the Dagu REST API. The /api/v1/dags endpoint returns the status of all DAGs, including whether any are stuck in a running state:

1. In Vigilmon, add an HTTP / HTTPS monitor.
2. URL: http://your-server:8080/api/v1/dags
3. Set Expected HTTP status to 200.
4. Set Check interval to 2 minutes.
5. Click Save.

For deeper step execution monitoring, add a script-based heartbeat that checks for stuck runs:

#!/bin/bash
# /usr/local/bin/dagu-step-health.sh

# Get all DAG statuses from Dagu API
RESPONSE=$(curl -s -u admin:YOUR_PASSWORD http://localhost:8080/api/v1/dags)

# Check if API responded
if [ -z "$RESPONSE" ]; then
  exit 1
fi

# Optionally: check for DAGs stuck in "running" for too long
# (pipe through jq if available)
echo "$RESPONSE" | grep -q '"status"' && \
  curl -s https://vigilmon.online/heartbeat/STEP_HEALTH_HEARTBEAT_ID

Run this from cron every 5 minutes to confirm the API responds and basic step status data is available.

Step 4: Monitor the Log Storage Service

Dagu writes execution logs to disk (typically under ~/.local/share/dagu/logs/). If the disk fills up or the log directory becomes unwritable, DAG runs will fail with cryptic I/O errors, or worse — fail silently with truncated logs.

Add a disk health heartbeat:

1. In Vigilmon, add another Cron Heartbeat monitor.
2. Set the expected ping interval to 30 minutes.
3. Add a cron job on your Dagu host:

# /etc/cron.d/dagu-log-health
*/30 * * * * root \
  /usr/local/bin/dagu-log-health.sh

# /usr/local/bin/dagu-log-health.sh
#!/bin/bash
LOG_DIR="${HOME}/.local/share/dagu/logs"

# Check log directory is writable
if [ ! -w "$LOG_DIR" ]; then
  exit 1
fi

# Check disk usage on the log partition is below 85%
DISK_USAGE=$(df "$LOG_DIR" | awk 'NR==2{print $5}' | tr -d '%')
if [ "$DISK_USAGE" -ge 85 ]; then
  exit 1
fi

# Check we can write a test file
TEST_FILE="$LOG_DIR/.vigilmon-health-test"
if ! touch "$TEST_FILE" 2>/dev/null; then
  exit 1
fi
rm -f "$TEST_FILE"

curl -s https://vigilmon.online/heartbeat/LOG_HEALTH_HEARTBEAT_ID

If any check fails, the heartbeat is never sent and Vigilmon alerts you.

Step 5: Monitor the REST API Endpoint Response Times

Dagu's REST API at /api/v1/ is used by external tooling, CI/CD pipelines, and potentially other services to trigger or check DAG runs. A slow API means triggered runs experience delays.

1. Open the HTTP / HTTPS monitor for http://your-server:8080/api/v1/dags.
2. Under Response time, set Alert if response time exceeds 3000 ms.
3. Click Save.

Also add a direct API health endpoint monitor if your Dagu version exposes one:

curl -s http://your-server:8080/api/v1/health
# {"status":"healthy"}

Check the Dagu API documentation for your installed version. If a health endpoint is available, add it as a separate Vigilmon monitor with a body contains check for "healthy" to distinguish an up-but-degraded API from a fully healthy one.

Step 6: Monitor Scheduler Cron Health

Beyond the heartbeat DAG (Step 2), confirm that Dagu's internal cron engine is ticking correctly by monitoring the gap between successive scheduled DAG runs. This catches the edge case where a DAG runs once and then the scheduler silently drops subsequent triggers.

Set up a Vigilmon monitor that watches DAG execution history via the API:

#!/bin/bash
# /usr/local/bin/dagu-cron-health.sh
# Checks that the heartbeat DAG ran within the last 15 minutes

DAG_NAME="vigilmon-heartbeat"
API_BASE="http://localhost:8080/api/v1"

# Fetch last run time for the heartbeat DAG
LAST_RUN=$(curl -s -u admin:YOUR_PASSWORD \
  "${API_BASE}/dags/${DAG_NAME}" | \
  grep -o '"finishedAt":"[^"]*"' | head -1 | cut -d'"' -f4)

if [ -z "$LAST_RUN" ]; then
  echo "No run data found"
  exit 1
fi

# Parse the timestamp and compare to now (requires date -d on Linux)
LAST_RUN_EPOCH=$(date -d "$LAST_RUN" +%s 2>/dev/null || echo 0)
NOW_EPOCH=$(date +%s)
AGE_MINUTES=$(( (NOW_EPOCH - LAST_RUN_EPOCH) / 60 ))

if [ "$AGE_MINUTES" -le 15 ]; then
  curl -s https://vigilmon.online/heartbeat/CRON_HEALTH_HEARTBEAT_ID
fi

Run this from cron every 10 minutes. This wraps your heartbeat DAG with an external verification layer — even if the heartbeat DAG somehow fails to ping Vigilmon, this script provides a second path to detect scheduler drift.

Step 7: Configure Alert Channels

1. Go to Alert Channels in Vigilmon and add Slack, email, or PagerDuty.
2. Set Consecutive failures before alert to 2 for the web UI and REST API monitors.
3. Set it to 1 for the scheduler heartbeat — a single missed heartbeat may mean a DAG run was dropped.
4. Use Maintenance windows in Vigilmon before restarting Dagu or deploying configuration changes to suppress expected downtime alerts.

Create a dedicated alert channel for Dagu so workflow failures are separated from infrastructure alerts. Route the scheduler heartbeat alert to on-call if any of your DAGs process time-sensitive work.

Summary

| Monitor | Target | What It Catches | |---|---|---| | Web UI | http://your-server:8080 | Dagu process crash, reverse proxy failure | | Scheduler heartbeat | Heartbeat URL (from DAG) | Scheduler freeze, missed cron triggers | | REST API | /api/v1/dags | API routing failure, authentication issues | | API response time | /api/v1/dags | Database/storage slowdown, resource exhaustion | | Log storage heartbeat | Heartbeat URL | Disk full, log directory unwritable | | Cron health heartbeat | Heartbeat URL | DAG run gap detection, scheduler drift |

Dagu's strength is its simplicity — YAML DAGs, a lightweight Go binary, and a clean web UI. But that simplicity means there's no built-in alerting when the scheduler stops or a log write fails. Vigilmon's combination of HTTP monitors, cron heartbeats, and API response time checks fills that gap, giving you the observability layer that lets Dagu workflows run reliably in production without babysitting.