tutorial

Monitoring Dagger with Vigilmon

Dagger lets you run CI/CD pipelines as code anywhere — but when the Dagger Engine API goes down, every pipeline execution fails instantly. Here's how to monitor the Dagger Engine, container runtime, pipeline cache, and registry connectivity with Vigilmon.

Dagger reimagines CI/CD by letting you define your entire build, test, and deployment pipeline as code — in Go, Python, TypeScript, or any language with a Dagger SDK — and run that same pipeline locally, in any CI system, or on a self-hosted Dagger Engine cluster. Automatic caching dramatically reduces pipeline execution time, and container-native execution makes pipelines reproducible across environments. But when the Dagger Engine itself has a problem — a stuck GraphQL API, an exhausted cache disk, a container runtime socket that went away — every pipeline execution fails immediately and silently. Vigilmon gives you deep observability into the Dagger Engine API, container runtime health, cache storage, and concurrent pipeline execution so you catch infrastructure failures before they halt your entire development workflow.

What You'll Set Up

  • Dagger Engine GraphQL API availability monitor (port 8080)
  • Container runtime health checks (containerd/Docker daemon)
  • Pipeline cache health and disk space monitoring
  • Concurrent pipeline execution health via heartbeat
  • Container registry connectivity checks
  • Pipeline execution success rate monitoring
  • Cache storage backend health (NFS/S3/local)
  • Resource utilization alerts (CPU/memory/OOM)

Prerequisites

  • Self-hosted Dagger Engine running as a container or systemd service
  • At least one Dagger pipeline executing via Go, Python, or TypeScript SDK
  • A free Vigilmon account

Step 1: Monitor the Dagger Engine GraphQL API

The Dagger Engine exposes a GraphQL API on port 8080. Every SDK client — Go, Python, TypeScript, Java — connects exclusively through this API. If the Engine is unreachable, all pipeline executions fail immediately with a connection error, regardless of the CI system invoking them.

  1. Log in to vigilmon.online and click Add Monitor.
  2. Set Type to HTTP / HTTPS.
  3. Enter the Dagger Engine API URL: http://your-dagger-engine:8080/query.
  4. Set Check interval to 1 minute.
  5. Set Expected HTTP status to 200 (the GraphQL introspection query returns 200 even when queried via GET).
  6. Click Save.

For a more meaningful health signal, add an introspection query monitor. Dagger's GraphQL API should respond to {"query":"{ __typename }"}:

curl -s -X POST http://your-dagger-engine:8080/query \
  -H 'Content-Type: application/json' \
  -d '{"query":"{ __typename }"}' | jq '.data.__typename'
# Expected output: "Query"

If your Engine is behind a reverse proxy, add an HTTP monitor for the proxy endpoint as well.

Step 2: Add a Health Endpoint to Your Dagger Engine Deployment

If you're running Dagger Engine as a container via Docker Compose or Kubernetes, add a sidecar health service that Vigilmon can probe independently from the GraphQL API:

# docker-compose.yml
services:
  dagger-engine:
    image: registry.dagger.io/engine:latest
    privileged: true
    volumes:
      - dagger-cache:/var/lib/dagger
    healthcheck:
      test: ["CMD", "wget", "-qO-", "http://localhost:8080/query", "--post-data", '{"query":"{ __typename }"}']
      interval: 30s
      timeout: 10s
      retries: 3
      start_period: 10s

  dagger-health:
    image: nginx:alpine
    ports:
      - "8081:80"
    depends_on:
      dagger-engine:
        condition: service_healthy
    volumes:
      - ./nginx-health.conf:/etc/nginx/conf.d/default.conf

volumes:
  dagger-cache:

# nginx-health.conf
server {
    listen 80;
    location /health {
        return 200 '{"status":"ok"}';
        add_header Content-Type application/json;
    }
}

Point Vigilmon at http://your-server:8081/health — the sidecar only returns 200 when the dagger-engine container passes its own health check.

Step 3: Monitor Container Runtime Health

Dagger Engine uses containerd or Docker to execute pipeline steps. If the container runtime socket becomes unavailable — a common failure after kernel upgrades, storage pressure, or daemon crashes — Dagger silently fails to start any new pipeline steps.

Add a Cron Heartbeat that probes the container runtime and fires on success:

  1. In Vigilmon, click Add Monitor → Cron Heartbeat.
  2. Name it Dagger Container Runtime Heartbeat.
  3. Set Expected ping interval to 5 minutes.
  4. Copy the generated heartbeat URL.

Set up a lightweight cron job on your Dagger Engine host:

# /etc/cron.d/dagger-health
*/5 * * * * root /usr/local/bin/dagger-runtime-check.sh

# /usr/local/bin/dagger-runtime-check.sh
#!/bin/bash
set -e

# Check containerd socket
if ! ctr version > /dev/null 2>&1; then
  echo "containerd unreachable"
  exit 1
fi

# Check Docker socket (if using Docker runtime)
if ! docker info > /dev/null 2>&1; then
  echo "Docker daemon unreachable"
  exit 1
fi

# Ping Vigilmon
curl -fsS "$VIGILMON_RUNTIME_HB_URL"

Set the VIGILMON_RUNTIME_HB_URL environment variable in /etc/environment. If containerd crashes at 3 AM, Vigilmon pages on-call within 5 minutes.

Step 4: Monitor Pipeline Cache Health

Dagger's automatic caching is the feature that makes pipelines dramatically faster — but the cache has its own failure modes: full disk, corrupted layer entries, and stale cache references that consume space without accelerating builds.

Add disk space monitoring for your Dagger cache volume:

# /usr/local/bin/dagger-cache-check.sh
#!/bin/bash
set -e

CACHE_DIR="/var/lib/dagger"
THRESHOLD_PERCENT=85

# Check disk usage
USAGE=$(df "$CACHE_DIR" | awk 'NR==2 {print $5}' | tr -d '%')

if [ "$USAGE" -gt "$THRESHOLD_PERCENT" ]; then
  echo "Dagger cache disk usage critical: ${USAGE}%"
  exit 1
fi

# Check cache directory is writable
touch "$CACHE_DIR/.vigilmon-write-test" && rm "$CACHE_DIR/.vigilmon-write-test"

curl -fsS "$VIGILMON_CACHE_HB_URL"

Add a Vigilmon Cron Heartbeat with a 15 minute interval to run this check. When disk usage exceeds 85%, the heartbeat fails and Vigilmon alerts before builds start failing with no space left on device.

For S3 or NFS cache backends, add TCP monitors:

  1. Add Monitor → TCP Port Check.
  2. NFS server: your NFS host, port 2049.
  3. Check interval: 2 minutes.

Step 5: Monitor Concurrent Pipeline Execution Health

Dagger supports parallel pipeline step execution via worker goroutines. Deadlocked pipelines, resource contention, and stuck executions silently consume resources while reporting no errors.

Add a pipeline execution heartbeat that fires after each successful pipeline run:

// main.go — Dagger pipeline with Vigilmon heartbeat
package main

import (
    "context"
    "fmt"
    "net/http"
    "os"

    "dagger.io/dagger"
)

func main() {
    ctx := context.Background()

    client, err := dagger.Connect(ctx, dagger.WithLogOutput(os.Stdout))
    if err != nil {
        panic(err)
    }
    defer client.Close()

    // Your pipeline steps
    _, err = client.Container().
        From("golang:1.22").
        WithDirectory("/src", client.Host().Directory(".")).
        WithWorkdir("/src").
        WithExec([]string{"go", "test", "./..."}).
        Sync(ctx)

    if err != nil {
        fmt.Fprintf(os.Stderr, "Pipeline failed: %v\n", err)
        os.Exit(1)
    }

    // Ping Vigilmon on success — not on failure
    hbURL := os.Getenv("VIGILMON_PIPELINE_HB_URL")
    if hbURL != "" {
        http.Get(hbURL) // nolint: errcheck
    }
    fmt.Println("Pipeline completed successfully")
}

Set the heartbeat interval to your pipeline cadence plus a buffer (e.g., 30 minutes for a pipeline that runs every 15 minutes). Stuck pipelines that never complete will miss the window and trigger an alert.

Add a max-execution-time guard to detect deadlocked pipelines:

ctx, cancel := context.WithTimeout(context.Background(), 20*time.Minute)
defer cancel()

Step 6: Monitor SDK Connectivity and Version Compatibility

Dagger SDK clients connect to the Engine's GraphQL API and require schema version compatibility. An Engine upgrade without an SDK update (or vice versa) causes schema mismatch errors that fail all pipelines.

Add a pre-pipeline SDK compatibility check:

# Python SDK — schema version check
import dagger
import os
import sys

async def check_compatibility():
    config = dagger.Config(log_output=sys.stderr)
    async with dagger.Connection(config) as client:
        # Query the Engine version
        version = await client.version()
        print(f"Dagger Engine version: {version}")
        return version

import asyncio
if __name__ == "__main__":
    version = asyncio.run(check_compatibility())
    # Optionally: assert minimum version
    major = int(version.split('.')[0])
    if major < 0:
        sys.exit(1)

Run this check as a pre-flight step in CI and gate the main pipeline execution on its success.

Step 7: Monitor Container Registry Connectivity

Dagger pulls base images and pushes output images during pipeline execution. Docker Hub pull rate limits, ECR token expiry, and GHCR authentication failures all cause mid-pipeline failures after expensive compute has already completed.

Add TCP monitors for each registry:

  1. Add Monitor → TCP Port Check.
  2. Docker Hub: host registry-1.docker.io, port 443.
  3. GHCR: host ghcr.io, port 443.
  4. ECR: host <account>.dkr.ecr.<region>.amazonaws.com, port 443.
  5. Check interval: 2 minutes per registry.

Add a Docker Hub rate limit check to your pre-pipeline steps:

# Check Docker Hub pull rate limit
TOKEN=$(curl -s "https://auth.docker.io/token?service=registry.docker.io&scope=repository:ratelimitpreview/test:pull" | jq -r .token)
LIMITS=$(curl -s -H "Authorization: Bearer $TOKEN" \
  -I https://registry-1.docker.io/v2/ratelimitpreview/test/manifests/latest 2>&1)
echo "$LIMITS" | grep -i ratelimit

Step 8: Monitor Cache Storage Backend Health

If your Dagger Engine uses a shared NFS mount or S3 bucket as its cache backend for distributed builds, that storage is a single point of failure for cache performance.

For NFS backends, add mount health monitoring:

# /usr/local/bin/dagger-nfs-check.sh
#!/bin/bash
set -e

NFS_MOUNT="/mnt/dagger-cache"

# Check mount is responsive
timeout 5 ls "$NFS_MOUNT" > /dev/null 2>&1 || {
  echo "NFS mount $NFS_MOUNT unresponsive"
  exit 1
}

# Check write health
TEST_FILE="$NFS_MOUNT/.vigilmon-$(date +%s)"
echo "test" > "$TEST_FILE" && rm "$TEST_FILE" || {
  echo "NFS mount $NFS_MOUNT is read-only or full"
  exit 1
}

curl -fsS "$VIGILMON_NFS_HB_URL"

Set up a Vigilmon Cron Heartbeat with a 5 minute interval for this check.

Step 9: Monitor Resource Utilization

Dagger's container-native execution is CPU and memory intensive for large parallel pipelines. OOM kills and CPU starvation cause stuck or failed pipeline steps with cryptic error messages.

Set up resource monitoring on your Dagger Engine host. If you use Docker stats:

# /usr/local/bin/dagger-resource-check.sh
#!/bin/bash

# Check if any Dagger containers are consuming >90% memory
HIGH_MEM=$(docker stats --no-stream --format "{{.Name}} {{.MemPerc}}" \
  $(docker ps --filter "label=dagger" -q) 2>/dev/null | \
  awk '{gsub(/%/, "", $2); if ($2 > 90) print $0}')

if [ -n "$HIGH_MEM" ]; then
  echo "WARNING: High memory usage in Dagger containers:"
  echo "$HIGH_MEM"
  exit 1
fi

curl -fsS "$VIGILMON_RESOURCE_HB_URL"

Set a 10 minute heartbeat interval. Add a separate alert channel for resource utilization warnings (lower priority than API downtime, but important for capacity planning).

Step 10: Configure Alerting

In Vigilmon, go to Alert Channels and configure tiered alerts:

  • PagerDuty — page on-call for Dagger Engine API downtime (port 8080) and container runtime failures. These block all pipeline execution immediately.
  • Slack #platform — notify for cache disk pressure, NFS mount failures, and pipeline heartbeat misses.
  • Slack #releases — notify for container registry TCP failures and Docker Hub rate limit warnings.
  • Email digest — weekly resource utilization summary and SDK compatibility audit.

Recommended thresholds:

  • Engine GraphQL API: alert after 2 consecutive failures (immediate all-pipelines-down impact).
  • Container runtime heartbeat: alert after missed window + 5 minutes.
  • Cache disk heartbeat: alert after 1 failure (disk full blocks writes immediately).
  • Registry TCP monitors: alert after 3 consecutive failures (transient network blips expected).
  • Pipeline execution heartbeat: alert after missed window + 50% of expected interval.

Conclusion

Dagger's promise — run your pipeline anywhere, with automatic caching and container-native execution — depends entirely on the Dagger Engine staying healthy. When the GraphQL API stalls, the containerd socket disappears, or the cache volume fills up, every pipeline execution fails immediately and CI grinds to a halt. With Vigilmon monitoring the Engine API, container runtime, cache health, registry connectivity, and pipeline execution heartbeats, your Dagger infrastructure becomes as observable as your production services. You'll know about Engine failures, OOM pressure, and NFS mount issues before your developers start wondering why dagger run is hanging.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles