tutorial

Monitoring Woodpecker CI with Vigilmon: Server Health, Agent Connectivity, Pipeline Queue Depth & Build Success Rates

How to monitor Woodpecker CI with Vigilmon — server health checks, agent connectivity validation, pipeline queue depth alerting, build success rate monitoring, and webhook delivery verification.

Woodpecker CI is the open-source CI/CD system forked from Drone, built around a simple server-agent architecture where the server manages the pipeline queue and webhooks, and agents pull jobs to execute them. When the Woodpecker server loses connectivity to its database, new webhook-triggered pipelines queue up silently and never start executing — your commits pile up without any build status. When all agents disconnect, the queue depth climbs while every pending pipeline shows pending indefinitely. When a webhook delivery from your SCM (GitHub, Gitea, Forgejo) fails due to a network error, the pipeline never triggers and you have no idea a commit is untested. Vigilmon gives you external visibility into Woodpecker's API, agent registration endpoint, and SCM webhook delivery path, so you catch queue stalls and agent outages before your team wonders why CI is broken.

What You'll Build

  • An HTTP monitor on Woodpecker's API health endpoint to detect server failures
  • Agent connectivity validation via Woodpecker's agent registration API
  • Pipeline queue depth alerting using Woodpecker's Prometheus metrics
  • Build success rate monitoring with Prometheus alerting rules
  • Webhook delivery endpoint monitoring for your SCM integration
  • SSL certificate monitoring for the Woodpecker server
  • An alerting runbook mapping Woodpecker failure modes to Vigilmon monitor states

Prerequisites

  • Woodpecker CI 2.x with server and at least one agent running
  • PostgreSQL or SQLite database backend (MySQL also supported)
  • Prometheus configured to scrape Woodpecker's /metrics endpoint
  • Woodpecker's API accessible over HTTPS (default port 443 or 8000)
  • SCM webhook integration configured (GitHub, Gitea, Forgejo, Bitbucket)
  • A free account at vigilmon.online

Step 1: Understand Woodpecker's Health Surface

Woodpecker CI exposes health and metrics endpoints on the server:

# Server health check — returns 200 when database is connected and server is ready
curl https://ci.example.com/healthz
# Returns: {"status":"ok"}

# Prometheus metrics — requires WOODPECKER_METRICS_SERVER_ADDR configured
curl http://ci.example.com:9001/metrics | grep woodpecker_

# Key metrics
woodpecker_pipelines_total          # Total pipelines by status (pending/running/success/failure)
woodpecker_pending_jobs             # Current queue depth
woodpecker_running_jobs             # Jobs actively executing on agents
woodpecker_agents                   # Connected agent count
woodpecker_worker_count             # Total available worker slots

For the agent side, each Woodpecker agent connects back to the server via gRPC. Check agent registration:

# List connected agents via API
curl -s -H "Authorization: Bearer $WOODPECKER_TOKEN" \
  https://ci.example.com/api/agents | jq '.[] | {id, name, platform, capacity}'

# Check pipeline queue via API
curl -s -H "Authorization: Bearer $WOODPECKER_TOKEN" \
  https://ci.example.com/api/queue/info | jq '{pending: .pending, running: .running, agent_count: .agents}'

Step 2: Monitor the Server Health Endpoint

The /healthz endpoint confirms that Woodpecker's server is running and connected to its database. This is the most important baseline check:

  1. Log in to Vigilmon → Add Monitor → HTTP.
  2. URL: https://ci.example.com/healthz.
  3. Check interval: 60 seconds.
  4. Response timeout: 10 seconds.
  5. Expected status: 200.
  6. Keyword: ok.
  7. Label: Woodpecker CI server health.
  8. Click Save.

This monitor catches:

  • Database connection failures that prevent the server from accepting new pipeline triggers
  • Server process crashes and container restarts
  • Kubernetes liveness probe failures that leave the server pod unresponsive
  • Network failures between the server and its reverse proxy

Alert after: 1 consecutive failure. A failed health check means no new pipelines will start.

Step 3: Monitor the Woodpecker API Endpoint

The API endpoint validates that authenticated operations work — not just that the server process is alive:

  1. Add Monitor → HTTP.
  2. URL: https://ci.example.com/api/info.
  3. Check interval: 2 minutes.
  4. Response timeout: 10 seconds.
  5. Expected status: 200.
  6. Keyword: woodpecker.
  7. Label: Woodpecker API availability.
  8. Click Save.

The /api/info endpoint returns server version and configuration metadata. It requires no authentication and confirms the API layer is fully operational — not just the health probe shim.

Step 4: Configure Prometheus Alerting for Agent Connectivity and Queue Depth

Woodpecker's Prometheus metrics are where you detect the most critical operational failures — agents disconnecting and pipeline queues backing up:

# prometheus-woodpecker-alerts.yaml
groups:
  - name: woodpecker_ci
    rules:
      - alert: WoodpeckerNoAgents
        expr: woodpecker_agents == 0
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "Woodpecker CI has no connected agents"
          description: "All Woodpecker agents have disconnected — no pipelines can execute. Check agent logs and network connectivity to the server."

      - alert: WoodpeckerQueueDepthHigh
        expr: woodpecker_pending_jobs > 20
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Woodpecker pipeline queue depth is high"
          description: "{{ $value }} pipelines are pending — agents may be insufficient for current load or stuck on a long-running job."

      - alert: WoodpeckerHighFailureRate
        expr: |
          rate(woodpecker_pipelines_total{status="failure"}[30m]) /
          rate(woodpecker_pipelines_total{status=~"success|failure"}[30m]) > 0.5
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "Woodpecker CI failure rate above 50%"
          description: "More than half of recent pipelines are failing — check for a bad commit, broken test infrastructure, or misconfigured pipeline."

      - alert: WoodpeckerQueueStalled
        expr: woodpecker_pending_jobs > 5 and woodpecker_running_jobs == 0
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "Woodpecker pipeline queue is stalled"
          description: "Pipelines are queued but none are running — agents are disconnected or unable to pull work."

Apply and verify:

kubectl apply -f prometheus-woodpecker-alerts.yaml
curl http://prometheus:9090/api/v1/rules | jq '.data.groups[].rules[] | select(.name | startswith("Woodpecker"))'

Step 5: Monitor Webhook Delivery Endpoints

Woodpecker CI triggers pipelines from SCM webhooks. If the webhook delivery path is broken — due to a DNS failure, certificate error, or firewall rule — no pipelines trigger and CI appears idle while silently doing nothing:

# Check Woodpecker's webhook endpoint reachability
# An unauthenticated POST returns 400 (signature validation fails) — confirms endpoint is live
curl -X POST https://ci.example.com/api/hook \
  -H "Content-Type: application/json" \
  -d '{}' \
  -w "%{http_code}"
# Expected: 400 (not 000/connection refused/5xx)
  1. Add Monitor → HTTP.
  2. URL: https://ci.example.com/api/hook.
  3. Check interval: 5 minutes.
  4. Response timeout: 10 seconds.
  5. Expected status: 400 (unauthenticated POST confirms endpoint is live; HMAC signature validation rejects the empty payload correctly).
  6. Label: Woodpecker webhook endpoint.
  7. Click Save.

Monitoring for HTTP 400 is the correct approach: the webhook endpoint requires a valid HMAC signature. An empty POST is rejected with 400, confirming the server is reachable and TLS is valid without triggering a build.

Step 6: Monitor Agent Availability via the API

Individual agent health can be checked via the Woodpecker API. Create a monitor using a read-only API token:

# Create a monitoring token via the Woodpecker UI
# Settings → Tokens → Add Token → role: viewer

# Test the queue endpoint
curl -s -H "Authorization: Bearer $MONITORING_TOKEN" \
  https://ci.example.com/api/queue/info | jq '{pending, running}'
  1. Add Monitor → HTTP.
  2. URL: https://ci.example.com/api/queue/info.
  3. Check interval: 2 minutes.
  4. Response timeout: 10 seconds.
  5. Expected status: 200.
  6. Keyword: "pending".
  7. Custom header: Authorization: Bearer <your-monitoring-token>.
  8. Label: Woodpecker agent queue health.
  9. Click Save.

Step 7: SSL Certificate Monitoring

Woodpecker CI must be accessible via HTTPS for SCM webhooks to deliver securely. Certificate expiry breaks webhook delivery from all connected repositories:

  1. Add Monitor → SSL Certificate.
  2. Domain: ci.example.com.
  3. Alert when expiry is within: 30 days.
  4. Alert again: 14 days, 7 days, 3 days.
  5. Click Save.

When the Woodpecker TLS certificate expires, GitHub/Gitea webhook deliveries start failing with SSL errors — your SCM may automatically disable the webhook after repeated delivery failures, requiring manual re-activation for each repository.

Step 8: Configure Alerting for CI Failures

In Vigilmon under Settings → Notifications, configure alert channels and map them to the correct response runbook:

| Monitor | Trigger | Immediate action | |---|---|---| | Server health /healthz | Non-200 or keyword missing | Check database connectivity; check container/pod status; review server logs | | API availability /api/info | Non-200 or keyword missing | API layer has failed despite healthy process; check reverse proxy configuration | | Webhook endpoint /api/hook | Non-400 or connection refused | SCM cannot deliver webhooks; check firewall rules and TLS configuration | | Agent queue health | Non-200 or missing pending keyword | Agent connectivity broken; check agent process status and gRPC connectivity to server | | SSL certificate | < 30 days to expiry | Renew certificate; SCM webhook delivery will fail on TLS errors after expiry |

Build failure alerting: For team-level awareness of build failures, use Woodpecker's native notification plugins (Slack, email, Matrix) rather than routing build failures through Vigilmon. Vigilmon is best suited for infrastructure-level health checks — server availability, agent connectivity, webhook reachability — rather than per-build outcome tracking.

Common Woodpecker Failure Modes and What Vigilmon Catches

| Scenario | Vigilmon signal | |---|---| | Database connection lost | Server health monitor fires; /healthz returns non-200 | | All agents disconnected | Queue stall Prometheus alert fires; agent count drops to 0 | | Agent gRPC connection timeout | Queue stall alert fires; running jobs drop to 0 while pending climbs | | Webhook delivery fails (TLS error) | SSL monitor fires 30 days before expiry; webhook endpoint monitor fires on connection failure | | SCM network blocks Woodpecker IP | Webhooks stop arriving; monitor queue depth stagnation in Prometheus | | Pipeline queue exceeds capacity | Queue depth Prometheus alert fires | | High failure rate from broken test infra | Failure rate Prometheus alert fires; not visible to Vigilmon endpoint checks | | Server container OOM killed | Server health monitor fires within 60 seconds of restart | | Reverse proxy misconfiguration | API availability monitor fires; server process may be healthy behind broken proxy | | Agent resource limits prevent job execution | Queue stall alert fires; running jobs at 0 despite connected agents |

Woodpecker CI is a lightweight but operationally sensitive system: every layer — the server, the agent fleet, the database, and the webhook delivery path — must be healthy for your CI pipeline to fire reliably. When the queue stalls silently, your team commits code that appears untested while the CI system looks idle. Vigilmon gives you an external vantage point on every layer: server health checks, API availability, webhook endpoint reachability, and SSL certificate tracking that operate independently of Woodpecker itself. When Woodpecker can't tell you something is wrong, Vigilmon can.

Start monitoring your Woodpecker CI infrastructure in under 5 minutes — register free at vigilmon.online.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles