Monitoring Concourse CI with Vigilmon: Web UI, Worker Health, Pipeline API, Build Heartbeats & SSL Alerts

Concourse CI is a declarative, container-native continuous integration system designed around pipelines, resources, and jobs. Unlike SaaS CI platforms, Concourse runs entirely on your own infrastructure — which means when the ATC (Air Traffic Controller) process crashes, workers disconnect, or the database becomes unavailable, your pipelines stall silently. Vigilmon adds the external observability layer that Concourse doesn't include out of the box: uptime checks, health endpoint validation, worker connectivity heartbeats, pipeline API monitoring, and SSL certificate alerts.

What You'll Build

• An HTTP monitor for the Concourse web UI
• A health endpoint check via /api/v1/info
• A pipeline API check confirming the ATC is serving requests
• A worker connectivity heartbeat to catch disconnected workers
• A build completion heartbeat for end-to-end pipeline verification
• SSL certificate expiry alerts

Prerequisites

• Concourse CI deployed (Docker Compose, binary, or BOSH)
• At least one worker registered and running
• A domain configured for your Concourse server with HTTPS
• A free account at vigilmon.online

Step 1: Monitor the Concourse Web UI

The Concourse dashboard shows all pipeline states, recent builds, and worker status. A down dashboard leaves your team with no visibility into CI health:

1. Log in to Vigilmon → Add Monitor → HTTP.
2. URL: https://concourse.yourdomain.com
3. Check interval: 120 seconds.
4. Expected status: 200.
5. Label: Concourse Web UI
6. Click Save.

Concourse's root path serves the Elm frontend. A 200 response confirms the ATC web server process is alive and your reverse proxy is routing correctly.

Step 2: Check the Concourse Health Endpoint

Concourse exposes /api/v1/info — a lightweight endpoint that returns server version and cluster info, and fails immediately when the ATC cannot reach its database:

1. Add Monitor → HTTP.
2. URL: https://concourse.yourdomain.com/api/v1/info
3. Check interval: 60 seconds.
4. Expected status: 200.
5. Keyword: version (confirms the JSON body contains the server version field).
6. Label: Concourse API Info
7. Click Save.

The /api/v1/info endpoint bypasses the frontend entirely and directly exercises the ATC's API layer and PostgreSQL connectivity. If the database drops, this endpoint returns a 500 or connection error while the cached frontend may still render — making it your primary alert signal.

Step 3: Monitor the Pipeline API

Concourse's REST API is what drives all fly CLI operations and webhook triggers. Monitor it with a periodic script that lists pipelines and sends a heartbeat on success:

#!/bin/bash
# /etc/cron.d/concourse-api-check — runs every 5 minutes

TOKEN=$(curl -s -X POST https://concourse.yourdomain.com/sky/issuer/token \
  -d "grant_type=password&username=YOUR_USER&password=YOUR_PASS&scope=openid+profile+email+federated:id+groups" \
  | python3 -c "import sys,json; print(json.load(sys.stdin).get('access_token',''))")

if [ -n "$TOKEN" ]; then
  STATUS=$(curl -s -o /dev/null -w "%{http_code}" \
    -H "Authorization: Bearer $TOKEN" \
    https://concourse.yourdomain.com/api/v1/pipelines)

  if [ "$STATUS" = "200" ]; then
    curl -fsS -X POST https://vigilmon.online/api/heartbeat/YOUR-API-HEARTBEAT-ID
  fi
fi

In Vigilmon:

1. Add Monitor → Heartbeat.
2. Expected interval: 5 minutes.
3. Grace period: 15 minutes.
4. Label: Concourse Pipeline API

The token fetch exercises the OAuth/OIDC stack; the pipeline list exercises the ATC's database queries. Both must succeed for the heartbeat to fire.

Step 4: Worker Connectivity Heartbeat

Concourse workers are separate processes (or containers) that execute jobs. A disconnected worker causes all new jobs to queue indefinitely. Monitor worker health from the Concourse host:

#!/bin/bash
# /etc/cron.d/concourse-workers — runs every 5 minutes

TOKEN=$(curl -s -X POST https://concourse.yourdomain.com/sky/issuer/token \
  -d "grant_type=password&username=YOUR_USER&password=YOUR_PASS&scope=openid+profile+email+federated:id+groups" \
  | python3 -c "import sys,json; print(json.load(sys.stdin).get('access_token',''))")

RUNNING=$(curl -s -H "Authorization: Bearer $TOKEN" \
  https://concourse.yourdomain.com/api/v1/workers \
  | python3 -c "import sys,json; w=json.load(sys.stdin); print(sum(1 for x in w if x.get('state')=='running'))")

if [ "${RUNNING:-0}" -ge 1 ] 2>/dev/null; then
  curl -fsS -X POST https://vigilmon.online/api/heartbeat/YOUR-WORKER-HEARTBEAT-ID
fi

In Vigilmon:

1. Add Monitor → Heartbeat.
2. Expected interval: 5 minutes.
3. Grace period: 15 minutes.
4. Label: Concourse Worker Health

The script checks that at least one worker reports state: running. If all workers stall, restart, or lose connectivity to the ATC, the heartbeat stops within the grace period and Vigilmon alerts you.

Step 5: SSL Certificate Alerts

Concourse relies on HTTPS for all fly CLI connections, web sessions, and webhook deliveries. An expired certificate breaks every client interaction immediately:

1. Add Monitor → SSL Certificate.
2. Domain: concourse.yourdomain.com
3. Alert when expiry is within: 30 days.
4. Alert again: 14 days, 7 days, 3 days.
5. Click Save.

A 30-day lead time gives you room to renew before the certificate breaks fly CLI authentication or pipeline webhook delivery.

Step 6: Build Completion Heartbeat

Add a heartbeat ping at the end of a critical pipeline to verify end-to-end CI health — not just that Concourse is running, but that it is successfully completing builds:

# pipeline.yml
jobs:
  - name: build-and-test
    plan:
      - get: source-code
        trigger: true
      - task: run-tests
          config:
            platform: linux
            image_resource:
              type: docker-image
              source: {repository: node, tag: "20"}
            inputs:
              - name: source-code
            run:
              path: sh
              args:
                - -c
                - |
                  cd source-code
                  npm ci && npm test

      - task: notify-vigilmon
        config:
          platform: linux
          image_resource:
            type: docker-image
            source: {repository: curlimages/curl}
          run:
            path: curl
            args:
              - -fsS
              - -X
              - POST
              - https://vigilmon.online/api/heartbeat/YOUR-BUILD-HEARTBEAT-ID

In Vigilmon:

1. Add Monitor → Heartbeat.
2. Expected interval: Match your pipeline trigger frequency (e.g., 60 minutes for hourly pipelines).
3. Grace period: 30 minutes.
4. Label: Concourse Build Completion

The heartbeat only fires after all preceding tasks succeed. If the test task fails, or if Concourse stops scheduling jobs entirely, the heartbeat goes silent and Vigilmon alerts you.

Step 7: TCP Port Check for ATC and PostgreSQL

Concourse's ATC listens on port 8080 (HTTP) or 443 (HTTPS via reverse proxy), and depends on PostgreSQL on port 5432. Monitor both with TCP checks:

1. Add Monitor → TCP.
2. Host: concourse.yourdomain.com, Port: 443.
3. Check interval: 60 seconds.
4. Label: Concourse ATC Port
5. Click Save.

For PostgreSQL (if on the same host, from an internal monitor):

1. Add Monitor → TCP.
2. Host: localhost (or your DB host), Port: 5432.
3. Label: Concourse PostgreSQL

A TCP failure on port 443 before the HTTP monitor fires means your reverse proxy is the problem — not the ATC — which narrows the diagnosis immediately.

Common Concourse CI Failure Modes and What Vigilmon Catches

| Scenario | Vigilmon monitor | |---|---| | ATC process crash | /api/v1/info and web UI monitors fire within 60 s | | PostgreSQL connection failure | API info endpoint returns 500 | | All workers disconnect | Worker heartbeat stops within grace period | | Job queue stalls | Build completion heartbeat stops | | SSL certificate expires | SSL monitor alerts at 30 days | | Reverse proxy misconfiguration | TCP check fires; HTTP monitors also fire | | OAuth/OIDC authentication failure | Pipeline API heartbeat stops | | Builds consistently failing | Build completion heartbeat goes silent | | DNS misconfiguration | All HTTP and SSL monitors fire simultaneously |

Self-hosting Concourse gives you declarative, reproducible pipelines with no vendor lock-in. Vigilmon ensures those pipelines keep running — watching the ATC, worker pool, API layer, build completion, and SSL certificate simultaneously so you catch failures before developers wonder why their commits aren't building.

Start monitoring Concourse CI in under 5 minutes — register free at vigilmon.online.