Monitoring GitLab CI with Vigilmon: Pipeline Health, Runner Availability, Instance Uptime & TLS Certificate Alerts

GitLab CI is the integrated CI/CD system built directly into GitLab — every project gets pipelines, runners, environments, and a container registry without installing any additional tool. When your self-hosted GitLab instance goes down, every team's CI pipeline halts and deployments stop. When a GitLab Runner goes offline, jobs queue indefinitely and engineers see no clear error — just pipelines stuck in "pending." When the container registry is unreachable, Docker builds succeed but pushes fail silently in the last step. Vigilmon gives you external visibility into GitLab instance availability, runner health endpoints, pipeline API reachability, container registry uptime, and the TLS layer protecting your entire CI/CD platform.

What You'll Build

• HTTP monitor on the GitLab instance to detect web service outages
• SSL certificate monitor for the GitLab HTTPS endpoint
• HTTP monitor on the GitLab Runner health endpoint to detect stuck or offline runners
• TCP monitors on GitLab and registry ports for network-layer health
• Heartbeat monitor for scheduled GitLab CI pipelines
• Alerting runbook mapping GitLab CI failure modes to the right components

Prerequisites

• Self-hosted GitLab CE/EE (13.0+) or GitLab.com connectivity
• At least one GitLab Runner registered and executing jobs
• A free account at vigilmon.online

Step 1: Understand GitLab CI's Health Surface

GitLab exposes several health check endpoints that Vigilmon can probe directly:

# GitLab liveness check (checks if the Rails process is alive)
curl https://gitlab.example.com/-/liveness
# Expected: {"status":"ok"}

# GitLab readiness check (checks if GitLab is ready to serve traffic)
curl https://gitlab.example.com/-/readiness
# Expected: {"status":"ok"} plus individual component checks

# GitLab health endpoint (legacy, maps to readiness)
curl https://gitlab.example.com/-/health
# Expected: GitLab OK

# Check GitLab version and API reachability
curl https://gitlab.example.com/api/v4/version
# Expected: {"version":"16.x.x","revision":"..."}

The key external signals to monitor:

1. GitLab web service (HTTP / readiness)
2. GitLab Runner health endpoint (per-runner HTTP)
3. GitLab container registry
4. TLS certificate on the GitLab domain
5. Scheduled pipeline heartbeats

Step 2: Monitor the GitLab Instance

Create an HTTP monitor on GitLab's readiness endpoint — it checks the Rails web service, database connection, Redis, and Sidekiq all at once:

# Verify the readiness endpoint returns 200
curl -I https://gitlab.example.com/-/readiness
# Expected: HTTP/1.1 200 OK

# Check the component breakdown
curl https://gitlab.example.com/-/readiness | jq '.status, .master_check, .db_check, .redis_check'

1. Log in to Vigilmon → Add Monitor → HTTP.
2. URL: https://gitlab.example.com/-/readiness.
3. Check interval: 60 seconds.
4. Response timeout: 15 seconds.
5. Expected status: 200.
6. Keyword: ok (present in the JSON response).
7. Label: GitLab instance readiness.
8. Click Save.

Add a second monitor on the root URL to catch ingress or CDN failures that the readiness endpoint might not detect:

1. Add Monitor → HTTP.
2. URL: https://gitlab.example.com.
3. Expected status: 200.
4. Keyword: GitLab (present in the page title).
5. Label: GitLab web UI.
6. Click Save.

Step 3: Monitor GitLab Runner Health

GitLab Runner exposes its own health endpoint when started with the --listen-address flag. A healthy runner responds with a metrics page; an offline runner responds with nothing — and pipelines queue silently:

# Start GitLab Runner with the metrics endpoint exposed
# In /etc/gitlab-runner/config.toml:
# listen_address = "0.0.0.0:9252"

# Test the runner health endpoint
curl http://runner.example.com:9252/metrics
# Expected: Prometheus metrics including gitlab_runner_jobs_total

# Check the specific health endpoint
curl http://runner.example.com:9252/
# Expected: "GitLab Runner" page

# Alternatively, check the jobs API for stuck pending jobs
curl --header "PRIVATE-TOKEN: <your_token>" \
  "https://gitlab.example.com/api/v4/jobs?scope=pending" | jq 'length'
# If count grows over time, runners are offline

Configure the runner metrics endpoint:

# /etc/gitlab-runner/config.toml
listen_address = "0.0.0.0:9252"

[[runners]]
  name = "my-runner"
  url = "https://gitlab.example.com/"
  # ... rest of runner config

# Restart GitLab Runner to apply
sudo systemctl restart gitlab-runner

# Verify the metrics endpoint is available
curl -s http://localhost:9252/metrics | grep gitlab_runner_version

1. Add Monitor → HTTP.
2. URL: http://runner.example.com:9252/metrics (or HTTPS if TLS-terminated).
3. Check interval: 60 seconds.
4. Expected status: 200.
5. Keyword: gitlab_runner (present in the Prometheus metrics response).
6. Label: GitLab Runner metrics.
7. Click Save.

For runners not exposing a metrics endpoint, monitor the runner registration status via the GitLab API:

# List runners and their online status
curl --header "PRIVATE-TOKEN: <token>" \
  "https://gitlab.example.com/api/v4/runners?type=instance_type&status=online"
# If empty, all runners are offline

Step 4: Monitor the GitLab Container Registry

The GitLab container registry listens on a separate port (5050 by default for self-hosted) and is critical for Docker build → push workflows:

# Check the registry API
curl -I https://gitlab.example.com:5050/v2/
# Expected: HTTP/1.1 200 OK (or 401 with Www-Authenticate header — both mean the registry is up)

# Login and check registry connectivity
docker login gitlab.example.com:5050
echo $?  # 0 = success

# Check the registry health endpoint
curl https://gitlab.example.com:5050/v2/_catalog
# Expected: {"repositories":[...]} or 401 (registry is up)

1. Add Monitor → HTTP.
2. URL: https://gitlab.example.com:5050/v2/.
3. Check interval: 60 seconds.
4. Expected status codes: 200, 401 (both indicate a running registry — 401 means auth is required, not that it's down).
5. Label: GitLab container registry.
6. Click Save.

Add a TCP monitor to detect network-level failures on the registry port:

1. Add Monitor → TCP.
2. Host: gitlab.example.com.
3. Port: 5050.
4. Check interval: 60 seconds.
5. Label: GitLab registry TCP (5050).
6. Click Save.

Step 5: Monitor the TLS Certificate

GitLab's HTTPS certificate covers the web UI, Git over HTTPS, and the container registry API. Expiry breaks every Git push, Docker login, and GitLab Runner registration simultaneously:

# Check the certificate expiry date
echo | openssl s_client -connect gitlab.example.com:443 2>/dev/null | \
  openssl x509 -noout -dates
# notAfter=Dec 31 00:00:00 2026 GMT

# Check the registry certificate separately if on a different port
echo | openssl s_client -connect gitlab.example.com:5050 2>/dev/null | \
  openssl x509 -noout -dates

1. Add Monitor → SSL Certificate.
2. Domain: gitlab.example.com.
3. Alert when expiry is within: 30 days.
4. Alert again: 14 days, 7 days, 3 days, 1 day.
5. Click Save.

If the container registry uses a separate certificate:

1. Add Monitor → SSL Certificate.
2. Domain: registry.gitlab.example.com (or gitlab.example.com:5050).
3. Alert when expiry is within: 30 days.
4. Click Save.

Let's Encrypt renewal: If you use gitlab_rails['initial_root_password'] and auto-TLS via the Omnibus package, certificate renewal is managed by the let's encrypt setting in gitlab.rb. A failed renewal can leave the old cert in place until it expires. The Vigilmon 30-day alert gives you enough time to run gitlab-ctl reconfigure and force renewal manually.

Step 6: Heartbeat Monitoring for Scheduled Pipelines

GitLab's pipeline scheduling feature runs pipelines on a cron schedule. If the pipeline scheduler daemon fails or the cron expression is wrong, scheduled pipelines never run — with no alert unless you explicitly monitor them:

# List scheduled pipelines for a project
curl --header "PRIVATE-TOKEN: <token>" \
  "https://gitlab.example.com/api/v4/projects/<id>/pipeline_schedules"
# Check "next_run_at" — if in the past, the schedule is broken

Add a Vigilmon heartbeat to your scheduled pipeline:

1. In Vigilmon → Add Monitor → Cron Heartbeat.
2. Expected interval: match your pipeline schedule (e.g. 60 minutes for hourly).
3. Copy the heartbeat URL: https://vigilmon.online/heartbeat/abc123.
4. In your GitLab CI pipeline (.gitlab-ci.yml), add the heartbeat ping as the final step:

# .gitlab-ci.yml
stages:
  - build
  - test
  - notify

ping-vigilmon:
  stage: notify
  script:
    - curl -s https://vigilmon.online/heartbeat/abc123
  when: on_success
  only:
    - schedules

This ensures the heartbeat fires only when the pipeline completes successfully. If the scheduler goes down, or the pipeline fails before the notify stage, Vigilmon alerts after the expected interval.

For pipelines triggered by external events (not schedules), use the GitLab API in your CI config:

notify-vigilmon:
  stage: notify
  script:
    - |
      if [ "$CI_PIPELINE_SOURCE" = "push" ]; then
        curl -s https://vigilmon.online/heartbeat/abc123
      fi
  when: on_success

Step 7: Monitor the GitLab TCP Port

Add a TCP monitor on port 443 to detect network-layer failures that the HTTP monitor might miss:

# Verify TCP connectivity
nc -zv gitlab.example.com 443
# Expected: Connection to gitlab.example.com 443 port [tcp/https] succeeded!

1. Add Monitor → TCP.
2. Host: gitlab.example.com.
3. Port: 443.
4. Check interval: 60 seconds.
5. Label: GitLab TCP (443).
6. Click Save.

A TCP success with an HTTP failure indicates the network connection is alive but the web service is broken — useful for distinguishing Rails process failures from network/load balancer issues.

Step 8: Configure Alerting

In Vigilmon under Settings → Notifications, configure alert channels and response runbooks:

| Monitor | Trigger | Immediate action | |---|---|---| | GitLab readiness | Non-200 or timeout | Check Rails: sudo gitlab-ctl status puma; check Sidekiq: sudo gitlab-ctl status sidekiq | | GitLab web UI | Non-200 or keyword missing | Check nginx: sudo gitlab-ctl status nginx; check ingress routing | | GitLab Runner metrics | Non-200 or timeout | sudo systemctl status gitlab-runner; check runner logs: sudo journalctl -u gitlab-runner -n 50 | | Container registry HTTP | Non-200/401 | sudo gitlab-ctl status registry; check gitlab_rails['registry_enabled'] in gitlab.rb | | TLS certificate | < 30 days | sudo gitlab-ctl reconfigure; check letsencrypt['enable'] in gitlab.rb | | Scheduled pipeline heartbeat | Missed ping | Check pipeline schedule: gitlab.example.com/<project>/-/pipeline_schedules; check Sidekiq cron jobs | | TCP (443) | Connection refused | Check load balancer/firewall; check sudo gitlab-ctl status nginx |

Alert grouping: Create a gitlab-ci monitor group. When GitLab's Puma web server crashes, the readiness check, web UI monitor, and TCP monitor all fire together — the group view makes the blast radius immediately visible.

Common GitLab CI Failure Modes and What Vigilmon Catches

| Scenario | Vigilmon signal | |---|---| | Rails Puma process OOM-killed | Readiness HTTP fires with 503 | | Sidekiq worker stopped (jobs queue) | Readiness HTTP fires; pipeline jobs stuck pending | | GitLab Runner offline | Runner metrics HTTP fires; pipeline jobs accumulate in pending state | | Container registry service crash | Registry HTTP fires with connection refused | | TLS certificate expired | SSL monitor fires; all Git-over-HTTPS and Docker operations fail | | Let's Encrypt renewal blocked by firewall | SSL monitor fires at 30-day threshold; advance warning to unblock port 80 | | Scheduled pipeline scheduler crash | Heartbeat monitor fires after expected interval | | GitLab upgrade downtime | Readiness and web UI monitors fire during migration window | | PostgreSQL connection failure | Readiness check fails with database component error | | Redis connection failure | Readiness check fails; sessions and queues disrupted | | Ingress controller misconfiguration | TCP monitor stays green; HTTP monitor fires with 502 | | DNS propagation failure after IP change | HTTP and TCP monitors fire; SSL monitor may fire with SNI mismatch |

GitLab CI is the control plane for your entire software delivery pipeline — every commit, every build, every deployment flows through it. Vigilmon gives you external visibility into instance health, runner availability, registry uptime, and certificate validity from outside the cluster. When a Sidekiq worker crash stops pipeline jobs or a runner going offline queues your entire team's work, Vigilmon alerts you before engineers start asking why their pipelines are stuck.

Start monitoring your GitLab CI instance in under 5 minutes — register free at vigilmon.online.