tutorial

Monitoring Prometheus with Vigilmon: Web UI, Health & Ready Endpoints, Scrape Target Health & SSL Alerts

How to monitor Prometheus with Vigilmon — web UI availability, /-/healthy and /-/ready endpoint checks, scrape target health, and SSL certificate monitoring.

Prometheus is the metrics backbone of most cloud-native observability stacks. It scrapes targets, evaluates alerting rules, and serves PromQL queries to Grafana dashboards. When Prometheus itself goes down, you don't just lose metrics — you lose your alerting system. AlertManager stops receiving fire-and-forget alerts, Grafana dashboards go blank, and your on-call team loses visibility at exactly the moment they need it most. Vigilmon monitors Prometheus externally so your alerting infrastructure has its own independent health check.

What You'll Build

  • A monitor on Prometheus's web UI to detect process crashes
  • /-/healthy and /-/ready endpoint checks for the two distinct health states
  • A scrape target health check via the targets API
  • SSL certificate monitoring for your Prometheus domain
  • Alert routing that distinguishes startup from operational failures

Prerequisites

  • Prometheus running (v2.x) accessible over HTTP or HTTPS
  • A domain or IP for Prometheus (e.g., https://prometheus.example.com or http://host:9090)
  • A free account at vigilmon.online

Step 1: Understand Prometheus's Health Endpoints

Prometheus exposes two health endpoints with distinct meanings:

| Endpoint | HTTP 200 means | HTTP non-200 means | |---|---|---| | /-/healthy | Prometheus process is running | Binary has crashed or is in a fatal state | | /-/ready | Prometheus finished startup; it's accepting queries | Still loading TSDB, replaying WAL, or in shutdown |

A Prometheus instance can be healthy but not ready — for example, during startup when it's replaying the write-ahead log. During that window, the process is alive but Grafana queries return errors. Monitor both endpoints to distinguish these states.

Verify both endpoints:

curl -o /dev/null -s -w "%{http_code}" http://localhost:9090/-/healthy
# → 200

curl -o /dev/null -s -w "%{http_code}" http://localhost:9090/-/ready
# → 200 (ready) or 503 (not ready yet)

Step 2: Create the /-/healthy Monitor

  1. Log in to Vigilmon → Add Monitor → HTTP.
  2. URL: https://prometheus.example.com/-/healthy.
  3. Check interval: 60 seconds.
  4. Response timeout: 10 seconds.
  5. Expected status: 200.
  6. Keyword: Prometheus is Healthy. (Prometheus returns this exact string).
  7. Label: Prometheus healthy.
  8. Click Save.

The keyword check matters here: some reverse proxies return HTTP 200 for all requests even when Prometheus is down. Checking for the exact response body Prometheus is Healthy. confirms the Prometheus process answered the request, not a proxy serving a cached or fallback page.

This monitor catches:

  • Prometheus process crash or OOM kill
  • Systemd service failures
  • Container restarts mid-scrape
  • Fatal configuration errors on reload (kill -HUP)

Step 3: Create the /-/ready Monitor

  1. Add Monitor → HTTP.
  2. URL: https://prometheus.example.com/-/ready.
  3. Check interval: 60 seconds.
  4. Response timeout: 15 seconds.
  5. Expected status: 200.
  6. Keyword: Prometheus is Ready.
  7. Label: Prometheus ready.
  8. Click Save.

Alert threshold: Set this monitor to alert after 2 consecutive failures. A single /-/ready failure may occur during a configuration reload (TSDB compaction can make Prometheus briefly not-ready). Two consecutive failures (2 minutes at 60-second intervals) indicates a genuine problem — a failed restart, a corrupt TSDB block, or a WAL replay that's stuck.

When the /-/healthy monitor is green but /-/ready fires, Prometheus is alive but not serving queries — check for TSDB corruption, a stuck WAL replay, or a storage issue.

Step 4: Monitor Scrape Target Health

Prometheus exposes its current scrape target state via the API:

curl https://prometheus.example.com/api/v1/targets | python3 -m json.tool | grep health | head -10

The response includes each target's health field:

{
  "status": "success",
  "data": {
    "activeTargets": [
      {
        "labels": {"job": "node", "instance": "host1:9100"},
        "health": "up",
        "lastError": ""
      },
      {
        "labels": {"job": "postgres", "instance": "db1:9187"},
        "health": "down",
        "lastError": "connection refused"
      }
    ]
  }
}

Create a monitor on the targets API to confirm Prometheus can reach its targets:

  1. Add Monitor → HTTP.
  2. URL: https://prometheus.example.com/api/v1/targets.
  3. Check interval: 2 minutes.
  4. Response timeout: 15 seconds.
  5. Expected status: 200.
  6. Keyword: "status":"success" (confirms the API responded correctly).
  7. Label: Prometheus targets API.
  8. Click Save.

Per-target alerting: Vigilmon's targets API monitor confirms Prometheus is functioning and can enumerate its targets. For per-target down alerts, use Prometheus's own up == 0 alerting rule in AlertManager — Vigilmon is the right tool for Prometheus's own health, while Prometheus is the right tool for scrape target failures.

Step 5: Monitor the Web UI

The Prometheus web UI at the root path (/) serves the expression browser and target list. Monitoring it separately catches web layer failures (misconfigured reverse proxy, TLS issues) that might not affect the /-/healthy endpoint directly:

  1. Add Monitor → HTTP.
  2. URL: https://prometheus.example.com/.
  3. Check interval: 5 minutes.
  4. Response timeout: 15 seconds.
  5. Expected status: 200.
  6. Keyword: Prometheus Time Series Collection.
  7. Label: Prometheus web UI.
  8. Click Save.

This is a coarser check — a 5-minute interval is enough since web UI failures without a corresponding health endpoint failure usually indicate a proxy issue, not a Prometheus crash.

Step 6: Monitor SSL Certificates

Prometheus is often accessed by Grafana, AlertManager, and other internal services over HTTPS. Certificate expiry causes:

  • Grafana's Prometheus data source to fail TLS verification (all dashboards go blank)
  • AlertManager's prometheus_alertmanager_alerts scrape to fail
  • Any federated Prometheus instances to lose their upstream connection
  1. Add Monitor → SSL Certificate.
  2. Domain: prometheus.example.com.
  3. Alert when expiry is within: 30 days.
  4. Alert again at: 14 days, 7 days, 3 days.
  5. Click Save.

Step 7: Configure Alerting

In Vigilmon under Settings → Notifications, configure your escalation channels:

| Monitor | Trigger | Response | |---|---|---| | /-/healthy | Non-200 or keyword missing | Prometheus process down; check systemctl status prometheus | | /-/ready | 2+ consecutive failures | TSDB issue or stuck WAL; check Prometheus logs | | Targets API | Non-200 or keyword missing | API layer degraded; Grafana queries failing | | Web UI | Non-200 or timeout | Reverse proxy issue; Prometheus may still be healthy | | SSL certificate | < 30 days to expiry | Renew cert; Grafana data source will break on expiry |

Alert routing: Send /-/healthy failures directly to your on-call channel — Prometheus being down means no alerts from your entire stack until it recovers. For /-/ready and scrape API failures, a team Slack channel with lower urgency is appropriate.

Common Prometheus Failure Modes and What Vigilmon Catches

| Scenario | Vigilmon monitor | |---|---| | Prometheus OOM killed (large TSDB) | /-/healthy fires within 60 s | | TSDB corruption after crash | /-/ready fires; /-/healthy may pass | | WAL replay stuck after unclean shutdown | /-/ready fires for extended period | | Config reload failure (--web.enable-lifecycle) | Both endpoints may still pass; check reload API separately | | Disk full (TSDB can't write blocks) | /-/healthy may pass briefly; then process exits | | Reverse proxy 502 (Prometheus restarting) | Web UI and API monitors fire; health endpoints may pass | | SSL certificate expires | SSL monitor alerts; Grafana data source breaks | | Remote write queue stalls | Not caught by endpoint checks; use Prometheus alerts on prometheus_remote_storage_queue_highest_sent_timestamp_seconds | | Alerting rules fail to evaluate | Not caught by endpoint checks; AlertManager fires no alerts until Prometheus recovers |

Prometheus is the core of most alerting stacks — which means it's the component most dangerous to leave unmonitored. A down Prometheus instance silences all your other alerts. Vigilmon provides independent, external checks on Prometheus's health, readiness, and API availability from outside your observability stack. You'll know Prometheus is down before your team notices that all the dashboards went blank.

Monitor your Prometheus instance free — register at vigilmon.online.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles