Monitoring Healthchecks.io (Self-Hosted) with Vigilmon

Healthchecks.io is an open-source cron job and scheduled task monitor: each job pings a unique URL on successful completion, and Healthchecks.io alerts when a ping is missed past its grace period. The self-hosted version (Python/Django, BSD license) is functionally identical to the commercial SaaS but runs entirely on your own infrastructure. When the self-hosted Healthchecks instance goes down — container crash, database connection failure, or background worker exit — every monitored cron job stops reporting its liveness status without any alert. The irony is complete: the tool that monitors your scheduled jobs has no one watching it. Vigilmon gives you external monitoring for Healthchecks web UI availability, API health, ping receiver reachability, background worker health, and SSL certificate expiry so your heartbeat monitoring infrastructure is itself reliably monitored.

What You'll Set Up

• Healthchecks web UI availability via HTTP monitor (/login/ page)
• REST API endpoint availability and authenticated response check (/api/v1/checks/)
• Ping receiver reachability via TCP/HTTP check
• Background worker health via self-monitoring canary heartbeat loop
• SSL certificate expiry alerts for HTTPS Healthchecks deployments

Prerequisites

• Healthchecks running via Docker or pip, accessible on port 8000
• A Healthchecks API key (from your Healthchecks account settings)
• A free Vigilmon account

Step 1: Monitor the Healthchecks Web UI

The Healthchecks login page at /login/ is a reliable availability indicator for the entire Django web application layer. If the Django application server (gunicorn or uWSGI) is down, the database is unreachable, or the container has been stopped, the login page stops responding:

1. In Vigilmon, click Add Monitor → HTTP/HTTPS.
2. Enter your Healthchecks URL: http://your-server:8000/login/ (or https://hc.yourdomain.com/login/ if proxied).
3. Set Check interval to 2 minutes.
4. Set Expected HTTP status to 200.
5. Under Response body check, add keyword: Log In (the login form label that appears on all valid Healthchecks login pages).
6. Click Save.

The body keyword check ensures Vigilmon distinguishes between a real Healthchecks login page and a reverse proxy error page — Nginx returning 200 with a "Service Unavailable" body would pass a status-code-only check but fails when the keyword is absent.

Step 2: Monitor the REST API Endpoint

Healthchecks exposes a REST API at /api/v1/checks/ that returns the list of all configured checks for your account. Monitoring this endpoint with an authenticated request confirms both the web application and database layers are functioning — the API requires a database query to return check data:

1. In Vigilmon, click Add Monitor → HTTP/HTTPS.
2. Enter the API URL: http://your-server:8000/api/v1/checks/
3. Set Check interval to 5 minutes.
4. Set Expected HTTP status to 200.
5. Under Request headers, add: X-Api-Key: your-api-key
6. Under Response body check, add keyword: checks (the root key of the JSON response).
7. Click Save.

To retrieve your API key, log in to Healthchecks and navigate to Settings → API Access. Use a read-only API key (scoped to read permissions) for the monitor — it does not need write access.

A valid API response looks like:

{
  "checks": [
    {
      "name": "Daily Backup",
      "slug": "daily-backup",
      "status": "up",
      ...
    }
  ]
}

If the API returns 200 but without the checks key — for example, if a misconfigured reverse proxy is returning a cached error response — the body check catches it.

Step 3: Monitor the Ping Receiver Reachability

The /ping/:uuid/ endpoint is the core of Healthchecks.io — every monitored job sends its success ping to a URL in this format. If the ping receiver endpoint becomes unreachable, all monitored jobs silently fail to report, and Healthchecks.io has no data to alert on. The ping receiver must be externally reachable from wherever your cron jobs run:

1. In Vigilmon, click Add Monitor → HTTP/HTTPS.
2. Enter the ping receiver URL with a valid check UUID: http://your-server:8000/ping/your-check-uuid (use a real UUID from one of your checks).
3. Set Check interval to 2 minutes.
4. Set Expected HTTP status to 200.
5. Click Save.

Get a valid check UUID by navigating to any check in the Healthchecks dashboard and copying the UUID from the ping URL. Sending a live ping via the Vigilmon monitor will also count as a successful ping for that check — which is actually useful for monitoring infrastructure checks that run less frequently than the Vigilmon check interval.

For a non-destructive alternative that only tests reachability without sending an actual ping:

# Test the ping endpoint returns 200 without actually recording a ping
# (Use a check with a very long grace period, or create a dedicated canary check)
curl -s -o /dev/null -w "%{http_code}" http://your-server:8000/ping/your-canary-uuid

Create a dedicated "Vigilmon Connectivity Test" check in Healthchecks with a 1-hour period and 1-hour grace period. Point the Vigilmon monitor at its UUID. Vigilmon's 2-minute checks will keep this canary check green, and you have confirmed ping receiver reachability.

Step 4: Self-Monitoring Loop via Canary Heartbeat

Healthchecks.io's background worker processes ping events and sends alerts when checks go missing. The worker runs as a Django management command (manage.py sendalerts) and is typically invoked by a periodic task or supervisor. If the worker exits silently, Healthchecks.io continues to receive pings and update check statuses in its database — but it never sends the alert emails or webhooks when a check goes missing.

Set up a self-monitoring loop using Vigilmon as the external observer:

1. In Vigilmon, click Add Monitor → Cron Heartbeat.
2. Set Expected ping interval to 5 minutes.
3. Copy the heartbeat URL (e.g., https://vigilmon.online/heartbeat/abc123).

Create a worker health check script on the server running Healthchecks:

#!/bin/bash
# /usr/local/bin/healthchecks-worker-check.sh

HEARTBEAT_URL="https://vigilmon.online/heartbeat/abc123"

# Check that the sendalerts worker process is running
WORKER_RUNNING=$(docker compose -f /path/to/docker-compose.yml ps worker 2>/dev/null | grep -c "Up" || \
  pgrep -f "manage.py sendalerts" | wc -l)

if [ "$WORKER_RUNNING" -gt 0 ]; then
  echo "Healthchecks worker running"
  curl -s "$HEARTBEAT_URL"
else
  echo "Healthchecks worker not running - alerts will not fire"
  exit 1
fi

Install as a cron job:

chmod +x /usr/local/bin/healthchecks-worker-check.sh
(crontab -l 2>/dev/null; echo "*/5 * * * * /usr/local/bin/healthchecks-worker-check.sh >> /var/log/healthchecks-worker-check.log 2>&1") | crontab -

The deeper self-monitoring loop is: configure a Healthchecks check for the healthchecks-worker-check.sh cron job itself (the script pings its own Healthchecks check on success), and also have Vigilmon watch the Vigilmon heartbeat. This creates a three-way monitoring chain: cron job → Healthchecks.io → Vigilmon. If the cron job fails to ping Healthchecks, Healthchecks.io alerts. If the Healthchecks worker that processes those pings fails, Vigilmon alerts. If Vigilmon can't reach Healthchecks at all, Vigilmon alerts independently.

Step 5: SSL Certificate Monitoring

For Healthchecks deployments proxied behind Nginx or Caddy, an expired SSL certificate blocks all external ping deliveries — every cron job running on a remote server that tries to ping https://hc.yourdomain.com/ping/uuid will receive a certificate error and fail silently. Set up SSL monitoring:

1. In Vigilmon, click Add Monitor → SSL Certificate.
2. Enter your Healthchecks hostname: hc.yourdomain.com
3. Set Alert when certificate expires in less than 14 days.
4. Set Check interval to 1 day.
5. Click Save.

An expired certificate is particularly dangerous for Healthchecks because many cron jobs that send pings may be configured to fail silently on ping delivery errors (using curl -s without error checking). Those jobs continue running successfully while their pings stop being received — Healthchecks will eventually alert the check as missed, but only after the grace period expires, which may be hours or days later.

Step 6: Set Up Alert Channels

Configure Vigilmon to route Healthchecks monitoring alerts appropriately:

1. Go to Alert Channels in Vigilmon.
2. Add PagerDuty for the background worker monitor — a dead worker means no alerts fire for any monitored job.
3. Add a Slack webhook to #infrastructure-alerts for web UI and ping receiver monitors.
4. Add email notification for the SSL certificate monitor.
5. Set Consecutive failures before alert to 2 for the web UI monitor to ignore transient container restarts.

Route Vigilmon alerts for Healthchecks to a channel that is completely independent of Healthchecks itself. If Healthchecks is down, alerts configured through Healthchecks will not fire — Vigilmon's independent monitoring and alerting ensures you are notified regardless of Healthchecks' availability.

Summary

| Monitor | Target | What It Catches | |---|---|---| | HTTP monitor (web UI) | /login/ + body check | Django app down, database unreachable, container crash | | HTTP monitor (API) | /api/v1/checks/ + auth header | API processing failure, database connection loss | | HTTP monitor (ping receiver) | /ping/:uuid/ | Ping endpoint unreachable from external jobs | | Cron heartbeat (worker) | sendalerts process check | Worker exit, silent alert delivery failure | | SSL monitor | hc.yourdomain.com | Certificate expiry blocking external pings |

The most dangerous failure mode for self-hosted Healthchecks is a dead sendalerts worker: pings arrive, the database records them correctly, but no alerts fire when checks go missing. Without external monitoring of the worker process, you would only discover this failure when a monitored job silently stops running and no alert arrives — potentially days later. Vigilmon closes this loop, making the monitoring infrastructure itself observable.

Start monitoring for free at vigilmon.online.