tutorial

Monitoring Grafana OnCall with Vigilmon

Grafana OnCall is your self-hosted on-call management and incident response platform — but it has no built-in uptime dashboard. Here's how to monitor every critical component, from the alert processing engine to notification delivery, with Vigilmon.

Grafana OnCall is an open-source on-call management platform that centralizes alerting, escalation policies, and on-call schedules across your engineering team. Deployed as a Python/Django + Go hybrid (engine on port 8080, web UI proxied through Grafana on port 3000), it sits in the critical path of your incident response workflow. When OnCall goes down, alerts don't route, pages don't fire, and incidents go unacknowledged. Vigilmon keeps watch over every OnCall component so your on-call tooling never silently fails.

What You'll Set Up

  • HTTP availability monitor for the OnCall web UI (via Grafana)
  • Alert group processing engine health check
  • PostgreSQL database connectivity monitor
  • Redis cache and Celery task queue health
  • Outbound notification delivery (Telegram, Slack, PagerDuty) monitors
  • Inbound webhook alert receiver health check
  • On-call schedule calculation service monitor
  • Escalation policy execution engine monitor
  • Mobile push notification relay monitor

Prerequisites

  • Grafana OnCall deployed (Docker Compose or Kubernetes) with the engine accessible
  • Grafana running on port 3000 with the OnCall plugin installed
  • A free Vigilmon account

Step 1: Monitor the Grafana OnCall Web UI

The OnCall plugin surfaces through the Grafana web interface. Monitor the Grafana instance to catch any UI availability issues:

  1. Log in to vigilmon.online and click Add Monitor.
  2. Set Type to HTTP / HTTPS.
  3. Enter your Grafana URL: https://grafana.yourdomain.com/api/health.
  4. Set Check interval to 1 minute.
  5. Set Expected HTTP status to 200.
  6. Click Save.

The /api/health endpoint returns a JSON payload with Grafana's database status — a richer signal than a static page check:

{"commit":"abc123","database":"ok","version":"10.x.x"}

If database is not ok, the Grafana backend is degraded and OnCall data may be unavailable.

Step 2: Monitor the Alert Processing Engine

The OnCall engine (Go service, port 8080 by default) processes incoming alert groups and drives escalations. Probe its health endpoint directly:

  1. Add a new HTTP / HTTPS monitor.
  2. URL: http://oncall-engine:8080/health/ (or your publicly routable equivalent).
  3. Set Expected HTTP status to 200.
  4. Set Check interval to 1 minute.

If the engine is behind a private network, expose the health endpoint through your reverse proxy with restricted access, or use Vigilmon's agent-based monitoring if your architecture supports it.

The engine health endpoint returns:

{"status": "ok"}

A non-200 response or connection timeout means alert routing is broken — a severity-1 condition worth an immediate page.

Step 3: Monitor PostgreSQL Connectivity

OnCall stores alert groups, schedules, escalation policies, and user data in PostgreSQL. Use a health check endpoint that validates the DB connection:

Add a /ready or /healthz probe to your OnCall setup that confirms the database is reachable. If you use the official OnCall Docker Compose setup, the engine exposes /ready:

  1. Add a new HTTP / HTTPS monitor.
  2. URL: http://oncall-engine:8080/ready/.
  3. Expected status: 200.
  4. Interval: 1 minute.

Alternatively, monitor PostgreSQL directly at the TCP level:

  1. Add a new TCP monitor.
  2. Host: your-postgres-host, Port: 5432.
  3. Interval: 1 minute.

A TCP failure here means OnCall cannot persist alert data and escalations will fail.

Step 4: Monitor Redis and the Celery Task Queue

OnCall uses Redis as a message broker for Celery workers that handle async tasks (notification dispatching, schedule recalculation, webhook delivery). Monitor Redis availability:

  1. Add a TCP monitor.
  2. Host: your-redis-host, Port: 6379.
  3. Interval: 1 minute.

For deeper Celery queue health, expose a queue-depth endpoint from your application and monitor it with an HTTP check. If the Celery queue depth grows beyond a threshold, workers are falling behind on notification dispatch:

# Example Django view to expose queue depth
from django.http import JsonResponse
from redis import Redis

def celery_health(request):
    r = Redis.from_url(settings.CELERY_BROKER_URL)
    queue_length = r.llen('celery')
    status = 'ok' if queue_length < 1000 else 'degraded'
    return JsonResponse({'status': status, 'queue_length': queue_length})

Add an HTTP monitor pointing to this endpoint with expected status 200 and a keyword check for "ok".

Step 5: Monitor Outbound Notification Delivery

OnCall sends alerts via Telegram, Slack, PagerDuty, and SMS. While you cannot directly probe third-party APIs, you can monitor the connector services and use Vigilmon cron heartbeats to verify end-to-end delivery.

Heartbeat for notification dispatch

In your OnCall Celery task that dispatches notifications, ping a Vigilmon heartbeat URL on each successful batch:

import requests

def dispatch_notifications_task():
    # ... dispatch logic ...
    notify_users()
    # Ping Vigilmon heartbeat after successful dispatch
    requests.get('https://vigilmon.online/heartbeat/YOUR_HEARTBEAT_ID', timeout=5)
  1. In Vigilmon, click Add Monitor → Cron Heartbeat.
  2. Set the expected interval to match your notification dispatch frequency (e.g. 5 minutes).
  3. Paste the heartbeat URL into your task.

If notifications stop dispatching for any reason — Celery crash, Redis failure, API rate limit — the heartbeat goes silent and Vigilmon alerts you.

Step 6: Monitor the Inbound Webhook Alert Receiver

OnCall receives alerts from Prometheus Alertmanager, Grafana alerts, and other sources via inbound webhooks. These endpoints must stay available or alerts never enter OnCall.

  1. Add an HTTP / HTTPS monitor.
  2. URL: https://oncall.yourdomain.com/integrations/v1/alertmanager/YOUR_TOKEN/.
  3. Set Expected HTTP status to 405 (Method Not Allowed — GET on a POST-only endpoint, which confirms the endpoint exists and is routing correctly).
  4. Interval: 1 minute.

If the webhook receiver returns a 502, 503, or times out, incoming alerts from Alertmanager and other sources are being dropped.

Step 7: Monitor the On-Call Schedule Service

On-call schedule calculation — determining who is on-call right now — runs as part of the OnCall engine. Verify schedule resolution is working by probing the schedules API:

curl -H "Authorization: Bearer YOUR_API_TOKEN" \
  https://oncall.yourdomain.com/api/v1/schedules/ \
  -o /dev/null -w "%{http_code}"

Add an HTTP monitor for the schedules API endpoint with expected status 200. If schedules fail to resolve, your escalation policies will page the wrong people (or nobody).

Step 8: Monitor Escalation Policy Execution

Escalation policies are the heart of OnCall. Add a cron heartbeat that fires whenever the escalation engine processes a cycle:

# In your escalation processing Celery beat task
def process_escalations():
    run_escalation_cycle()
    requests.get('https://vigilmon.online/heartbeat/ESCALATION_HEARTBEAT_ID', timeout=5)
  1. Add a Cron Heartbeat monitor in Vigilmon.
  2. Set the interval to match your escalation cycle frequency (typically 1 minute).

Step 9: Configure Alerts and Summary

Set up alert channels in Vigilmon (Alert Channels → Add Slack or email) and configure thresholds:

  • Consecutive failures before alert: 2 for web UI monitors (allow one transient failure).
  • Consecutive failures before alert: 1 for the alert processing engine — any failure is critical.
  • Use Maintenance windows during OnCall upgrades to suppress false alerts.

| Monitor | Target | What It Catches | |---|---|---| | Grafana web UI | /api/health | Grafana backend failure, DB disconnect | | Engine health | :8080/health/ | Alert processing engine down | | Engine readiness | :8080/ready/ | PostgreSQL connectivity lost | | Redis TCP | :6379 | Cache/broker unavailable, Celery stalled | | Notification heartbeat | Heartbeat URL | Notification dispatch stopped | | Webhook receiver | Alertmanager integration URL | Inbound alerts dropped | | Schedules API | /api/v1/schedules/ | Schedule resolution broken | | Escalation heartbeat | Heartbeat URL | Escalation engine stalled |

Grafana OnCall is your incident response nervous system — when it fails, your team loses visibility at exactly the wrong moment. With Vigilmon watching every layer from the web UI to the Celery workers, you'll know about OnCall issues before they silently swallow the next real incident.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles