Monitoring Cabot with Vigilmon

Cabot is an open source alert management and on-call scheduling platform that gives you Nagios/PagerDuty-style alerting for your self-hosted infrastructure. It runs a Python/Django web application on port 5000 with Celery workers executing the actual health checks in the background, backed by Redis and PostgreSQL. The irony of running a monitoring system is real: if Cabot itself goes down, your entire alerting pipeline goes silent. Vigilmon closes that gap, independently monitoring every layer of Cabot so your on-call system has its own watchdog.

What You'll Set Up

• Web application availability monitor on port 5000
• Celery worker health monitoring via heartbeat
• Redis queue broker connectivity checks
• PostgreSQL database connectivity monitoring
• Check execution health (HTTP, TCP, and Jenkins checks)
• Alert delivery pipeline health for PagerDuty, Slack, and HipChat
• Graphite metrics data source connectivity
• API endpoint response time monitoring

Prerequisites

• Cabot installed and accessible at http://your-server:5000
• Celery workers running (typically via supervisord or systemd)
• Redis and PostgreSQL running as backing services
• A free Vigilmon account

Step 1: Monitor the Cabot Web Application

Cabot's Django web app is the interface for managing on-call schedules, configuring checks, and viewing alert history. Downtime here means your team can't update schedules or acknowledge incidents.

1. Log in to vigilmon.online and click Add Monitor.
2. Set Type to HTTP / HTTPS.
3. Enter your Cabot URL: http://your-server:5000.
4. Set Check interval to 1 minute.
5. Set Expected HTTP status to 200.
6. Click Save.

Add a keyword check to verify the app is fully rendered (not a Django error page):

1. Open the monitor settings.
2. Under Content verification, enter Cabot or on-call as a required keyword.
3. Save — this catches cases where Django is running but serving 500 error pages.

Also monitor the login endpoint to confirm the authentication flow is alive:

1. Click Add Monitor → HTTP / HTTPS.
2. URL: http://your-server:5000/accounts/login/.
3. Expected status: 200.
4. Keyword: username or password (confirms the login form rendered).

Step 2: Monitor Celery Worker Health

Celery workers are the most critical component of Cabot — they execute every check (HTTP, TCP, Jenkins) on schedule. If the workers die, your checks stop running silently. Cabot's web UI will still appear healthy while alerts are never triggered.

Expose a Celery worker health endpoint by adding a management command or a simple probe script:

# cabot/management/commands/worker_health.py
from django.core.management.base import BaseCommand
from celery.app.control import Control
from cabot.celery import app

class Command(BaseCommand):
    def handle(self, *args, **kwargs):
        inspect = app.control.inspect(timeout=2)
        active = inspect.active()
        if active:
            self.stdout.write('ok')
        else:
            self.stderr.write('no workers')
            exit(1)

Wrap it in a lightweight HTTP health server and monitor http://localhost:9001/celery-health expecting 200 ok. Alternatively, use Vigilmon's cron heartbeat approach: configure the Celery beat scheduler to send a ping to Vigilmon after each successful scheduling loop.

# In your Celery beat task
from celery import shared_task
import requests

@shared_task
def heartbeat_ping():
    requests.get('https://vigilmon.online/heartbeat/abc123', timeout=5)

Add this task to CELERYBEAT_SCHEDULE at a 5-minute interval. In Vigilmon:

1. Click Add Monitor → Cron Heartbeat.
2. Set the expected interval to 5 minutes.
3. Paste the heartbeat URL into your task. If Celery beat stops scheduling tasks, the heartbeat stops and Vigilmon alerts.

Step 3: Monitor Redis Queue Broker

Cabot uses Redis as its Celery message broker. If Redis goes down, Celery workers stop receiving tasks and all checks halt immediately.

1. Click Add Monitor → TCP Port.
2. Host: localhost (or your Redis host), Port: 6379.
3. Set Check interval to 1 minute.
4. Click Save.

For a richer Redis health signal, add a script that runs PING and exposes the result:

#!/bin/bash
# /opt/cabot/scripts/redis-health.sh
result=$(redis-cli -h localhost -p 6379 PING 2>&1)
if [ "$result" = "PONG" ]; then
  echo "HTTP/1.1 200 OK\r\nContent-Type: text/plain\r\n\r\nok"
else
  echo "HTTP/1.1 503 Service Unavailable\r\nContent-Type: text/plain\r\n\r\nerror"
fi

Serve this via ncat or a minimal daemon and monitor the HTTP endpoint for ok.

Step 4: Monitor PostgreSQL Database Connectivity

Cabot stores all check results, alert history, on-call schedules, and user accounts in PostgreSQL. A database failure makes Cabot completely inoperable.

1. Click Add Monitor → TCP Port.
2. Host: localhost (or your PostgreSQL host), Port: 5432.
3. Set Check interval to 1 minute.

Add a deeper database probe using a sidecar script:

#!/bin/bash
# /opt/cabot/scripts/db-health.sh
PGPASSWORD="$CABOT_DB_PASS" psql -U cabot -h localhost -d cabot \
  -c "SELECT 1;" > /dev/null 2>&1
echo $?

Expose the result via HTTP and set Vigilmon to expect 0 in the response body (or a 200 status from a wrapper), ensuring Cabot can actually query — not just that the port is open.

Step 5: Monitor Check Execution Health

Cabot runs three types of checks against your infrastructure: HTTP checks, TCP checks, and Jenkins checks. These run inside Celery workers on a schedule. Beyond monitoring Celery itself, confirm that checks are actually executing by monitoring Cabot's check status API:

1. Click Add Monitor → HTTP / HTTPS.
2. URL: http://your-server:5000/api/v1/status-checks/.
3. Set an API key header: Authorization: ApiKey your-key-here.
4. Expected status: 200.
5. Set Check interval to 5 minutes.

This probe confirms the API is responding and Cabot's check registry is accessible. If the response contains "status": "FAILING" for checks that should be passing, that's a signal from Cabot itself — but the endpoint being reachable at all confirms the execution pipeline is alive.

Step 6: Monitor Alert Delivery Channels

Cabot sends alerts through PagerDuty, Slack, HipChat, and email. A misconfiguration or third-party API change can silently break alert delivery — your checks still run, but no one gets notified.

Test alert delivery health with a canary check. Create a low-priority Cabot check that monitors a Vigilmon endpoint — effectively creating a loop:

1. In Cabot, add an HTTP check targeting a Vigilmon status page URL.
2. Set it to a very generous threshold so it only alerts on genuine Vigilmon downtime.
3. Configure it to deliver via your primary alert channel (Slack or PagerDuty).

If this canary check fires but you don't receive the alert, your delivery channel is broken — the issue is in Cabot's notification pipeline, not in the monitored service.

For each active alert channel in Cabot, add a TCP/HTTP monitor to the external service:

| Channel | Monitor Target | |---|---| | PagerDuty | https://events.pagerduty.com (HTTP, expect 404 at root) | | Slack | https://slack.com/api/api.test (HTTP, expect 200) | | Email/SMTP | SMTP server TCP port 587 or 465 |

Step 7: Monitor Graphite Data Source and API Endpoints

Cabot can use Graphite metric data to evaluate thresholds. If your Graphite instance is unreachable, metric-based checks will fail or behave unpredictably.

1. Click Add Monitor → HTTP / HTTPS.
2. URL: http://your-graphite:8080/render (or your Graphite render endpoint).
3. Expected status: 200 or 400 (a missing query parameter returns 400, which still proves Graphite is alive).
4. Set Check interval to 2 minutes.

Also monitor Cabot's own REST API response times for each critical endpoint:

http://your-server:5000/api/v1/services/
http://your-server:5000/api/v1/status-checks/

Set a Response time threshold of 2000ms in each monitor. Slow API responses often indicate database query bottlenecks before they escalate to full outages.

Summary

| Monitor | Target | What It Catches | |---|---|---| | Web app | http://your-server:5000 | Django crash, reverse proxy failure | | Login endpoint | /accounts/login/ | Auth system broken | | Celery heartbeat | Heartbeat URL | Workers stopped, beat scheduler failed | | Redis (TCP) | localhost:6379 | Broker down, check queue halted | | PostgreSQL (TCP) | localhost:5432 | Database down, history/schedule lost | | Check execution API | /api/v1/status-checks/ | API unavailable | | PagerDuty (HTTP) | events.pagerduty.com | Alert delivery broken | | Graphite (HTTP) | Graphite render endpoint | Metric checks failing silently | | API response times | /api/v1/services/ | Slow DB queries before crash |

Cabot is designed to be your infrastructure's watchdog — but a watchdog that goes silent is worse than no watchdog at all. With Vigilmon independently monitoring every layer of the Cabot stack, from Celery workers to alert delivery channels, you can trust that your on-call system will actually page someone when something goes wrong.