Monitoring Supervisor with Vigilmon

Supervisor (supervisord) is the go-to process control system for Unix servers running Python applications, Celery workers, and other background daemons. It starts, stops, and restarts managed processes — but supervisord itself is just another process, and if it crashes or hangs, all your workers stop silently. Vigilmon gives Supervisor the external oversight it needs: probing the XML-RPC web interface, checking supervisorctl-reported status, and sending heartbeat alerts when managed process groups go quiet.

What You'll Set Up

• HTTP uptime monitor for the Supervisor web interface (port 9001)
• XML-RPC endpoint health check for /RPC2
• Keyword-based process status check via supervisorctl output
• Heartbeat monitoring for Supervisor-managed process groups (workers, Celery)
• SSL certificate alerts for proxied Supervisor web interfaces

Prerequisites

• Supervisor 4.0+ installed (pip install supervisor or apt install supervisor)
• supervisord running with the [inet_http_server] section enabled
• A reverse proxy (nginx, Caddy) if using HTTPS
• A free Vigilmon account

Step 1: Enable the Supervisor Web Interface

Add the [inet_http_server] section to /etc/supervisor/supervisord.conf:

[inet_http_server]
port=127.0.0.1:9001
username=admin
password=yourpassword

Reload Supervisor:

sudo supervisorctl reload

Verify the web interface:

curl -u admin:yourpassword http://localhost:9001/
# Returns the Supervisor web UI HTML

For external access, proxy through nginx with TLS rather than exposing port 9001 directly.

Step 2: Monitor the Supervisor Web Interface

Add a Vigilmon HTTP monitor for the Supervisor web dashboard:

1. Log in to vigilmon.online and click Add Monitor.
2. Set Type to HTTP / HTTPS.
3. Enter the URL:
   • Direct: http://your-server:9001
   • Proxied: https://supervisor.yourdomain.com
4. If Supervisor requires HTTP Basic Auth, add credentials via Vigilmon's Custom headers field:

   Authorization: Basic BASE64(admin:yourpassword)
   

5. Set Check interval to 1 minute.
6. Set Expected HTTP status to 200.
7. Click Save.

The Supervisor web interface returns a full HTML page. A 200 response confirms supervisord is running and accepting HTTP connections.

Step 3: Monitor the /RPC2 XML-RPC Endpoint

Supervisor's core API is an XML-RPC interface at /RPC2. Monitoring this endpoint verifies that the RPC layer — which supervisorctl relies on — is functional:

1. Click Add Monitor → HTTP / HTTPS.
2. Set the URL to https://supervisor.yourdomain.com/RPC2.
3. Add the Basic Auth header as in Step 2.
4. Set Expected HTTP status to 200.
5. Enable Keyword check and enter methodResponse as the required string.
6. Set Check interval to 2 minutes.
7. Click Save.

The XML-RPC endpoint responds with an XML envelope even to unauthenticated GET requests (returning a fault code). The presence of methodResponse in the response body confirms the RPC server is alive.

Step 4: supervisorctl Status Keyword Check

Use Vigilmon to monitor supervisorctl's output as served through the web interface. The Supervisor web UI shows process names and their running state. You can check for expected process names in the response:

1. Duplicate the web interface monitor from Step 2.
2. Enable Keyword check and enter a critical process name (e.g., celery or gunicorn).
3. Enable Alert on keyword absence — if your key process name disappears from the status page, something is wrong.

For a more direct approach, use a small script that hits the XML-RPC API and sends a heartbeat only when all processes are RUNNING (see Step 5).

Step 5: Heartbeat Monitoring for Process Groups

Workers and Celery tasks don't expose HTTP endpoints. Use Vigilmon's cron heartbeat to confirm they're running and doing work:

Create heartbeat monitors:

1. In Vigilmon, click Add Monitor → Cron Heartbeat.
2. Create one heartbeat per critical process group:
   • celery-worker — expected interval: 2 minutes
   • gunicorn-app — expected interval: 2 minutes
3. Copy each heartbeat URL.

Ping the heartbeat from a wrapper script:

Create /usr/local/bin/supervisor-heartbeat.sh:

#!/bin/bash
# Check that all required Supervisor processes are RUNNING, then ping heartbeat
CELERY_STATUS=$(supervisorctl status celery 2>/dev/null | grep RUNNING)
if [ -n "$CELERY_STATUS" ]; then
  curl -s https://vigilmon.online/heartbeat/celery-abc123 --max-time 5
fi

GUNICORN_STATUS=$(supervisorctl status gunicorn 2>/dev/null | grep RUNNING)
if [ -n "$GUNICORN_STATUS" ]; then
  curl -s https://vigilmon.online/heartbeat/gunicorn-abc123 --max-time 5
fi

chmod +x /usr/local/bin/supervisor-heartbeat.sh

Add to crontab:

* * * * * /usr/local/bin/supervisor-heartbeat.sh

This pings the heartbeat every minute only when the process is confirmed RUNNING. If Supervisor crashes or the process enters STOPPED/FATAL state, the heartbeat stops and Vigilmon alerts after 2 minutes.

Alternative: ping from within your application:

For long-running workers (Celery beat, queue consumers), ping from the application itself after each task cycle:

import requests

class MonitoredTask(celery.Task):
    def on_success(self, retval, task_id, args, kwargs):
        requests.get(
            'https://vigilmon.online/heartbeat/celery-abc123',
            timeout=5
        )

Step 6: SSL Certificate Alerts for Proxied Setups

If the Supervisor web interface is proxied over HTTPS, monitor the certificate:

1. Open the HTTPS monitor created in Step 2.
2. Scroll to the SSL certificate section.
3. Enable Monitor SSL certificate.
4. Set Alert when certificate expires in less than 21 days.
5. Click Save.

Check certificate expiry manually:

echo | openssl s_client -connect supervisor.yourdomain.com:443 2>/dev/null \
  | openssl x509 -noout -enddate

Step 7: Configure Alert Channels

1. Go to Alert Channels in Vigilmon and add Slack, email, PagerDuty, or a webhook.
2. Group all Supervisor monitors in a Vigilmon project named after the server.
3. Set Consecutive failures before alert to 2 for the web interface monitor.
4. Set Consecutive failures before alert to 1 for heartbeat monitors — a missed heartbeat from a Celery worker means tasks have stopped processing.

Summary

| Monitor | Target | What It Catches | |---|---|---| | Web interface | https://supervisor.yourdomain.com | supervisord crash, port down | | XML-RPC endpoint | /RPC2 with keyword | RPC layer failure | | Process keyword | Process name in web UI | Missing process group | | SSL certificate | HTTPS domain | Certificate expiry | | Cron heartbeat (Celery) | Vigilmon heartbeat URL | Worker stopped processing | | Cron heartbeat (Gunicorn) | Vigilmon heartbeat URL | App server process down |

Supervisor keeps your background processes alive through crashes and deploys — but it needs an external observer. With Vigilmon monitoring the web interface and XML-RPC endpoint from outside your server, and heartbeats confirming each managed process group is actively running, you get complete visibility into your Supervisor-managed stack without any additional infrastructure.