Monitoring ArchiveBox with Vigilmon

ArchiveBox turns URLs into permanent local archives using wget, headless Chrome screenshots, PDF export, WARC files, and more — so you never lose a web page to link rot. But because ArchiveBox runs as a Django application with background archiving workers, silent failures are common: the web interface stays up while the archiving pipeline has stalled. Vigilmon gives you end-to-end coverage: Django server health, admin backend availability, REST API liveness, SSL certificate expiry, and a heartbeat that fires only when the archiving queue is actually processing new URLs.

What You'll Set Up

• HTTP monitor for the ArchiveBox public web interface (port 8000)
• HTTP monitor for the Django admin backend (/admin/)
• HTTP monitor for the REST API endpoint (/api/v1/)
• SSL certificate expiry alerts for HTTPS ArchiveBox deployments
• Heartbeat monitor that confirms the archiving worker pipeline is accepting and processing URLs

Prerequisites

• ArchiveBox deployed and accessible (Docker-based is typical; web interface on port 8000)
• An ArchiveBox admin account and API key (or username/password for API access)
• A free Vigilmon account

Step 1: Monitor the ArchiveBox Web Interface

The ArchiveBox public web interface is served by Django on port 8000. A GET / returns the public archive page or redirects to the login page — a 200 response confirms the Django application server is running and connected to the database.

1. Log in to vigilmon.online and click Add Monitor.
2. Set Type to HTTP / HTTPS.
3. Enter your ArchiveBox URL: http://your-server:8000 (or https://archive.yourdomain.com if reverse-proxied).
4. Set Check interval to 1 minute.
5. Set Expected HTTP status to 200.
6. Click Save.

If ArchiveBox is configured to redirect unauthenticated visitors to /login/, set the expected status to 200 and enable Follow redirects — Vigilmon will follow the redirect and check the final page.

Step 2: Monitor the Django Admin Backend

The ArchiveBox admin panel at /admin/ confirms not just that Django is running, but that the Django ORM has a healthy database connection. A GET /admin/ returns the Django administration login page containing the text "Django administration".

1. Click Add Monitor in Vigilmon.
2. Set Type to HTTP / HTTPS.
3. Enter the admin URL: http://your-server:8000/admin/
4. Set Expected HTTP status to 200.
5. Under Response body check, enter Django administration (or ArchiveBox) to verify the admin UI loaded correctly rather than returning an error page.
6. Set Check interval to 5 minutes.
7. Click Save.

The body keyword check is what separates this monitor from a simple TCP probe — if Django is misconfigured or the database is unreachable, the page will return an error instead of the admin login form.

Step 3: Monitor the REST API Endpoint

ArchiveBox ships with a Django REST Framework API at /api/v1/. A GET /api/v1/ returns the DRF API root (a JSON object listing available endpoints), and /api/v1/docs/ returns the Swagger UI. Either confirms the DRF layer is wired up correctly.

1. Click Add Monitor in Vigilmon.
2. Set Type to HTTP / HTTPS.
3. Enter the API URL: http://your-server:8000/api/v1/
4. Set Expected HTTP status to 200.
5. Under Response body check, enter api to confirm a JSON API response rather than an error page.
6. Set Check interval to 5 minutes.
7. Click Save.

If your ArchiveBox instance requires authentication for the API root, use /api/v1/docs/ instead — the Swagger UI is typically accessible without credentials and still confirms DRF is healthy.

Step 4: SSL Certificate Alerts for HTTPS Deployments

ArchiveBox is commonly reverse-proxied behind nginx or Caddy, which handles TLS termination. The reverse proxy's Let's Encrypt certificate must stay valid — an expired certificate locks every user and automated client out of the archive.

1. Open the HTTPS monitor for your ArchiveBox domain (created in Step 1).
2. Enable Monitor SSL certificate under the SSL settings.
3. Set Alert when certificate expires in less than 21 days.
4. Click Save.

For nginx-based deployments, verify your renewal cron job or systemd timer is active:

# Check certbot renewal timer
systemctl status certbot.timer

# Test renewal without issuing a new certificate
certbot renew --dry-run

A 21-day alert window gives you three weeks to investigate and manually renew before users see certificate errors.

Step 5: Heartbeat Monitoring for the Archiving Worker Pipeline

The ArchiveBox web interface staying up does not mean archiving is working — background workers can stall silently while the Django server continues to respond. The REST API POST /api/v1/add/ endpoint submits URLs to the archiving queue; a successful response confirms the API is accepting new URLs and the task queue is reachable.

Create the heartbeat monitor first:

1. In Vigilmon, click Add Monitor → Cron Heartbeat.
2. Set Expected ping interval to 60 minutes.
3. Copy the heartbeat URL (e.g. https://vigilmon.online/heartbeat/abc123).

Then set up a cron job on the ArchiveBox host to submit a test URL via the API and ping Vigilmon on success:

#!/bin/bash
# /etc/cron.hourly/archivebox-heartbeat

ARCHIVEBOX_URL="http://localhost:8000"
ARCHIVEBOX_API_KEY="your-api-key-here"
VIGILMON_HEARTBEAT="https://vigilmon.online/heartbeat/abc123"

# Submit a lightweight test URL to the archiving queue
RESPONSE=$(curl -s -o /dev/null -w "%{http_code}" \
  -X POST "${ARCHIVEBOX_URL}/api/v1/add/" \
  -H "Authorization: Token ${ARCHIVEBOX_API_KEY}" \
  -H "Content-Type: application/json" \
  -d '{"urls": ["https://example.com"], "tag": "healthcheck"}')

# Only ping Vigilmon if the API accepted the URL (201 Created)
if [ "$RESPONSE" = "201" ] || [ "$RESPONSE" = "200" ]; then
  curl -s "${VIGILMON_HEARTBEAT}" > /dev/null
fi

Make the script executable and install it:

chmod +x /etc/cron.hourly/archivebox-heartbeat

If you use Docker, run the script from a sidecar container or via docker exec:

docker exec archivebox python -c "
import requests, os
resp = requests.post(
    'http://localhost:8000/api/v1/add/',
    headers={'Authorization': 'Token ' + os.environ['ARCHIVEBOX_API_KEY']},
    json={'urls': ['https://example.com'], 'tag': 'healthcheck'}
)
if resp.status_code in (200, 201):
    requests.get('https://vigilmon.online/heartbeat/abc123', timeout=5)
"

If the archiving worker stalls or the task queue becomes unreachable, the POST /api/v1/add/ will fail and Vigilmon will alert after the heartbeat interval passes.

Step 6: Configure Alert Channels

1. Go to Alert Channels in Vigilmon and add Slack, email, or a webhook.
2. For the web interface and admin monitors, set Consecutive failures before alert to 2 — transient Django startup delays can cause a single probe to fail.
3. For the heartbeat monitor, a single missed ping is sufficient to alert — set Grace period to 0.

Summary

| Monitor | Target | What It Catches | |---|---|---| | Web interface | http://your-server:8000/ | Django server down, database unreachable | | Admin backend | /admin/ | Django ORM failure, misconfiguration | | REST API | /api/v1/ | DRF layer broken, API routing misconfigured | | SSL certificate | Archive domain | Reverse proxy certificate expiry | | Cron heartbeat | Heartbeat URL | Archiving worker stalled, task queue failure |

ArchiveBox is only as reliable as its archiving pipeline — a running web interface with a stalled worker means you're accumulating URLs in a queue but nothing is being preserved. With Vigilmon watching the Django stack from multiple angles and a heartbeat that proves the archiving pipeline is end-to-end healthy, you'll know immediately when your archive stops working rather than discovering it weeks later when you need a saved page.