Monitoring Mayan EDMS with Vigilmon

Mayan EDMS gives you a full enterprise document management platform on your own infrastructure: OCR, document conversion, full-text search, granular permissions, and workflow automation. The system runs across several interdependent processes — a Django web server, Celery workers, a Redis broker, and a search backend — and a failure in any one can silently stall document processing while the web UI stays up. Vigilmon monitors each layer so you catch failures before they turn into a backlog of unprocessed documents.

What You'll Set Up

• Web application availability monitor
• Celery worker queue health via the admin API
• Redis broker TCP connectivity check
• OCR and document conversion pipeline health check
• Database connectivity verification
• Search index and file storage monitoring
• Cron heartbeat for background tasks

Prerequisites

• Mayan EDMS 4.4+ running on port 80 (or 443)
• Admin credentials for the Mayan web UI
• A free Vigilmon account

Step 1: Monitor the Web Application

Mayan EDMS serves its web interface through Gunicorn behind an nginx reverse proxy on port 80. The root URL redirects to the login page, which confirms Django and the web server are running.

1. Log in to vigilmon.online and click Add Monitor.
2. Set Type to HTTP / HTTPS.
3. Enter: http://your-mayan-host/ (or https:// if TLS is configured)
4. Set Check interval to 1 minute.
5. Set Expected HTTP status to 200 (the login page returns 200).
6. Click Save.

For a more precise health check, add Mayan's built-in health endpoint if you have a custom middleware or proxy probe configured. Otherwise the root URL is the standard signal.

Step 2: Add an Application Health Endpoint

Mayan EDMS does not ship a /health endpoint by default, but you can add one with a minimal Django view. In your Mayan custom apps directory:

# mayan_health/views.py
from django.http import JsonResponse
from django.db import connection

def health(request):
    try:
        connection.ensure_connection()
        db_ok = True
    except Exception:
        db_ok = False
    status = 200 if db_ok else 503
    return JsonResponse({'db': db_ok}, status=status)

# mayan_health/urls.py
from django.urls import path
from . import views
urlpatterns = [path('health/', views.health)]

Register the URL pattern in Mayan's main urls.py and restart Gunicorn. Then update your Vigilmon monitor to probe http://your-mayan-host/health/ with expected status 200. This check exercises Django + the database in a single probe.

Step 3: Monitor Redis Broker Connectivity

Mayan uses Redis as the Celery message broker. If Redis goes down, Celery workers stop receiving tasks and the web UI queues tasks that never run.

1. Click Add Monitor → TCP Port.
2. Host: your Redis host (localhost or a dedicated Redis server IP).
3. Port: 6379 (default Redis port).
4. Check interval: 1 minute.
5. Click Save.

A TCP connection failure on port 6379 means Celery workers are already disconnected, even if the Mayan web interface is still accepting uploads.

Step 4: Monitor the Celery Worker Queue

Mayan exposes Celery task queue information through its web interface. Create a dedicated API token and probe the admin endpoint:

1. Log in to Mayan as an administrator.
2. Navigate to API → API Tokens and generate a token.
3. In Vigilmon, add an HTTP / HTTPS monitor:
   • URL: http://your-mayan-host/api/v4/celery/worker_status/
   • Add header Authorization: Token YOUR_API_TOKEN
   • Expected HTTP status: 200
   • Expected body contains: "online"
   • Check interval: 2 minutes
4. Click Save.

If all Celery workers are offline, the response will not contain "online" and Vigilmon will alert.

Step 5: Monitor the OCR Processing Pipeline

OCR is one of the most resource-intensive tasks in Mayan. A crashed OCR worker means documents sit with empty text content. Monitor the OCR queue length via the API:

1. Add an HTTP / HTTPS monitor.
2. URL: http://your-mayan-host/api/v4/document_states/
3. Header: Authorization: Token YOUR_API_TOKEN
4. Expected HTTP status: 200
5. Check interval: 5 minutes
6. Click Save.

This confirms the document state API (which backs OCR status) is responding. For deeper queue monitoring, consider pairing Vigilmon with a Celery Flower instance and monitoring its API endpoint.

Step 6: Monitor the Database Connectivity

Mayan EDMS uses PostgreSQL by default. A database failure brings down the entire application — but sometimes the Django process keeps running while connection attempts fail silently.

Add a TCP monitor for your PostgreSQL port:

1. Click Add Monitor → TCP Port.
2. Host: your PostgreSQL host.
3. Port: 5432.
4. Check interval: 1 minute.
5. Click Save.

If you deployed the /health/ endpoint from Step 2, this is partially redundant — but a dedicated TCP check on port 5432 catches PostgreSQL failures independently of the Django process.

Step 7: Monitor the Search Index

Mayan EDMS supports multiple search backends (Elasticsearch, Whoosh, Django ORM search). The search API endpoint confirms the index is responding:

1. Add an HTTP / HTTPS monitor.
2. URL: http://your-mayan-host/api/v4/search/
3. Header: Authorization: Token YOUR_API_TOKEN
4. Expected HTTP status: 200
5. Check interval: 5 minutes
6. Click Save.

A 500 or 503 from this endpoint typically means the search backend (Elasticsearch, if configured) is unreachable.

Step 8: Celery Worker Heartbeat

Wrap your Celery workers in a heartbeat ping to Vigilmon so you know immediately if a worker process exits:

1. In Vigilmon, click Add Monitor → Cron Heartbeat.
2. Expected interval: 5 minutes.
3. Copy the heartbeat URL.
4. Create a periodic Celery task that pings the heartbeat:

# tasks.py
import requests
from celery import shared_task

@shared_task
def vigilmon_heartbeat():
    requests.get('https://vigilmon.online/heartbeat/abc123', timeout=5)

Register it in CELERYBEAT_SCHEDULE:

CELERYBEAT_SCHEDULE = {
    'vigilmon-heartbeat': {
        'task': 'mayan.apps.myapp.tasks.vigilmon_heartbeat',
        'schedule': 300,  # every 5 minutes
    },
}

If any Celery beat or worker process crashes, the heartbeat stops and Vigilmon alerts after 5 minutes.

Step 9: Configure Alert Channels

1. Go to Alert Channels in Vigilmon and add Slack, email, or PagerDuty.
2. For the web UI monitor, set Consecutive failures before alert to 2 — Gunicorn restarts can cause a brief blip.
3. For the Redis TCP monitor, set Consecutive failures before alert to 1 — Redis downtime immediately stalls all document processing.
4. For the Celery heartbeat, set Consecutive failures before alert to 1 — a missed heartbeat means workers are not processing.

Summary

| Monitor | Target | What It Catches | |---|---|---| | Web application | /:80 or /health/ | Django crash, nginx failure | | Redis broker | :6379 TCP | Celery task delivery failure | | Celery worker status | /api/v4/celery/worker_status/ | All workers offline | | OCR/document state API | /api/v4/document_states/ | OCR pipeline stall | | PostgreSQL | :5432 TCP | Database connectivity failure | | Search index | /api/v4/search/ | Full-text search failure | | Celery heartbeat | Heartbeat URL | Worker process crash |

Mayan EDMS is built for reliability at enterprise scale, but self-hosted deployments own their own uptime. With Vigilmon covering every layer — web, broker, workers, database, and search — you get immediate alerts when any component fails, long before document processing queues overflow.