tutorial

How to Monitor Your Self-Hosted Frappe / ERPNext Instance (Workers, Redis, Database, and More)

Frappe and ERPNext have a lot of moving parts — web workers, background job workers, Redis queues, MariaDB, file storage, and email. Here's how to monitor them all and get alerted before users notice a problem.

How to Monitor Your Self-Hosted Frappe / ERPNext Instance (Workers, Redis, Database, and More)

Frappe and ERPNext are among the most operationally complex open source applications you can self-host. A production Frappe deployment involves a Gunicorn web server, multiple background job workers (default, long, short, and scheduler queues), Redis for caching and queueing, MariaDB, file storage, and a transactional email queue. When any of these fails, business operations — invoicing, payroll, inventory, purchasing — break in ways that aren't always obvious.

This guide covers comprehensive monitoring for every layer of your Frappe/ERPNext stack.

Where Frappe deployments break

Web server failures — Gunicorn crashes or the Nginx proxy fails. Users see connection refused or a 502 Bad Gateway. No self-healing; requires manual restart.

Background worker failures — Frappe uses RQ (Redis Queue) workers for background jobs. Worker crashes leave jobs stuck in the queue. Email queues stop draining. Scheduled reports don't run. The UI shows no indication — document status just stops changing.

Redis queue failures — Redis is both the session cache and the job queue backend. A Redis failure stops all background processing and degrades session performance. This is one of the highest-impact single-point failures in a Frappe deployment.

MariaDB connection pool exhaustion — under heavy load, MariaDB connections can be exhausted. Requests start failing with connection errors. The pool rarely recovers without intervention.

File storage failures — Frappe stores attachments, print formats, and backup files on the local filesystem (or S3/NFS in enterprise setups). A full disk or broken mount causes upload failures that surface as cryptic errors to users.

Email delivery queue stalls — Frappe's email queue runs through background workers. If workers stop, emails pile up invisibly. Users expect notifications that never arrive.

Webhook delivery failures — outbound webhooks (to Zapier, Slack, or internal services) go through the same worker queues. Silent worker failures kill webhook delivery.

Step 1: Enable and test Frappe's health endpoint

Recent versions of Frappe include a built-in health endpoint at /api/method/frappe.monitor.health:

curl https://erp.yourdomain.com/api/method/frappe.monitor.health
# {"message":{"status":"healthy","worker":"alive","scheduler":"alive"}}

If you're on an older Frappe version, create a custom whitelisted method:

# apps/your_app/your_app/api/health.py
import frappe
import redis

@frappe.whitelist(allow_guest=True)
def health():
    checks = {}
    
    # Database check
    try:
        frappe.db.sql("SELECT 1")
        checks['database'] = 'ok'
    except Exception:
        checks['database'] = 'error'

    # Redis cache check
    try:
        cache = frappe.cache()
        cache.set('health_check', 'ok', ex=30)
        val = cache.get('health_check')
        checks['redis_cache'] = 'ok' if val == b'ok' else 'error'
    except Exception:
        checks['redis_cache'] = 'error'

    # Redis queue check
    try:
        import redis as redis_lib
        conf = frappe.conf
        r = redis_lib.Redis(
            host=conf.get('redis_queue', 'redis://localhost:11000').split('/')[-2].split(':')[0],
            port=int(conf.get('redis_queue', 'redis://localhost:11000').split(':')[-1]),
        )
        r.ping()
        checks['redis_queue'] = 'ok'
    except Exception:
        checks['redis_queue'] = 'error'

    # File storage check
    site_path = frappe.get_site_path()
    checks['file_storage'] = 'ok' if frappe.os.access(site_path, frappe.os.W_OK) else 'error'

    status_ok = all(v == 'ok' for v in checks.values())
    frappe.response['http_status_code'] = 200 if status_ok else 500
    return {
        'status': 'healthy' if status_ok else 'degraded',
        'checks': checks
    }

Register the endpoint in your hooks.py:

# apps/your_app/your_app/hooks.py
override_whitelisted_methods = {
    "your_app.api.health.health": "your_app.api.health.health"
}

Step 2: Set up HTTP monitoring in Vigilmon

With health endpoints available, configure monitoring in Vigilmon:

  1. Sign up at vigilmon.online
  2. Click New Monitor → HTTP

Create these monitors:

| Monitor name | URL | Expected | |---|---|---| | Frappe Web Server | https://erp.yourdomain.com/api/method/frappe.monitor.health | 200 + "status":"healthy" | | ERPNext Login Page | https://erp.yourdomain.com/login | 200 | | Frappe REST API | https://erp.yourdomain.com/api/resource/DocType | 200 or 403 (alive, needs auth) | | Frappe Desk | https://erp.yourdomain.com/app | 200 |

The REST API check accepting a 403 (Forbidden) works the same way as the OAuth check pattern — a 403 means the API is alive and enforcing auth. A 500 means the Python layer is broken.

Step 3: Monitor background job workers

Frappe background workers are the most common source of silent failures. When they stop, the application appears functional but stops processing jobs.

Worker heartbeat via a recurring scheduled task

Create a Frappe scheduled task that pings a heartbeat URL:

# apps/your_app/your_app/tasks/heartbeat.py
import frappe
import requests

def ping_worker_heartbeat():
    """Called by the Frappe scheduler. Pings Vigilmon to confirm workers are alive."""
    heartbeat_url = frappe.conf.get('vigilmon_worker_heartbeat_url')
    if heartbeat_url:
        try:
            requests.get(heartbeat_url, timeout=5)
        except Exception:
            frappe.log_error("Failed to ping worker heartbeat", "Vigilmon")

Register it in hooks.py:

# apps/your_app/your_app/hooks.py
scheduler_events = {
    "all": [
        "your_app.tasks.heartbeat.ping_worker_heartbeat"
    ]
}

The all event fires every minute. In Vigilmon, set the heartbeat monitor interval to 5 minutes (with tolerance for scheduler jitter). If the worker queue stops processing, this task stops running and Vigilmon alerts you within 5 minutes.

Set the heartbeat URL in your site config:

# bench set-config -g vigilmon_worker_heartbeat_url "https://vigilmon.online/heartbeats/your-token"

Or edit sites/common_site_config.json:

{
  "vigilmon_worker_heartbeat_url": "https://vigilmon.online/heartbeats/your-worker-token"
}

Step 4: Separate heartbeats for each worker queue

Frappe runs workers on multiple named queues: default, long, short, and scheduler. A failure in the long queue (which handles large report generation) won't affect the default queue — so you want separate heartbeats:

# apps/your_app/your_app/tasks/heartbeat.py
import frappe
import requests

def ping_default_heartbeat():
    _ping(frappe.conf.get('vigilmon_default_heartbeat_url'))

def ping_long_heartbeat():
    _ping(frappe.conf.get('vigilmon_long_heartbeat_url'))

def ping_short_heartbeat():
    _ping(frappe.conf.get('vigilmon_short_heartbeat_url'))

def _ping(url):
    if url:
        try:
            requests.get(url, timeout=5)
        except Exception:
            pass

Register each in hooks.py with appropriate frequencies:

scheduler_events = {
    "all": [
        "your_app.tasks.heartbeat.ping_default_heartbeat",
        "your_app.tasks.heartbeat.ping_short_heartbeat",
    ],
    "hourly": [
        "your_app.tasks.heartbeat.ping_long_heartbeat",
    ],
}

Configure all three in common_site_config.json:

{
  "vigilmon_default_heartbeat_url": "https://vigilmon.online/heartbeats/token-default",
  "vigilmon_long_heartbeat_url": "https://vigilmon.online/heartbeats/token-long",
  "vigilmon_short_heartbeat_url": "https://vigilmon.online/heartbeats/token-short"
}

In Vigilmon, create a heartbeat monitor for each with appropriate intervals (2 minutes for all-queue heartbeats, 90 minutes for long-queue).

Step 5: Monitor Redis connectivity

Redis failures in Frappe are catastrophic. Monitor Redis connectivity directly with a cron-based heartbeat:

#!/bin/bash
# /opt/frappe-jobs/check-redis.sh

REDIS_CACHE_PORT=${REDIS_CACHE_PORT:-13000}
REDIS_QUEUE_PORT=${REDIS_QUEUE_PORT:-11000}
REDIS_SOCKETIO_PORT=${REDIS_SOCKETIO_PORT:-12000}

ALL_OK=true

# Check Redis cache
if ! redis-cli -p $REDIS_CACHE_PORT ping | grep -q PONG; then
    echo "[$(date)] Redis cache (port $REDIS_CACHE_PORT) NOT responding" >&2
    ALL_OK=false
fi

# Check Redis queue
if ! redis-cli -p $REDIS_QUEUE_PORT ping | grep -q PONG; then
    echo "[$(date)] Redis queue (port $REDIS_QUEUE_PORT) NOT responding" >&2
    ALL_OK=false
fi

if [ "$ALL_OK" = true ]; then
    curl -fsS --retry 3 "$VIGILMON_REDIS_HEARTBEAT_URL" > /dev/null 2>&1
    echo "[$(date)] Redis health check passed"
fi

*/5 * * * * frappe /opt/frappe-jobs/check-redis.sh >> /var/log/frappe/redis-health.log 2>&1

Step 6: Database connection pool monitoring

MariaDB connection pool exhaustion causes request failures across the entire Frappe application. Monitor it with a health check that probes connection availability:

# apps/your_app/your_app/tasks/db_health.py
import frappe
import requests

def check_database_pool():
    """Checks DB connectivity and connection count. Runs every 5 minutes."""
    try:
        # Test basic connectivity
        frappe.db.sql("SELECT 1")

        # Check active connection count (requires PROCESS privilege)
        result = frappe.db.sql(
            "SELECT COUNT(*) as conn_count FROM information_schema.PROCESSLIST WHERE db = %s",
            (frappe.conf.db_name,),
            as_dict=True
        )
        conn_count = result[0]['conn_count'] if result else 0

        # Warn if connections are high (tune this threshold for your setup)
        max_connections = int(frappe.conf.get('max_db_connections', 100))
        if conn_count < max_connections * 0.9:  # < 90% of max
            heartbeat_url = frappe.conf.get('vigilmon_db_heartbeat_url')
            if heartbeat_url:
                requests.get(heartbeat_url, timeout=5)
        else:
            frappe.log_error(
                f"DB connection pool near limit: {conn_count}/{max_connections}",
                "DB Health Warning"
            )
    except Exception as e:
        frappe.log_error(str(e), "DB Health Check Failed")

Register in hooks.py:

scheduler_events = {
    "cron": {
        "*/5 * * * *": ["your_app.tasks.db_health.check_database_pool"]
    }
}

Step 7: Email delivery queue heartbeat

Frappe's email queue runs through background workers. Monitor whether emails are actually being sent:

# apps/your_app/your_app/tasks/email_health.py
import frappe
import requests
from datetime import datetime, timedelta

def check_email_queue():
    """Alerts if emails are stuck in the queue for > 30 minutes."""
    cutoff = datetime.now() - timedelta(minutes=30)

    stuck_emails = frappe.db.count('Email Queue', {
        'status': 'Not Sent',
        'creation': ['<', cutoff]
    })

    if stuck_emails == 0:
        # Queue is draining — send heartbeat
        heartbeat_url = frappe.conf.get('vigilmon_email_heartbeat_url')
        if heartbeat_url:
            try:
                requests.get(heartbeat_url, timeout=5)
            except Exception:
                pass
    else:
        frappe.log_error(
            f"{stuck_emails} email(s) stuck in queue for > 30 minutes",
            "Email Queue Alert"
        )

Register in hooks.py:

scheduler_events = {
    "cron": {
        "*/15 * * * *": ["your_app.tasks.email_health.check_email_queue"]
    }
}

Step 8: File storage and report rendering heartbeats

File storage

#!/bin/bash
# /opt/frappe-jobs/check-file-storage.sh

SITE_PATH="/home/frappe/frappe-bench/sites/your-site.com"
PRIVATE_DIR="$SITE_PATH/private/files"
PUBLIC_DIR="$SITE_PATH/public/files"
MIN_FREE_GB=5

FREE_BYTES=$(df --output=avail "$SITE_PATH" | tail -1)
FREE_GB=$(( FREE_BYTES / 1024 / 1024 ))

if [ "$FREE_GB" -gt "$MIN_FREE_GB" ] && [ -w "$PRIVATE_DIR" ] && [ -w "$PUBLIC_DIR" ]; then
    curl -fsS "$VIGILMON_STORAGE_HEARTBEAT_URL" > /dev/null 2>&1
else
    echo "[$(date)] File storage check failed. Free: ${FREE_GB}GB, writable: $([ -w $PRIVATE_DIR ] && echo yes || echo NO)" >&2
fi

*/15 * * * * frappe /opt/frappe-jobs/check-file-storage.sh >> /var/log/frappe/storage.log 2>&1

Report rendering heartbeat

# apps/your_app/your_app/tasks/report_health.py
import frappe
import requests

def check_report_rendering():
    """Attempts to render a known-good report. Pings heartbeat on success."""
    try:
        # Render a lightweight built-in report
        result = frappe.get_doc('Report', 'ToDo').get_data()
        if result is not None:
            heartbeat_url = frappe.conf.get('vigilmon_reports_heartbeat_url')
            if heartbeat_url:
                requests.get(heartbeat_url, timeout=5)
    except Exception as e:
        frappe.log_error(str(e), "Report Rendering Health Check Failed")

Webhook delivery heartbeat

# apps/your_app/your_app/tasks/webhook_health.py
import frappe
import requests
from datetime import datetime, timedelta

def check_webhook_queue():
    """Alerts if webhooks are stuck in the queue."""
    cutoff = datetime.now() - timedelta(minutes=15)

    stuck = frappe.db.count('Webhook Log', {
        'status': 'Error',
        'creation': ['>', cutoff]
    })

    if stuck < 3:  # Tolerate occasional individual failures
        heartbeat_url = frappe.conf.get('vigilmon_webhook_heartbeat_url')
        if heartbeat_url:
            try:
                requests.get(heartbeat_url, timeout=5)
            except Exception:
                pass
    else:
        frappe.log_error(f"{stuck} webhook failures in last 15 minutes", "Webhook Health Alert")

Step 9: Alerts configuration

Configure notification channels in Vigilmon:

Slack:

  1. Create an incoming webhook in your Slack ops channel
  2. Go to Notifications → New Channel → Slack in Vigilmon
  3. Paste the webhook URL
  4. Enable on all Frappe monitors

Email:

  1. Go to Notifications → New Channel → Email
  2. Add your ERP admin or IT ops distribution list
  3. Enable on critical monitors (web server, Redis, database, default worker)

A worker heartbeat miss alert:

🔴 MISSED: Frappe Default Worker
Expected every: 2 minutes
Last received: 12 minutes ago
Action: Check bench logs — run "bench doctor" to inspect worker status

Supervision and auto-restart (complementary to monitoring)

Pair your monitoring with proper process supervision so failures self-heal faster:

# Check Frappe process status via supervisor
supervisorctl status | grep frappe

# Or via systemd (bench managed)
systemctl status frappe-default-worker@0.service
systemctl status frappe-schedule.service

Monitoring catches the failure; supervision auto-restarts the process. Both are necessary — supervision without monitoring means you don't know how often workers are crashing.

Complete monitoring coverage

| Component | Monitoring method | |---|---| | Web server (Gunicorn + Nginx) | HTTP monitor on /api/method/frappe.monitor.health | | Default queue workers | Heartbeat via scheduler task (every 1 min) | | Long queue workers | Heartbeat via scheduler task (hourly) | | Short queue workers | Heartbeat via scheduler task (every 1 min) | | Redis cache | Heartbeat via cron CLI check | | Redis queue | Heartbeat via cron CLI check | | MariaDB connection pool | Heartbeat via scheduler task (every 5 min) | | File storage | Heartbeat via cron disk check | | Email delivery queue | Heartbeat: alert if queue stalls > 30 min | | Report rendering | Heartbeat via scheduled report test | | Webhook delivery | Heartbeat: alert on repeated failures |

Next steps

  • Add a Frappe bench backup job heartbeat — backups are your last line of defense, and backup failures are almost never noticed until you need the backup
  • Monitor your MariaDB slow query log volume as a leading indicator of database stress before it reaches connection pool limits
  • Set up a synthetic login flow as a canary: if a user can log in and reach the desk, your full Frappe stack is functional

Get started free at vigilmon.online.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles