tutorial

How to Monitor BullMQ with Vigilmon

BullMQ powers background job processing for thousands of Node.js applications — but silent worker crashes, stalled jobs, and Redis connection failures can quietly halt your queues. Learn how to monitor queue depth, worker liveness, job failure rates, and Redis health with Vigilmon.

BullMQ is the leading Redis-backed job queue library for Node.js and TypeScript. Built on top of Redis data structures, it provides reliable job processing with features like priority queues, delayed jobs, rate limiting, and repeatable jobs. It's the queue powering background work in everything from email sending pipelines to video transcoding services.

But BullMQ failures are often silent. A worker process that crashes without restarting, a Redis connection that drops quietly, or a job that stalls indefinitely can leave your queue full of unprocessed work while no alerts fire. Vigilmon gives you external visibility into BullMQ health through HTTP monitors for queue health endpoints and heartbeat monitors for worker processes.

Why BullMQ Needs External Monitoring

BullMQ offers built-in queue metrics, but they require active inspection. External monitoring with Vigilmon adds:

  • Proactive alerting when worker processes crash or Redis connections are lost
  • Queue depth threshold checks before backlogs grow out of control
  • Stalled job detection before stuck jobs consume all worker concurrency slots
  • Failed job rate monitoring to catch processing bugs before they fill the failed set
  • Heartbeat monitoring for workers that must prove they are actively dequeuing jobs

Step 1: Build a BullMQ Health Endpoint

Create a lightweight Express health endpoint that queries BullMQ queue stats:

// src/health/bullmq.ts
import { Queue } from 'bullmq';
import { Redis } from 'ioredis';
import express from 'express';

const app = express();

const redis = new Redis({
  host: process.env.REDIS_HOST || 'localhost',
  port: parseInt(process.env.REDIS_PORT || '6379'),
  password: process.env.REDIS_PASSWORD,
  maxRetriesPerRequest: null,
});

const QUEUE_NAMES = (process.env.BULLMQ_QUEUES || 'default').split(',');
const WAITING_THRESHOLD = parseInt(process.env.WAITING_THRESHOLD || '500');
const FAILED_THRESHOLD = parseInt(process.env.FAILED_THRESHOLD || '50');

app.get('/health/bullmq', async (req, res) => {
  const issues: string[] = [];
  const queueStats: Record<string, object> = {};

  // Check Redis connectivity
  try {
    await redis.ping();
  } catch (err) {
    return res.status(503).json({ status: 'down', reason: 'redis_unreachable' });
  }

  for (const name of QUEUE_NAMES) {
    const queue = new Queue(name, { connection: redis });
    try {
      const counts = await queue.getJobCounts(
        'waiting', 'active', 'completed', 'failed', 'delayed', 'paused'
      );
      queueStats[name] = counts;

      if (counts.waiting > WAITING_THRESHOLD) {
        issues.push(`queue_${name}_waiting_${counts.waiting}`);
      }
      if (counts.failed > FAILED_THRESHOLD) {
        issues.push(`queue_${name}_failed_${counts.failed}`);
      }
      if (counts.active === 0 && counts.waiting > 0) {
        issues.push(`queue_${name}_no_active_workers`);
      }
    } catch (err) {
      issues.push(`queue_${name}_unreachable`);
    } finally {
      await queue.close();
    }
  }

  if (issues.length > 0) {
    return res.status(503).json({ status: 'degraded', issues, queues: queueStats });
  }

  return res.status(200).json({ status: 'ok', queues: queueStats });
});

app.listen(3007, () => console.log('BullMQ health server on :3007'));

Run this sidecar alongside your BullMQ workers. It exposes a /health/bullmq endpoint that Vigilmon can probe.

Step 2: Monitor Stalled Jobs

BullMQ marks jobs as stalled when a worker locks a job but fails to renew the lock (typically due to a crash or heavy GC pause). Add a stalled job check:

app.get('/health/bullmq/stalled', async (req, res) => {
  const issues: string[] = [];

  for (const name of QUEUE_NAMES) {
    const queue = new Queue(name, { connection: redis });
    try {
      // Jobs in active state with expired locks appear as stalled
      const active = await queue.getActive();
      const now = Date.now();
      const STALL_THRESHOLD_MS = 60_000; // 1 minute

      const stalled = active.filter(job => {
        const processedOn = job.processedOn ?? 0;
        return now - processedOn > STALL_THRESHOLD_MS;
      });

      if (stalled.length > 0) {
        issues.push(`queue_${name}_stalled_${stalled.length}`);
      }
    } finally {
      await queue.close();
    }
  }

  if (issues.length > 0) {
    return res.status(503).json({ status: 'degraded', issues });
  }
  return res.status(200).json({ status: 'ok' });
});

Step 3: Add Heartbeat Pings to Workers

A healthy queue health endpoint doesn't confirm workers are actually processing jobs. Use Vigilmon heartbeats to verify worker liveness:

// src/workers/email-worker.ts
import { Worker } from 'bullmq';
import { Redis } from 'ioredis';
import axios from 'axios';

const VIGILMON_URL = process.env.VIGILMON_HEARTBEAT_URL!;
let jobsProcessed = 0;
const HEARTBEAT_EVERY = parseInt(process.env.HEARTBEAT_EVERY || '10');

const worker = new Worker(
  'email',
  async (job) => {
    await processEmailJob(job);

    jobsProcessed++;
    if (jobsProcessed % HEARTBEAT_EVERY === 0) {
      axios.get(VIGILMON_URL).catch(() => {});
    }
  },
  {
    connection: new Redis({
      host: process.env.REDIS_HOST || 'localhost',
      maxRetriesPerRequest: null,
    }),
    concurrency: 5,
  }
);

// Also send a heartbeat on a time interval for low-traffic queues
setInterval(async () => {
  const waiting = await worker.getJobCounts?.();
  // Ping heartbeat to confirm worker is alive and polling
  axios.get(VIGILMON_URL).catch(() => {});
}, 60_000);

worker.on('failed', (job, err) => {
  console.error(`Job ${job?.id} failed:`, err);
});

async function processEmailJob(job: any) {
  // your email sending logic
}

For time-based heartbeats (useful when queues are idle for long periods):

// Send heartbeat every minute regardless of job volume
setInterval(() => {
  if (VIGILMON_URL) {
    fetch(VIGILMON_URL).catch(() => {});
  }
}, 60_000);

Step 4: Configure Vigilmon HTTP Monitors

  1. Log in to vigilmon.online and go to Monitors → New Monitor
  2. Choose HTTP / HTTPS
  3. Set the URL to your BullMQ health endpoint: https://your-app.example.com/health/bullmq
  4. Set the check interval to 1 minute
  5. Under Expected response, configure:
    • Status code: 200
    • Response body contains: "status":"ok"
    • Response time threshold: 3000ms
  6. Under Alert channels, assign your Slack or PagerDuty integration
  7. Save the monitor

Add monitors for each concern:

| Monitor URL | Purpose | Interval | |---|---|---| | /health/bullmq | Queue depth, failed count, worker presence | 1 min | | /health/bullmq/stalled | Stalled job detection | 2 min |

Step 5: Configure Heartbeat Monitors for Workers

  1. In Vigilmon, go to Monitors → New Monitor → Heartbeat
  2. Set the name: bullmq-email-worker
  3. Set the expected interval: 2 minutes (ping every job or every minute)
  4. Set the grace period: 5 minutes
  5. Save — copy the heartbeat URL: https://vigilmon.online/heartbeat/abc123xyz
  6. Set VIGILMON_HEARTBEAT_URL in your worker environment

Create one heartbeat monitor per worker type:

| Worker | Heartbeat Interval | Grace Period | |---|---|---| | email-worker | 2 min | 5 min | | notification-worker | 2 min | 5 min | | report-generation-worker | 10 min | 15 min |

Step 6: Alert Routing

| Monitor | Alert Channel | Priority | |---|---|---| | Queue health /health/bullmq | Slack + PagerDuty | P1 | | Stalled jobs /health/bullmq/stalled | Slack | P2 | | Worker heartbeats | Slack + PagerDuty | P1 |

Key thresholds to tune for your workload:

  • Waiting threshold: alert when waiting jobs exceed your normal peak — set this based on typical throughput, not a static number
  • Failed threshold: alert when failed jobs exceed what your retry policy should have caught
  • Stall detection window: 1 minute is a good default; increase if your jobs legitimately run for several minutes

Summary

BullMQ failures are invisible until jobs are overdue and users complain. External monitoring with Vigilmon catches problems before they become incidents:

| Monitor Type | What It Covers | |---|---| | HTTP monitor on /health/bullmq | Queue depth, failure count, worker presence | | HTTP monitor on /health/bullmq/stalled | Stalled job detection | | Heartbeat monitor | Worker process liveness and active dequeuing |

Get started free at vigilmon.online — your first BullMQ monitor is running in under two minutes.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles