tutorial

How to Monitor Graphile Worker with Vigilmon

Graphile Worker runs background jobs directly in PostgreSQL — but silent worker crashes, job failures, and queue backlog growth can stall your pipeline with no external signal. Learn how to monitor job throughput, worker liveness, queue depth, and Postgres connectivity with Vigilmon.

Graphile Worker is a job queue for Node.js that uses PostgreSQL as its backing store — no Redis, no RabbitMQ, no external broker needed. Jobs are rows in a Postgres table, workers are Node.js processes polling with LISTEN/NOTIFY, and the entire queue is as durable as your database. It's widely used in Postgraphile-powered applications and any Node.js project already committed to PostgreSQL.

But Graphile Worker failures are quiet. A Node.js worker that exits without supervisor restart, a Postgres connection pool that times out, or a job that accumulates permanent failures can silently drain throughput while your database shows the evidence in a graphile_worker.jobs table nobody is watching. Vigilmon gives you external visibility through HTTP monitors for queue health and heartbeat monitors for worker processes.

Why Graphile Worker Needs External Monitoring

Graphile Worker stores all state in PostgreSQL, giving you full ACID guarantees — but it offers no built-in alerting. External monitoring with Vigilmon adds:

  • Proactive alerting when worker Node.js processes exit or Postgres connectivity is lost
  • Queue depth threshold checks before pending jobs pile up beyond worker capacity
  • Failure rate monitoring to catch task types with consistently failing handlers
  • Locked job detection to identify jobs stuck in the running state after a worker crash
  • Heartbeat monitoring for worker processes that must prove they are polling and executing

Step 1: Build a Graphile Worker Health Endpoint

Create a Node.js Express health server that queries the graphile_worker schema directly:

// src/health/graphile-worker.ts
import express from 'express';
import { Pool } from 'pg';

const app = express();

const pool = new Pool({
  connectionString: process.env.DATABASE_URL,
  max: 3,
});

const PENDING_THRESHOLD = parseInt(process.env.GW_PENDING_THRESHOLD || '500');
const FAILED_THRESHOLD = parseInt(process.env.GW_FAILED_THRESHOLD || '50');
const LOCKED_STALE_MINUTES = parseInt(process.env.GW_LOCKED_STALE_MINUTES || '10');

app.get('/health/graphile-worker', async (req, res) => {
  const issues: string[] = [];

  // Check Postgres connectivity
  let client;
  try {
    client = await pool.connect();
  } catch (err) {
    return res.status(503).json({ status: 'down', reason: 'postgres_unreachable' });
  }

  try {
    // Queue depth by task identifier
    const { rows: pendingRows } = await client.query(`
      SELECT task_identifier, count(*) as count
      FROM graphile_worker.jobs
      WHERE locked_at IS NULL
        AND run_at <= NOW()
        AND attempts < max_attempts
      GROUP BY task_identifier
      ORDER BY count DESC
    `);

    const pending: Record<string, number> = {};
    let totalPending = 0;
    for (const row of pendingRows) {
      pending[row.task_identifier] = parseInt(row.count);
      totalPending += parseInt(row.count);
    }

    if (totalPending > PENDING_THRESHOLD) {
      issues.push(`pending_backlog_${totalPending}`);
    }

    // Failed jobs (exhausted all attempts)
    const { rows: failedRows } = await client.query(`
      SELECT task_identifier, count(*) as count
      FROM graphile_worker.jobs
      WHERE attempts >= max_attempts
      GROUP BY task_identifier
    `);

    const failed: Record<string, number> = {};
    let totalFailed = 0;
    for (const row of failedRows) {
      failed[row.task_identifier] = parseInt(row.count);
      totalFailed += parseInt(row.count);
    }

    if (totalFailed > FAILED_THRESHOLD) {
      issues.push(`failed_jobs_${totalFailed}`);
    }

    // Stale locked jobs (locked longer than threshold — indicate crashed worker)
    const { rows: staleRows } = await client.query(`
      SELECT count(*) as count
      FROM graphile_worker.jobs
      WHERE locked_at IS NOT NULL
        AND locked_at < NOW() - INTERVAL '${LOCKED_STALE_MINUTES} minutes'
    `);
    const staleCount = parseInt(staleRows[0].count);
    if (staleCount > 0) {
      issues.push(`stale_locked_jobs_${staleCount}`);
    }

    // Active workers (recently heartbeated)
    const { rows: workerRows } = await client.query(`
      SELECT count(*) as count
      FROM graphile_worker.known_crons
      WHERE last_execution > NOW() - INTERVAL '5 minutes'
    `).catch(() => ({ rows: [{ count: 0 }] }));

    const activeWorkers = parseInt(workerRows[0].count);

    if (issues.length > 0) {
      return res.status(503).json({
        status: 'degraded',
        issues,
        stats: { totalPending, totalFailed, staleCount, activeWorkers, pending, failed },
      });
    }

    return res.status(200).json({
      status: 'ok',
      stats: { totalPending, totalFailed, staleCount, activeWorkers, pending, failed },
    });
  } finally {
    client.release();
  }
});

app.listen(3009, () => console.log('Graphile Worker health server on :3009'));

Step 2: Add Heartbeat Pings to Workers

Wire Vigilmon heartbeat pings into your Graphile Worker task handlers:

// src/workers/index.ts
import { run, makeWorkerUtils } from 'graphile-worker';
import fetch from 'node-fetch';

const VIGILMON_URL = process.env.VIGILMON_HEARTBEAT_URL;
let jobsCompleted = 0;
const HEARTBEAT_EVERY = parseInt(process.env.HEARTBEAT_EVERY || '10');

function pingVigilmon() {
  if (!VIGILMON_URL) return;
  fetch(VIGILMON_URL).catch((err) => {
    console.error('Vigilmon heartbeat failed:', err.message);
  });
}

// Heartbeat on a timer for idle queues
setInterval(pingVigilmon, 60_000);

async function main() {
  const runner = await run({
    connectionString: process.env.DATABASE_URL,
    concurrency: 10,
    pollInterval: 1000,
    taskList: {
      send_email: async (payload, helpers) => {
        await sendEmail(payload as any);

        jobsCompleted++;
        if (jobsCompleted % HEARTBEAT_EVERY === 0) {
          pingVigilmon();
        }
      },

      generate_report: async (payload, helpers) => {
        await generateReport(payload as any);
        pingVigilmon(); // Always ping after long-running jobs
      },
    },
  });

  // Handle graceful shutdown
  process.on('SIGTERM', () => runner.stop());
  await runner.promise;
}

async function sendEmail(payload: { to: string; subject: string }) {
  // email sending logic
}

async function generateReport(payload: { reportId: string }) {
  // report generation logic
}

main().catch((err) => {
  console.error(err);
  process.exit(1);
});

Step 3: Monitor the graphile_worker.jobs Table Directly

For teams already running pgMonitor or Prometheus, Graphile Worker's queue state is a simple SQL query:

-- Pending jobs by type
SELECT task_identifier, count(*) AS pending
FROM graphile_worker.jobs
WHERE locked_at IS NULL
  AND run_at <= NOW()
  AND attempts < max_attempts
GROUP BY task_identifier;

-- Failed jobs (no retries left)
SELECT task_identifier, count(*) AS failed
FROM graphile_worker.jobs
WHERE attempts >= max_attempts
GROUP BY task_identifier;

-- Stale locked jobs (crashed workers leave these)
SELECT id, task_identifier, locked_at, attempts
FROM graphile_worker.jobs
WHERE locked_at IS NOT NULL
  AND locked_at < NOW() - INTERVAL '10 minutes';

These queries can also feed a Grafana dashboard if you're using a Postgres data source.

Step 4: Configure Vigilmon HTTP Monitors

  1. Log in to vigilmon.online and go to Monitors → New Monitor
  2. Choose HTTP / HTTPS
  3. Set the URL to your health endpoint: https://your-app.example.com/health/graphile-worker
  4. Set the check interval to 1 minute
  5. Under Expected response, configure:
    • Status code: 200
    • Response body contains: "status":"ok"
    • Response time threshold: 3000ms
  6. Under Alert channels, assign your Slack or PagerDuty integration
  7. Save the monitor

| Monitor URL | Purpose | Interval | |---|---|---| | /health/graphile-worker | Pending backlog, failed jobs, stale locks | 1 min |

Step 5: Configure Heartbeat Monitors

  1. In Vigilmon, go to Monitors → New Monitor → Heartbeat
  2. Set the name: graphile-worker-process
  3. Set the expected interval: 2 minutes
  4. Set the grace period: 5 minutes
  5. Save — copy the heartbeat URL: https://vigilmon.online/heartbeat/abc123xyz
  6. Set VIGILMON_HEARTBEAT_URL in your worker environment

| Worker Instance | Heartbeat Interval | Grace Period | |---|---|---| | worker-1 | 1 min | 3 min | | worker-2 | 1 min | 3 min | | cron-tasks | 5 min | 10 min |

Step 6: Alert Routing

| Monitor | Alert Channel | Priority | |---|---|---| | Queue health /health/graphile-worker | Slack + PagerDuty | P1 | | Worker heartbeats | Slack + PagerDuty | P1 |

Key failure modes to alert on:

  • Stale locked jobs: the clearest signal of a crashed worker; Graphile Worker will reclaim these after its unlock delay, but you want to know before that delay expires
  • Failed job accumulation: jobs at attempts >= max_attempts will never retry — these are permanent failures and need human attention
  • Heartbeat silence: if the worker process exits and your process supervisor doesn't restart it, Vigilmon fires before a single job deadline is missed

Summary

Graphile Worker's PostgreSQL backing means your jobs are as durable as your database — but it doesn't mean failures are visible. Vigilmon surfaces them externally before they become customer-facing incidents:

| Monitor Type | What It Covers | |---|---| | HTTP monitor on /health/graphile-worker | Queue depth, failed jobs, stale locked tasks | | Heartbeat monitor | Worker Node.js process liveness and active polling |

Get started free at vigilmon.online — your first Graphile Worker monitor is running in under two minutes.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles