How to Monitor Solid Queue with Vigilmon

Solid Queue ships as the default background job backend in Rails 8. Instead of Redis, it uses your existing PostgreSQL or MySQL database to persist jobs — eliminating a separate queue dependency while retaining all the reliability guarantees you need for production background processing.

But database-backed queues have their own failure modes. A dispatcher process that exits silently, a growing pile of failed jobs, or jobs stuck in a claimed state can all halt background processing while your app continues serving HTTP traffic without complaint. Vigilmon gives you external visibility into Solid Queue health through HTTP monitors for queue health endpoints and heartbeat monitors for worker processes.

Why Solid Queue Needs External Monitoring

Solid Queue stores all queue state in your database, but that doesn't mean it monitors itself. External monitoring with Vigilmon adds:

• Proactive alerting when worker or dispatcher processes crash
• Queue depth threshold checks before backlogs grow unacceptably large
• Blocked execution detection when jobs are stuck in claimed state without progressing
• Failed job rate monitoring to catch recurring exceptions before they fill your failure table
• Heartbeat monitoring for workers that must prove they are actively dequeuing jobs

Step 1: Build a Solid Queue Health Endpoint

Add a health controller that queries Solid Queue's database tables:

# app/controllers/health/solid_queue_controller.rb
module Health
  class SolidQueueController < ApplicationController
    skip_before_action :authenticate_user!, raise: false

    WAITING_THRESHOLD  = (ENV['SQ_WAITING_THRESHOLD']  || 500).to_i
    FAILED_THRESHOLD   = (ENV['SQ_FAILED_THRESHOLD']   || 50).to_i
    BLOCKED_THRESHOLD  = (ENV['SQ_BLOCKED_THRESHOLD']  || 20).to_i

    def show
      issues = []

      waiting  = SolidQueue::ReadyExecution.count
      failed   = SolidQueue::FailedExecution.count
      blocked  = SolidQueue::BlockedExecution.count
      claimed  = SolidQueue::ClaimedExecution.count
      workers  = SolidQueue::Process.where(kind: 'Worker').count
      dispatchers = SolidQueue::Process.where(kind: 'Dispatcher').count

      issues << "waiting_jobs_#{waiting}"   if waiting  > WAITING_THRESHOLD
      issues << "failed_jobs_#{failed}"     if failed   > FAILED_THRESHOLD
      issues << "blocked_jobs_#{blocked}"   if blocked  > BLOCKED_THRESHOLD
      issues << "no_workers"                if workers.zero? && waiting > 0
      issues << "no_dispatchers"            if dispatchers.zero?

      stats = {
        ready:       waiting,
        claimed:     claimed,
        failed:      failed,
        blocked:     blocked,
        workers:     workers,
        dispatchers: dispatchers,
      }

      if issues.any?
        render json: { status: 'degraded', issues: issues, stats: stats }, status: :service_unavailable
      else
        render json: { status: 'ok', stats: stats }
      end
    end
  end
end

Register the route:

# config/routes.rb
Rails.application.routes.draw do
  namespace :health do
    resource :solid_queue, only: :show
  end
  # ...
end

The endpoint returns 200 ok when the queue is healthy and 503 degraded with a list of issues when thresholds are breached or processes are missing.

Step 2: Detect Stuck Claimed Jobs

Jobs that remain in claimed state beyond your maximum expected runtime indicate a worker crash that didn't release the lock:

# app/controllers/health/solid_queue_controller.rb (add inside show or as separate endpoint)

def stalled
  stall_threshold = (ENV['SQ_STALL_MINUTES'] || 10).to_i.minutes.ago
  stalled_count = SolidQueue::ClaimedExecution
    .joins(:job)
    .where('solid_queue_claimed_executions.created_at < ?', stall_threshold)
    .count

  if stalled_count > 0
    render json: { status: 'degraded', stalled_jobs: stalled_count }, status: :service_unavailable
  else
    render json: { status: 'ok', stalled_jobs: 0 }
  end
end

Add the route:

namespace :health do
  resource :solid_queue, only: :show do
    get :stalled, on: :collection
  end
end

Step 3: Add Heartbeat Pings to Job Classes

A healthy queue health endpoint doesn't confirm that specific workers are processing jobs. Vigilmon heartbeats verify that particular job classes are executing successfully:

# app/jobs/nightly_report_job.rb
class NightlyReportJob < ApplicationJob
  queue_as :default

  def perform
    generate_report

    # Ping heartbeat only on success — a missed ping IS the alert
    if (url = ENV['VIGILMON_NIGHTLY_REPORT_HEARTBEAT'])
      Net::HTTP.get(URI(url))
    end
  end

  private

  def generate_report
    # your job logic
  end
end

For periodic background workers using Solid Queue's recurring job configuration:

# config/solid_queue.yml
dispatchers:
  - polling_interval: 1
    batch_size: 500
    recurring_tasks:
      nightly_report:
        class: NightlyReportJob
        schedule: "0 2 * * *"   # 02:00 UTC daily

Set the heartbeat URL in your environment:

VIGILMON_NIGHTLY_REPORT_HEARTBEAT=https://vigilmon.online/heartbeat/your-token-here

For high-frequency jobs, ping every N completions instead of every run:

class EmailDeliveryJob < ApplicationJob
  queue_as :emails
  HEARTBEAT_EVERY = (ENV['HEARTBEAT_EVERY'] || '25').to_i

  @@completions = Concurrent::AtomicFixnum.new(0)

  def perform(email_id)
    send_email(email_id)

    count = @@completions.increment
    if count % HEARTBEAT_EVERY == 0
      Net::HTTP.get(URI(ENV['VIGILMON_EMAIL_HEARTBEAT'])) rescue nil
    end
  end
end

Step 4: Configure Vigilmon HTTP Monitors

1. Log in to vigilmon.online and go to Monitors → New Monitor
2. Choose HTTP / HTTPS
3. Set the URL to your Solid Queue health endpoint: https://your-app.example.com/health/solid_queue
4. Set the check interval to 1 minute
5. Under Expected response, configure:
   • Status code: 200
   • Response body contains: "status":"ok"
   • Response time threshold: 5000ms
6. Under Alert channels, assign your Slack or email integration
7. Save the monitor

Add monitors for each concern:

| Monitor URL | Purpose | Interval | |---|---|---| | /health/solid_queue | Queue depth, failed count, worker presence | 1 min | | /health/solid_queue/stalled | Stuck claimed job detection | 2 min |

Step 5: Configure Heartbeat Monitors for Workers

1. In Vigilmon, go to Monitors → New Monitor → Heartbeat
2. Set the name: solid-queue-nightly-report
3. Set the expected interval matching your job schedule (e.g. 24 hours)
4. Set the grace period: 30 minutes
5. Save — copy the heartbeat URL: https://vigilmon.online/heartbeat/abc123xyz
6. Set VIGILMON_NIGHTLY_REPORT_HEARTBEAT in your production environment

Create one heartbeat monitor per critical recurring job:

| Job | Heartbeat Interval | Grace Period | |---|---|---| | NightlyReportJob | 24 hours | 30 min | | HourlyDataSyncJob | 1 hour | 10 min | | EmailDeliveryJob | 5 min | 10 min |

Step 6: Alert Routing

| Monitor | Alert Channel | Priority | |---|---|---| | Queue health /health/solid_queue | Slack + PagerDuty | P1 | | Stalled jobs /health/solid_queue/stalled | Slack | P2 | | Worker heartbeats | Slack + PagerDuty | P1 |

Key thresholds to tune for your workload:

• Waiting threshold: set to 2–3× your normal peak backlog — not a static number
• Failed threshold: alert when failures exceed what your retry count should have cleared
• Stall window: 10 minutes is a safe default; increase if your jobs legitimately run longer
• Dispatcher check: always alert if the dispatcher process count drops to zero — no dispatcher means scheduled recurring jobs stop firing entirely

Summary

Solid Queue failures are silent by design — the database persists state without raising exceptions visible to your HTTP layer. External monitoring with Vigilmon catches problems before they become incidents:

| Monitor Type | What It Covers | |---|---| | HTTP monitor on /health/solid_queue | Queue depth, failure count, process presence | | HTTP monitor on /health/solid_queue/stalled | Stuck claimed job detection | | Heartbeat monitor | Specific job class execution and scheduler liveness |

Get started free at vigilmon.online — your first Solid Queue monitor is running in under two minutes.