How to Monitor Oban with Vigilmon

Oban is the de facto standard for reliable background job processing in Elixir. Backed by PostgreSQL, it gives you durable queues, scheduled jobs, unique constraints, and automatic retries — all without an external message broker.

But "durable" and "reliable" don't mean "self-announcing." When an Oban queue stalls, a job exhausts its retry limit, or your Postgres connection pool runs dry, nothing notifies you unless you've wired up external monitoring.

This tutorial adds production observability to an Oban deployment:

• A health check endpoint that surfaces Oban queue state
• Heartbeat monitoring for critical recurring jobs
• HTTP uptime monitoring with Vigilmon
• Failure alerts via Slack
• A public status page for your job infrastructure

Why Monitor Oban?

Oban's BEAM process stays alive even when jobs are failing. The supervisor restarts workers; retries happen as scheduled; the dashboard might look healthy. Failures that warrant an alert include:

• Queue stalls — jobs inserted but not picked up (e.g. max_demand: 0 accidentally set, DB lock contention)
• Retry exhaustion — a job reaching max_attempts with no human awareness
• Partition failures — the scheduled cron job that should run every hour hasn't been seen in two
• PostgreSQL connectivity — Oban can't poll for jobs if its Repo can't talk to Postgres
• Error rate spikes — a batch job failing 80% of attempts is different from a transient one-off

External monitoring catches these because it observes outcomes, not internals.

Step 1: Add an Oban health check endpoint

Expose Oban's queue state over HTTP so Vigilmon can probe it.

# lib/my_app_web/controllers/health_controller.ex
defmodule MyAppWeb.HealthController do
  use MyAppWeb, :controller

  def oban(conn, _params) do
    checks = %{
      queues: check_queues(),
      database: check_database()
    }

    status = if all_ok?(checks), do: :ok, else: :service_unavailable

    conn
    |> put_status(status)
    |> json(%{status: status, checks: checks})
  end

  defp check_queues do
    case Oban.check_queue(MyApp.Oban, queue: :default) do
      %{running: _, available: _, paused: false} -> "ok"
      %{paused: true} -> "paused"
      _ -> "error"
    end
  rescue
    _ -> "error"
  end

  defp check_database do
    case Ecto.Adapters.SQL.query(MyApp.Repo, "SELECT 1", []) do
      {:ok, _} -> "ok"
      {:error, _} -> "error"
    end
  end

  defp all_ok?(%{queues: "ok", database: "ok"}), do: true
  defp all_ok?(_), do: false
end

Wire it into your router (before authentication plugs):

# lib/my_app_web/router.ex
scope "/health", MyAppWeb do
  pipe_through :api

  get "/oban", HealthController, :oban
end

Test it:

mix phx.server
curl http://localhost:4000/health/oban
# {"status":"ok","checks":{"queues":"ok","database":"ok"}}

A 503 response body tells you which check failed.

Step 2: Extended queue metrics

For richer monitoring, expose per-queue stats:

defp queue_stats do
  queues = Oban.config(MyApp.Oban).queues

  Enum.map(queues, fn {queue, _opts} ->
    queue_name = Atom.to_string(queue)

    counts =
      MyApp.Repo.all(
        from j in Oban.Job,
          where: j.queue == ^queue_name,
          group_by: j.state,
          select: {j.state, count(j.id)}
      )
      |> Map.new()

    %{
      queue: queue_name,
      available: Map.get(counts, "available", 0),
      executing: Map.get(counts, "executing", 0),
      retryable: Map.get(counts, "retryable", 0),
      discarded: Map.get(counts, "discarded", 0)
    }
  end)
end

Include this in your health response. A spike in retryable or any discarded jobs is a signal worth alerting on.

Step 3: Heartbeat monitoring for critical jobs

The most insidious Oban failure is a job that stops being scheduled without error. Your cron schedule is configured, Oban is running, and no exceptions are raised — but the job silently didn't enqueue.

Use Vigilmon's Heartbeat Monitor to detect this. The pattern: at the end of every successful job execution, ping a unique Vigilmon URL. If the ping doesn't arrive within the expected window, Vigilmon alerts you.

# lib/my_app/workers/nightly_report_worker.ex
defmodule MyApp.Workers.NightlyReportWorker do
  use Oban.Worker,
    queue: :reports,
    max_attempts: 3

  require Logger

  @impl Oban.Worker
  def perform(%Oban.Job{}) do
    with :ok <- fetch_data(),
         :ok <- generate_report(),
         :ok <- deliver_report() do
      ping_vigilmon()
      :ok
    else
      {:error, reason} ->
        Logger.error("NightlyReportWorker failed: #{inspect(reason)}")
        {:error, reason}
    end
  end

  defp ping_vigilmon do
    url = Application.get_env(:my_app, :vigilmon_heartbeats)[:nightly_report]

    if url do
      case Req.get(url, receive_timeout: 5_000) do
        {:ok, _} -> :ok
        {:error, reason} ->
          Logger.warning("Vigilmon heartbeat ping failed: #{inspect(reason)}")
      end
    end
  end

  defp fetch_data, do: :ok
  defp generate_report, do: :ok
  defp deliver_report, do: :ok
end

Configure the heartbeat URL via environment variable:

# config/runtime.exs
config :my_app, :vigilmon_heartbeats,
  nightly_report: System.get_env("VIGILMON_NIGHTLY_REPORT_HEARTBEAT_URL")

In Vigilmon:

1. Click New Monitor → Heartbeat
2. Set the expected interval to 24 hours (with a grace period of 1 hour)
3. Copy the ping URL and set it as VIGILMON_NIGHTLY_REPORT_HEARTBEAT_URL
4. Enable Slack notifications

Now if the job never runs — or never succeeds — you know within an hour of the missed window.

Step 4: Set up HTTP monitoring in Vigilmon

Point Vigilmon at your Oban health endpoint:

1. Sign up at vigilmon.online
2. Click New Monitor → HTTP
3. Enter https://yourdomain.com/health/oban
4. Set check interval: 1 minute (paid) or 5 minutes (free)
5. Under Advanced, add a Body keyword: "status":"ok"
6. Save

The keyword check catches the case where the endpoint returns 200 but reports a degraded state — important if you return 200 with error details rather than a 503.

Step 5: Slack alerts

In Vigilmon, go to Notifications → New Channel → Slack and paste your Slack incoming webhook URL.

Enable the channel on both your HTTP monitor and each heartbeat monitor. When Oban's health endpoint returns 503:

🔴 DOWN: yourdomain.com/health/oban
Body keyword "status":"ok" not found
Detected from: EU-West, US-East

When a heartbeat misses its window:

🔴 MISSED: Nightly Report Heartbeat
Last ping: 26 hours ago (expected: every 24 hours)

Step 6: Alert on discarded jobs with Oban.Telemetry

Oban emits telemetry events for every job state transition. Hook into [:oban, :job, :stop] and [:oban, :job, :exception] to trigger additional alerting:

# lib/my_app/oban_telemetry_handler.ex
defmodule MyApp.ObanTelemetryHandler do
  require Logger

  def attach do
    :telemetry.attach_many(
      "oban-handler",
      [
        [:oban, :job, :stop],
        [:oban, :job, :exception]
      ],
      &handle_event/4,
      nil
    )
  end

  def handle_event([:oban, :job, :stop], _measurements, %{state: :discarded} = meta, _config) do
    Logger.error(
      "Oban job discarded: worker=#{meta.worker} queue=#{meta.queue} " <>
        "attempt=#{meta.attempt}/#{meta.max_attempts}"
    )
    # Optionally post to your alerting webhook here
  end

  def handle_event([:oban, :job, :exception], _measurements, meta, _config) do
    Logger.error(
      "Oban job exception: worker=#{meta.worker} error=#{inspect(meta.reason)}"
    )
  end

  def handle_event(_, _, _, _), do: :ok
end

Attach it in your Application start:

# lib/my_app/application.ex
def start(_type, _args) do
  MyApp.ObanTelemetryHandler.attach()
  # ...
end

Step 7: Status page

1. In Vigilmon, go to Status Pages → New Status Page
2. Add your Oban HTTP monitor and each heartbeat monitor
3. Label them clearly: "Background Jobs", "Nightly Report", "Weekly Summary"
4. Share the URL with your team

When a stakeholder asks "is the nightly report running?", they can check the status page without access to your server.

What you've built

| What | How | |------|-----| | Oban health endpoint | /health/oban exposing queue and DB state | | HTTP uptime monitoring | Vigilmon HTTP monitor with keyword check | | Heartbeat monitoring | Per-job ping on successful completion | | Discarded job alerts | Oban.Telemetry hook + Logger | | Slack alerts | Vigilmon Slack notification channel | | Status page | Vigilmon public status page |

Oban handles the retry logic. Vigilmon handles the silence.

Next steps

• Add heartbeat monitors for every queue that processes business-critical work
• Use Vigilmon response time history to catch slow Postgres queries affecting job pickup latency
• Set up separate monitors per Oban queue if your app has distinct SLAs per queue
• Combine Oban.Pro's built-in metrics with Vigilmon's external probes for defense in depth

Get started free at vigilmon.online.