tutorial

How to Monitor NimblePool with Vigilmon

Track NimblePool resource pool health in your Elixir application and use Vigilmon to alert when pool checkout times spike, workers fail, or pool utilization reaches saturation.

How to Monitor NimblePool with Vigilmon

NimblePool is a lean, efficient resource pool library for Elixir. Unlike heavyweight alternatives, it provides a simple GenServer-based pool that manages a fixed number of workers — HTTP connections, database handles, external process ports, or any resource that is expensive to create but cheap to reuse. Phoenix applications use it under the hood for Mint HTTP connections; you can use it directly wherever you need a bounded pool of reusable resources.

Pool failures are subtle and cumulative. A pool that saturates under load causes callers to wait or time out. A pool that leaks checked-out resources shrinks over time until new requests queue indefinitely. A worker that crashes and does not reinitialize correctly drains pool capacity silently. None of these failures raise an exception immediately — they manifest as slow responses, connection timeouts, or stalled queues that are hard to trace back to pool exhaustion.

Vigilmon monitors give you early warning before pool saturation becomes a user-visible incident.

Why Monitor NimblePool?

Pool health issues degrade gracefully until they don't:

  • Pool exhaustion — all workers are checked out; new requests queue indefinitely or time out, causing cascading slowdowns across your application
  • Worker leak — a caller crashes after checkout without returning the worker, gradually draining the pool below its configured size
  • Slow worker initialization — a handle_checkout/4 callback that calls a slow external service causes pool startup to block the supervisor
  • Worker crash loop — a handle_checkin/4 error terminates a worker that is immediately restarted, burning CPU and causing transient failures for callers
  • Unbalanced usage — a subset of callers monopolizes the pool, starving other consumers and creating head-of-line blocking

These failures share a common signature: pool queue depth rises, checkout latency increases, and throughput drops — all before any error log appears.

Key Metrics to Track

| Metric | What it tells you | |--------|-------------------| | Pool queue depth | Number of callers waiting for a worker to become available | | Checkout wait time | How long callers block before receiving a worker | | Worker utilization | Percentage of pool capacity currently checked out | | Worker initialization time | Latency of handle_checkout/4 for fresh workers | | Checkout timeout rate | Frequency of :timeout results from NimblePool.checkout!/3 | | Worker restart rate | How often workers crash and are reinitialized |

Step 1: Add NimblePool to Your Project

# mix.exs
defp deps do
  [
    {:nimble_pool, "~> 1.1"},
    {:telemetry, "~> 1.2"},
    # rest of your deps
  ]
end

mix deps.get

Define a pool worker. This example manages a pool of Mint HTTP connections:

# lib/my_app/http_pool.ex
defmodule MyApp.HttpPool do
  @behaviour NimblePool

  require Logger

  @host "api.example.com"
  @port 443
  @connect_timeout 5_000

  def start_link(opts) do
    NimblePool.start_link(
      worker: {__MODULE__, opts},
      pool_size: Keyword.get(opts, :size, 10),
      name: __MODULE__
    )
  end

  def checkout!(fun, timeout \\ 5_000) do
    start = System.monotonic_time()

    result =
      NimblePool.checkout!(__MODULE__, :checkout, fn _from, conn ->
        try do
          {fun.(conn), conn}
        rescue
          e -> {{:error, e}, :discard}
        end
      end, timeout)

    duration = System.monotonic_time() - start

    :telemetry.execute(
      [:my_app, :http_pool, :checkout],
      %{duration: duration},
      %{result: elem(result, 0) |> result_tag()}
    )

    result
  rescue
    e in NimblePool.CheckoutError ->
      :telemetry.execute([:my_app, :http_pool, :timeout], %{count: 1}, %{})
      reraise e, __STACKTRACE__
  end

  # NimblePool callbacks

  @impl NimblePool
  def init_worker(_pool_state) do
    start = System.monotonic_time()
    {:ok, conn} = Mint.HTTP.connect(:https, @host, @port, timeout: @connect_timeout)
    duration = System.monotonic_time() - start
    :telemetry.execute([:my_app, :http_pool, :worker_init], %{duration: duration}, %{})
    {:ok, conn, _pool_state = nil}
  end

  @impl NimblePool
  def handle_checkout(:checkout, _from, conn, pool_state) do
    {:ok, conn, conn, pool_state}
  end

  @impl NimblePool
  def handle_checkin(conn, _from, _old_conn, pool_state) do
    if Mint.HTTP.open?(conn) do
      {:ok, conn, pool_state}
    else
      :telemetry.execute([:my_app, :http_pool, :worker_discard], %{count: 1}, %{reason: :closed})
      {:remove, :closed}
    end
  end

  @impl NimblePool
  def terminate_worker(reason, _conn, pool_state) do
    :telemetry.execute([:my_app, :http_pool, :worker_terminate], %{count: 1}, %{reason: reason})
    {:ok, pool_state}
  end

  defp result_tag({:ok, _}), do: :ok
  defp result_tag({:error, _}), do: :error
  defp result_tag(_), do: :unknown
end

Step 2: Attach Telemetry Handlers for Pool Events

# lib/my_app/pool_telemetry.ex
defmodule MyApp.PoolTelemetry do
  require Logger

  def attach do
    :telemetry.attach_many(
      "my-app-pool-handler",
      [
        [:my_app, :http_pool, :checkout],
        [:my_app, :http_pool, :timeout],
        [:my_app, :http_pool, :worker_init],
        [:my_app, :http_pool, :worker_discard],
        [:my_app, :http_pool, :worker_terminate]
      ],
      &handle_event/4,
      nil
    )
  end

  defp handle_event([:my_app, :http_pool, :checkout], %{duration: d}, %{result: r}, _) do
    ms = System.convert_time_unit(d, :native, :millisecond)
    if ms > 1_000 do
      Logger.warning("HttpPool: slow checkout #{ms}ms (result=#{r})")
    end
  end

  defp handle_event([:my_app, :http_pool, :timeout], _, _, _) do
    Logger.error("HttpPool: checkout timed out — pool may be exhausted")
  end

  defp handle_event([:my_app, :http_pool, :worker_discard], _, %{reason: reason}, _) do
    Logger.warning("HttpPool: worker discarded (reason=#{reason})")
  end

  defp handle_event(_, _, _, _), do: :ok
end

Attach handlers in your Application start:

# lib/my_app/application.ex
def start(_type, _args) do
  MyApp.PoolTelemetry.attach()

  children = [
    MyApp.Repo,
    {MyApp.HttpPool, size: 10},
    MyAppWeb.Endpoint
  ]

  Supervisor.start_link(children, strategy: :one_for_one, name: MyApp.Supervisor)
end

Step 3: Add Pool Health Metrics to Telemetry.Metrics

# lib/my_app/telemetry.ex
def metrics do
  [
    # Pool checkout latency distribution
    distribution("my_app.http_pool.checkout.duration",
      unit: {:native, :millisecond},
      reporter_options: [buckets: [10, 50, 100, 500, 1000, 5000]],
      tags: [:result]
    ),

    # Timeout counter
    counter("my_app.http_pool.timeout.count"),

    # Worker lifecycle
    counter("my_app.http_pool.worker_init.count"),
    distribution("my_app.http_pool.worker_init.duration",
      unit: {:native, :millisecond}
    ),
    counter("my_app.http_pool.worker_discard.count", tags: [:reason]),
    counter("my_app.http_pool.worker_terminate.count", tags: [:reason])
  ]
end

Step 4: Add a Pool Health Endpoint

# lib/my_app_web/plugs/pool_health.ex
defmodule MyAppWeb.Plugs.PoolHealth do
  import Plug.Conn

  def init(opts), do: opts

  def call(%Plug.Conn{request_path: "/health/pool"} = conn, _opts) do
    checks = %{
      http_pool_alive: check_pool(MyApp.HttpPool),
      http_pool_utilization: pool_utilization(MyApp.HttpPool)
    }

    status =
      cond do
        checks.http_pool_alive == :down -> 503
        checks.http_pool_utilization >= 0.9 -> 503
        true -> 200
      end

    conn
    |> put_resp_content_type("application/json")
    |> send_resp(status, Jason.encode!(%{
      status: if(status == 200, do: "ok", else: "degraded"),
      checks: checks
    }))
    |> halt()
  end

  def call(conn, _opts), do: conn

  defp check_pool(pool) do
    case Process.whereis(pool) do
      nil -> :down
      pid -> if Process.alive?(pid), do: :ok, else: :down
    end
  end

  defp pool_utilization(pool) do
    case :sys.get_state(pool) do
      %{workers: workers, available: available} ->
        Float.round(1.0 - length(available) / length(workers), 2)
      _ ->
        0.0
    end
  rescue
    _ -> 0.0
  end
end

Step 5: Create Monitors in Vigilmon

HTTP monitor for pool health endpoint:

  1. Sign in at vigilmon.online
  2. Click New Monitor → HTTP
  3. Set URL to https://your-app.example.com/health/pool
  4. Set interval to 60 seconds
  5. Alert condition: status must be 200

Heartbeat monitor for pool checkout success:

Add a periodic ping from your application when pool checkouts succeed above a threshold:

# lib/my_app/pool_heartbeat.ex
defmodule MyApp.PoolHeartbeat do
  use GenServer

  @heartbeat_url System.get_env("VIGILMON_POOL_HEARTBEAT_URL")
  @interval :timer.seconds(30)
  @timeout_threshold 0.05  # alert if >5% checkouts timeout

  def start_link(_), do: GenServer.start_link(__MODULE__, %{}, name: __MODULE__)

  def init(state), do: {:ok, state, {:continue, :start}}

  def handle_continue(:start, state) do
    schedule()
    {:noreply, state}
  end

  def handle_info(:ping, state) do
    if pool_healthy?() do
      if @heartbeat_url do
        :httpc.request(:get, {String.to_charlist(@heartbeat_url), []}, [], [])
      end
    end

    schedule()
    {:noreply, state}
  end

  defp pool_healthy? do
    case Process.whereis(MyApp.HttpPool) do
      nil -> false
      pid -> Process.alive?(pid)
    end
  end

  defp schedule, do: Process.send_after(self(), :ping, @interval)
end

Create a Heartbeat monitor in Vigilmon with a 2-minute expected interval to alert when the pool stops reporting healthy.

Step 6: Alerting

In Vigilmon, configure Notifications → New Channel:

Slack for pool exhaustion:

🔴 ALERT: NimblePool saturation — MyApp HttpPool
Monitor: /health/pool returning 503
Utilization: 90%+ (all workers checked out)
Action: Increase pool_size or investigate slow callers holding workers

PagerDuty for complete pool failure:

Configure a high-priority alert when the pool process itself goes down — this indicates a supervisor failure, not just load saturation.

Common NimblePool Issues and Fixes

Pool exhaustion under load:

# Increase pool size — the default is often too small for production load
{MyApp.HttpPool, size: 25}

# Or add a checkout timeout to fail fast rather than queue indefinitely
NimblePool.checkout!(__MODULE__, :checkout, fn _, conn -> {use(conn), conn} end, 2_000)

Worker leak from caller crash:

NimblePool handles caller crashes automatically — if a caller process dies while holding a checked-out worker, the pool receives a :DOWN message and calls handle_checkin/4 with :error. Ensure your callback handles this:

@impl NimblePool
def handle_checkin(:error, _from, _conn, pool_state) do
  # Caller crashed — discard the connection since we don't know its state
  {:remove, :caller_crash}
end

Worker initialization blocking startup:

# Use lazy initialization to avoid blocking the supervisor
{MyApp.HttpPool, size: 10, lazy: true}

What You've Built

| What | How | |------|-----| | Instrumented pool | Telemetry events on checkout, timeout, worker init/discard/terminate | | Pool health endpoint | HTTP check verifying pool process and utilization | | Telemetry.Metrics integration | Checkout latency distribution, timeout counter, worker lifecycle | | Pool liveness heartbeat | GenServer pinging Vigilmon every 30 seconds when healthy | | Infrastructure alerting | Vigilmon HTTP monitor on /health/pool | | Saturation alerting | 503 on >90% utilization triggers Slack notification |

NimblePool keeps your resource usage bounded and efficient. Vigilmon keeps you informed when the bounds are being tested.

Next Steps

  • Add per-caller tracking to identify which modules hold workers longest
  • Set up Grafana panels for checkout latency p95/p99 to catch slow external services
  • Consider a secondary pool with a larger size as a burst buffer for high-traffic periods
  • Use Vigilmon's response time history to correlate pool health with traffic spikes

Get started free at vigilmon.online.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles