How to Monitor Retry (Elixir) with Vigilmon

The Retry library provides composable retry logic for Elixir applications. With a clean DSL, it lets you wrap calls to external services — HTTP APIs, databases, message queues — with configurable retry strategies: linear backoff, exponential backoff, jitter, and custom predicates. When a transient network error or rate limit causes a call to fail, Retry handles the wait-and-retry so your application code stays clean.

When Retry logic itself starts causing problems, it's easy to miss. A retry exhaustion event — all attempts failing — means a business operation silently failed. A retry storm — many callers retrying simultaneously — can amplify load on an already-struggling dependency. And overlapping retries across multiple services can hide which upstream is actually failing.

Vigilmon lets you observe retry behavior across your application so you can distinguish transient blips from persistent failures and tune your retry policies before they hurt users.

Why Monitor Retry Behavior?

Retry is a resilience mechanism — but it can create its own problems:

• Retry exhaustion — all configured attempts fail; the operation is abandoned and the caller receives an error that may not be visible in logs
• Retry storm — many callers retrying simultaneously multiplies load on a struggling upstream service, turning a brief overload into a prolonged outage
• Retry masking real failures — a high retry success rate hides the fact that a dependency is flaky; you only notice when the retry budget runs out
• Backoff misconfiguration — too-aggressive backoff (short delays, many retries) overwhelms upstream; too-conservative (long delays, few retries) causes unnecessary latency
• Partial success ambiguity — operations that partially complete before failure may be retried in full, causing duplicate side effects (double-sent emails, double-charged payments)

Key Metrics to Track

| Metric | What it tells you | |--------|-------------------| | Retry attempt rate by service | How often transient failures occur per dependency | | Retry success rate | What fraction of retried operations eventually succeed | | Retry exhaustion count | How many operations fail even after all retries | | Attempt number distribution | Whether failures resolve on attempt 2 vs. attempt N | | Total time including retries | End-to-end latency impact of retry delays | | Retry rate by error type | Which error classes trigger the most retries |

Step 1: Add Retry to Your Project

# mix.exs
defp deps do
  [
    {:retry, "~> 0.18"}
  ]
end

Basic usage with exponential backoff:

# lib/my_app/external_api.ex
defmodule MyApp.ExternalAPI do
  use Retry

  def fetch_data(resource_id) do
    retry with: exponential_backoff() |> randomize() |> expiry(10_000) do
      HTTPoison.get("https://api.example.com/data/#{resource_id}")
    after
      result -> result
    else
      error -> {:error, error}
    end
  end
end

Step 2: Wrap Retry with Telemetry

The Retry library does not emit telemetry natively. Wrap it in a helper that instruments each attempt:

# lib/my_app/instrumented_retry.ex
defmodule MyApp.InstrumentedRetry do
  @moduledoc """
  Thin wrapper around the Retry library that emits telemetry for each attempt,
  final success, and exhaustion. Use this instead of bare `retry` macros
  in service modules so retry behavior is visible in metrics.
  """

  require Logger

  defmacro with_retry(opts \\ [], do: block) do
    service = Keyword.get(opts, :service, "unknown")
    max_attempts = Keyword.get(opts, :max_attempts, 3)
    base_delay_ms = Keyword.get(opts, :base_delay_ms, 100)

    quote do
      attempts = unquote(max_attempts)
      service = unquote(service)
      base_delay = unquote(base_delay_ms)

      start = System.monotonic_time()

      result =
        MyApp.InstrumentedRetry.__run__(
          fn -> unquote(block) end,
          service,
          attempts,
          base_delay,
          1
        )

      total_duration = System.monotonic_time() - start

      :telemetry.execute(
        [:my_app, :retry, :complete],
        %{duration: total_duration},
        %{service: service, result: MyApp.InstrumentedRetry.__result_tag__(result)}
      )

      result
    end
  end

  def __run__(fun, service, max_attempts, base_delay_ms, attempt) do
    attempt_start = System.monotonic_time()
    result = fun.()
    attempt_duration = System.monotonic_time() - attempt_start

    :telemetry.execute(
      [:my_app, :retry, :attempt],
      %{duration: attempt_duration},
      %{service: service, attempt: attempt, success: true}
    )

    result
  rescue
    e ->
      attempt_duration = System.monotonic_time() - attempt_start

      :telemetry.execute(
        [:my_app, :retry, :attempt],
        %{duration: attempt_duration},
        %{service: service, attempt: attempt, success: false, error: e.__struct__}
      )

      if attempt >= max_attempts do
        Logger.error("Retry exhausted: service=#{service} attempts=#{attempt} error=#{inspect(e)}")

        :telemetry.execute(
          [:my_app, :retry, :exhausted],
          %{count: 1},
          %{service: service, attempts: attempt}
        )

        reraise e, __STACKTRACE__
      else
        delay = trunc(base_delay_ms * :math.pow(2, attempt - 1))
        jitter = :rand.uniform(trunc(delay * 0.3))
        Process.sleep(delay + jitter)
        __run__(fun, service, max_attempts, base_delay_ms, attempt + 1)
      end
  end

  def __result_tag__({:ok, _}), do: :ok
  def __result_tag__({:error, _}), do: :error
  def __result_tag__(_), do: :ok
end

Usage:

defmodule MyApp.PaymentGateway do
  import MyApp.InstrumentedRetry

  def charge(amount, token) do
    with_retry service: "payment_gateway", max_attempts: 3, base_delay_ms: 200 do
      MyApp.Stripe.create_charge(amount, token)
    end
  end
end

Step 3: Define Telemetry Metrics

# lib/my_app/telemetry.ex
def metrics do
  [
    # Per-attempt metrics
    counter("my_app.retry.attempt.count",
      tags: [:service, :attempt, :success]
    ),
    distribution("my_app.retry.attempt.duration",
      unit: {:native, :millisecond},
      reporter_options: [buckets: [10, 50, 100, 500, 1000, 5000]]
    ),

    # Exhaustion — the important one
    counter("my_app.retry.exhausted.count",
      tags: [:service]
    ),

    # End-to-end latency including all delays
    distribution("my_app.retry.complete.duration",
      unit: {:native, :millisecond},
      reporter_options: [buckets: [100, 500, 1000, 5000, 10_000, 30_000]],
      tags: [:service, :result]
    )
  ]
end

Step 4: Build a Retry Health Check

Verify that your retry infrastructure is working correctly with a self-test that exercises a controlled failure and recovery:

# lib/my_app/retry_health.ex
defmodule MyApp.RetryHealth do
  @moduledoc """
  Self-test for retry infrastructure. Runs a fake call that fails on
  the first N attempts and succeeds on the last. Verifies that the
  retry machinery correctly handles transient errors.
  """

  @max_attempts 3

  def check do
    counter = :counters.new(1, [])
    start = System.monotonic_time(:millisecond)

    result =
      try do
        run_with_retry(counter)
        {:ok, System.monotonic_time(:millisecond) - start}
      rescue
        e -> {:error, inspect(e)}
      end

    result
  end

  defp run_with_retry(counter) do
    MyApp.InstrumentedRetry.__run__(
      fn ->
        attempt = :counters.get(counter, 1) + 1
        :counters.put(counter, 1, attempt)

        if attempt < @max_attempts do
          raise RuntimeError, "simulated transient failure on attempt #{attempt}"
        end

        :ok
      end,
      "health_probe",
      @max_attempts,
      10,
      1
    )
  end
end

Step 5: Track Retry Storms

A retry storm occurs when many callers retry simultaneously. Detect it by measuring aggregate retry rate over a short window:

# lib/my_app/retry_storm_detector.ex
defmodule MyApp.RetryStormDetector do
  use GenServer
  require Logger

  @window_seconds 60
  @storm_threshold 50
  @poll_interval :timer.seconds(10)

  def start_link(_), do: GenServer.start_link(__MODULE__, %{counts: %{}}, name: __MODULE__)

  def init(state) do
    :telemetry.attach(
      "retry-storm-detector",
      [:my_app, :retry, :attempt],
      &handle_retry_attempt/4,
      nil
    )

    schedule()
    {:ok, state}
  end

  def handle_retry_attempt(_event, _measurements, %{service: service, attempt: attempt}, _config)
      when attempt > 1 do
    GenServer.cast(__MODULE__, {:record_retry, service})
  end

  def handle_retry_attempt(_, _, _, _), do: :ok

  def handle_cast({:record_retry, service}, %{counts: counts} = state) do
    key = {service, System.system_time(:second)}
    updated = Map.update(counts, key, 1, &(&1 + 1))
    {:noreply, %{state | counts: updated}}
  end

  def handle_info(:check, state) do
    now = System.system_time(:second)
    cutoff = now - @window_seconds

    # Count retries in the recent window per service
    recent_counts =
      state.counts
      |> Enum.filter(fn {{_service, ts}, _} -> ts > cutoff end)
      |> Enum.group_by(fn {{service, _ts}, _} -> service end, fn {_, count} -> count end)
      |> Map.new(fn {service, counts} -> {service, Enum.sum(counts)} end)

    Enum.each(recent_counts, fn {service, count} ->
      :telemetry.execute(
        [:my_app, :retry, :rate],
        %{count: count},
        %{service: service, window_seconds: @window_seconds}
      )

      if count > @storm_threshold do
        Logger.warning("Retry storm detected: service=#{service} retries=#{count} in #{@window_seconds}s")
      end
    end)

    # Prune old entries
    pruned = Enum.filter(state.counts, fn {{_, ts}, _} -> ts > cutoff end) |> Map.new()

    schedule()
    {:noreply, %{state | counts: pruned}}
  end

  defp schedule, do: Process.send_after(self(), :check, @poll_interval)
end

Step 6: Add a Health Endpoint

# lib/my_app_web/controllers/health_controller.ex
defmodule MyAppWeb.HealthController do
  use MyAppWeb, :controller

  def index(conn, _params) do
    retry_check = MyApp.RetryHealth.check()

    checks = %{
      retry: format_retry_check(retry_check),
      database: check_db()
    }

    status = if match?({:ok, _}, retry_check), do: 200, else: 503

    conn
    |> put_status(status)
    |> json(%{
      status: if(status == 200, do: "ok", else: "degraded"),
      checks: checks
    })
  end

  defp format_retry_check({:ok, latency_ms}), do: %{status: "ok", latency_ms: latency_ms}
  defp format_retry_check({:error, reason}), do: %{status: "error", reason: reason}

  defp check_db do
    case Ecto.Adapters.SQL.query(MyApp.Repo, "SELECT 1", []) do
      {:ok, _} -> "ok"
      _ -> "error"
    end
  rescue
    _ -> "error"
  end
end

Step 7: Create Monitors in Vigilmon

HTTP monitor for the health endpoint:

1. Sign in at vigilmon.online
2. Click New Monitor → HTTP
3. Set URL to https://your-app.example.com/health
4. Set interval to 60 seconds
5. Alert condition: status 200 and body contains "retry":{"status":"ok"}

Heartbeat monitor for the retry pipeline:

# lib/my_app/retry_heartbeat.ex
defmodule MyApp.RetryHeartbeat do
  use GenServer

  @heartbeat_url System.get_env("VIGILMON_RETRY_HEARTBEAT_URL")
  @interval :timer.minutes(5)

  def start_link(_), do: GenServer.start_link(__MODULE__, %{}, name: __MODULE__)

  def init(state) do
    schedule()
    {:ok, state}
  end

  def handle_info(:check, state) do
    case MyApp.RetryHealth.check() do
      {:ok, _latency} ->
        ping_vigilmon()

      {:error, reason} ->
        require Logger
        Logger.error("Retry health check failed: #{inspect(reason)}")
    end

    schedule()
    {:noreply, state}
  end

  defp ping_vigilmon do
    if @heartbeat_url do
      :httpc.request(:get, {String.to_charlist(@heartbeat_url), []}, [{:timeout, 3000}], [])
    end
  end

  defp schedule, do: Process.send_after(self(), :check, @interval)
end

Create a Heartbeat monitor in Vigilmon with an 8-minute expected interval.

Step 8: Alerting

In Vigilmon, configure Notifications → New Channel:

Slack for retry exhaustion spike:

⚠️ DEGRADED: Retry Exhaustion — MyApp
Service: payment_gateway
Exhausted retries 12 times in the last 5 minutes
Action: Check payment_gateway health and circuit breaker state

PagerDuty for critical service exhaustion:

If payment_gateway, auth_service, or any revenue-critical dependency exhausts retries, page on-call immediately.

Grafana alert for retry storm:

- alert: RetryStormDetected
  expr: sum(rate(my_app_retry_attempt_count{success="false"}[5m])) by (service) > 10
  for: 2m
  annotations:
    summary: "Retry storm on {{ $labels.service }} — {{ $value }} failed attempts/min"

Common Retry Issues and Fixes

Retry storm — reduce concurrent pressure with jitter:

# Without jitter, many callers retry at the same time after a failure
retry with: exponential_backoff() |> randomize(0.5) |> expiry(30_000) do
  call_external_service()
end

The randomize/1 call adds ±50% jitter to each delay, spreading retries across time.

Partial success — don't retry non-idempotent operations:

# Only retry on specific error types; never retry 402 Payment Required
retry with: linear_backoff(500, 2) |> Stream.take(3) do
  result = stripe_charge(amount, token)
  # Don't retry if we got a definitive rejection
  if match?({:error, %{code: "card_declined"}}, result), do: {:halt, result}, else: result
end

Exhaustion alerting — log structured context:

# Include enough context to debug the failure without querying logs
Logger.error("Retry exhausted",
  service: "payment_gateway",
  attempts: 3,
  last_error: inspect(error),
  user_id: user_id,
  amount_cents: amount
)

Too-aggressive backoff causing latency:

# For user-facing requests, keep total retry budget under 5 seconds
retry with: exponential_backoff(100) |> expiry(4_000) do
  call_external_api()
end

# For background jobs, allow longer retry windows
retry with: exponential_backoff(1_000) |> expiry(60_000) do
  process_batch_job()
end

What You've Built

| What | How | |------|-----| | Instrumented retry wrapper | Telemetry on every attempt, success, and exhaustion | | Retry metrics | Counters and distributions for attempts, exhaustion, and duration | | Self-test health check | Controlled failure/recovery exercise proving retry works | | Retry storm detector | Sliding-window counter that alerts on burst retry activity | | Structured health endpoint | JSON health response with retry system status | | Liveness heartbeat | GenServer pinging Vigilmon after successful self-test | | Real-time alerting | Vigilmon HTTP monitor + PagerDuty on exhaustion spike |

Retry makes your Elixir application resilient to transient failures. Vigilmon makes sure that resilience is working — and alerts you when retry exhaustion indicates a persistent failure that needs operator attention.

Next Steps

• Use circuit breaker patterns (:fuse or :breaker) alongside Retry to trip automatically on exhaustion
• Add per-service retry dashboards in Grafana to compare reliability across your dependencies
• Set up Vigilmon monitors for each critical external service independently
• Use Vigilmon's downtime history to measure the real availability of your external dependencies, including retry recovery time

Get started free at vigilmon.online.