How to Monitor Recon with Vigilmon

Recon is the production diagnostics library for Erlang/OTP, and it works equally well from Elixir. It provides runtime introspection of processes, message queues, memory usage, and system performance without impacting production load — attaching to the running VM with zero-configuration. But Recon is a tool you reach for when something is already wrong. Pair it with Vigilmon to get alerted before you need to reach for it.

External monitoring combined with Recon's metrics gives you two layers: Vigilmon wakes you up, Recon tells you where to look. This tutorial covers:

• A health endpoint that exposes Recon's key system metrics
• HTTP uptime monitoring with Vigilmon
• Heartbeat monitoring for scheduled diagnostics
• Slack alerts and a status page

Step 1: Add Recon to your project

# mix.exs
defp deps do
  [
    {:recon, "~> 2.5"},
    {:req, "~> 0.4"}   # for heartbeat pings
  ]
end

Recon has no configuration required. It ships as a pure Erlang library and is available as soon as it's in your dependency tree. Verify it's working in iex -S mix:

iex> :recon.node_stats_list(1, 1)
# [[{:process_count, 63}, {:run_queue, 0}, {:error_logger_queue_len, 0},
#   {:memory_total, 42_123_456}, {:memory_procs, 12_345_678},
#   {:memory_atoms, 634_456}, {:memory_bin, 890_123},
#   {:memory_ets, 1_234_567}]]

Step 2: Build a health endpoint that exposes Recon metrics

# lib/my_app_web/plugs/health_check.ex
defmodule MyAppWeb.Plugs.HealthCheck do
  import Plug.Conn
  require Logger

  # Alert thresholds
  @max_process_count 50_000
  @max_run_queue 10
  @max_message_queue_len 1_000
  @memory_warn_threshold_mb 1_024   # 1 GB

  def init(opts), do: opts

  def call(%Plug.Conn{request_path: "/health"} = conn, _opts) do
    checks = run_checks()
    status = if checks.status == "ok", do: 200, else: 503

    conn
    |> put_resp_content_type("application/json")
    |> send_resp(status, Jason.encode!(checks))
    |> halt()
  end

  def call(conn, _opts), do: conn

  defp run_checks do
    system = system_check()
    memory = memory_check()
    procs = process_check()

    overall = system.status == "ok" and memory.status == "ok" and procs.status == "ok"

    %{
      status: if(overall, do: "ok", else: "degraded"),
      system: system,
      memory: memory,
      processes: procs
    }
  end

  defp system_check do
    stats = :recon.node_stats_list(1, 1)
    [{:process_count, proc_count}, {:run_queue, run_queue} | _] = List.first(stats)

    status =
      cond do
        proc_count > @max_process_count -> "warning"
        run_queue > @max_run_queue -> "warning"
        true -> "ok"
      end

    %{
      status: status,
      process_count: proc_count,
      run_queue: run_queue
    }
  end

  defp memory_check do
    memory = :recon_alloc.memory(:used)
    memory_mb = div(memory, 1_024 * 1_024)

    status = if memory_mb < @memory_warn_threshold_mb, do: "ok", else: "warning"

    fragmentation = :recon_alloc.fragmentation(:current)
    frag_ratio = fragmentation[:mbcs] || 0.0

    %{
      status: status,
      used_mb: memory_mb,
      fragmentation_ratio: Float.round(frag_ratio, 3)
    }
  end

  defp process_check do
    # Find top processes by message queue length
    top_by_queue =
      :recon.proc_count(:message_queue_len, 5)
      |> Enum.map(fn {pid, queue_len, [initial_call: call | _]} ->
        %{pid: inspect(pid), queue_len: queue_len, initial_call: inspect(call)}
      end)

    max_queue = Enum.reduce(top_by_queue, 0, fn p, acc -> max(p.queue_len, acc) end)
    status = if max_queue > @max_message_queue_len, do: "warning", else: "ok"

    %{
      status: status,
      top_by_message_queue: top_by_queue
    }
  end
end

Register the plug before the router:

# lib/my_app_web/endpoint.ex
plug MyAppWeb.Plugs.HealthCheck
plug MyAppWeb.Router

Test it:

mix phx.server
curl http://localhost:4000/health | jq .
# {
#   "status": "ok",
#   "system": {"status": "ok", "process_count": 63, "run_queue": 0},
#   "memory": {"status": "ok", "used_mb": 42, "fragmentation_ratio": 0.124},
#   "processes": {
#     "status": "ok",
#     "top_by_message_queue": [
#       {"pid": "#PID<0.234.0>", "queue_len": 3, "initial_call": "{GenServer, :init_it, 6}"}
#     ]
#   }
# }

Step 3: Add binary and ETS leak detection

Long-running Elixir systems commonly leak binaries (retained large binaries from ProcLib) and ETS tables (tables that grow without bounds). Add these checks:

defp run_checks do
  system = system_check()
  memory = memory_check()
  procs = process_check()
  bins = binary_check()

  overall =
    system.status == "ok" and
    memory.status == "ok" and
    procs.status == "ok" and
    bins.status == "ok"

  %{
    status: if(overall, do: "ok", else: "degraded"),
    system: system,
    memory: memory,
    processes: procs,
    binaries: bins
  }
end

defp binary_check do
  # Top processes by binary memory
  top_bin_procs =
    :recon.proc_count(:binary_memory, 3)
    |> Enum.map(fn {pid, bin_mem, [initial_call: call | _]} ->
      %{pid: inspect(pid), binary_memory_bytes: bin_mem, initial_call: inspect(call)}
    end)

  max_bin = Enum.reduce(top_bin_procs, 0, fn p, acc -> max(p.binary_memory_bytes, acc) end)

  # Warn if any single process holds more than 100 MB in binaries
  status = if max_bin < 100_000_000, do: "ok", else: "warning"

  %{status: status, top_by_binary_memory: top_bin_procs}
end

Step 4: Monitor the health endpoint with Vigilmon

1. Sign up at vigilmon.online
2. Click New Monitor → HTTP
3. Enter https://yourdomain.com/health
4. Set check interval: 1 minute
5. Enable keyword check: body must contain "status":"ok"
6. Save

The keyword check matters here: when process count spikes or memory exceeds your threshold, the endpoint returns HTTP 200 with "status":"degraded". Vigilmon catches that degradation before it becomes a crash.

Step 5: Scheduled diagnostics with heartbeat monitoring

Use a GenServer to run periodic Recon diagnostics and ping Vigilmon to confirm the diagnostics job is healthy:

# lib/my_app/workers/recon_diagnostics_worker.ex
defmodule MyApp.Workers.ReconDiagnosticsWorker do
  use GenServer
  require Logger

  @interval :timer.minutes(5)
  @proc_count_alert 10_000
  @queue_len_alert 500

  def start_link(_), do: GenServer.start_link(__MODULE__, %{}, name: __MODULE__)

  def init(state) do
    schedule()
    {:ok, state}
  end

  def handle_info(:diagnose, state) do
    run_diagnostics()
    ping_vigilmon()
    schedule()
    {:noreply, state}
  end

  defp run_diagnostics do
    # Log current process count
    proc_count = length(:erlang.processes())

    if proc_count > @proc_count_alert do
      Logger.warning("[Recon] high process count: #{proc_count}")
      log_top_procs()
    end

    # Log top message queues
    top_queues = :recon.proc_count(:message_queue_len, 3)

    Enum.each(top_queues, fn {pid, queue_len, info} ->
      if queue_len > @queue_len_alert do
        Logger.warning("[Recon] large message queue #{queue_len} for #{inspect(pid)}: #{inspect(info)}")
      end
    end)
  end

  defp log_top_procs do
    :recon.proc_count(:memory, 5)
    |> Enum.each(fn {pid, mem, [initial_call: call | _]} ->
      Logger.info("[Recon] top proc #{inspect(pid)} #{div(mem, 1024)}KB #{inspect(call)}")
    end)
  end

  defp ping_vigilmon do
    url = Application.get_env(:my_app, :vigilmon)[:heartbeat_url]

    if url do
      case Req.get(url, receive_timeout: 5_000) do
        {:ok, %{status: 200}} -> :ok
        err -> Logger.warning("[ReconDiagnosticsWorker] heartbeat ping failed: #{inspect(err)}")
      end
    end
  end

  defp schedule, do: Process.send_after(self(), :diagnose, @interval)
end

Add it to your supervision tree:

# lib/my_app/application.ex
children = [
  MyApp.Repo,
  MyAppWeb.Endpoint,
  MyApp.Workers.ReconDiagnosticsWorker
]

In Vigilmon, create a Heartbeat Monitor:

1. Click New Monitor → Heartbeat
2. Set expected interval: 10 minutes
3. Copy the ping URL
4. Set it as VIGILMON_HEARTBEAT_URL in your environment

If the diagnostics worker crashes or the Erlang VM hangs, the heartbeat stops — and Vigilmon alerts you.

Step 6: Use Recon tracing for intermittent issues

When Vigilmon alerts you to degradation, Recon's tracing tools help you investigate without stopping the application:

# Trace all calls to a slow function (limit to 10 calls)
:recon_trace.calls({MyApp.SomeModule, :slow_function, :_}, 10)

# Trace with message content
:recon_trace.calls({MyApp.Repo, :all, :_}, 5, scope: :local)

# Stop all tracing when done
:recon_trace.clear()

You can also use Recon to sample scheduler utilization during an alert window:

# Get scheduler utilization over 1 second
:recon.scheduler_usage(1_000)
# [{1, 0.23}, {2, 0.18}, {3, 0.41}, {4, 0.09}]

High scheduler utilization combined with a large run queue (visible in your health endpoint) pinpoints CPU-bound bottlenecks.

Step 7: Slack alerts and status page

Slack alerts:

1. In Vigilmon, go to Notifications → New Channel → Slack
2. Paste your Slack incoming webhook URL
3. Enable the channel on your Recon health monitor and heartbeat monitor

When memory or process count spikes:

🔴 DOWN: yourdomain.com/health
Keyword "status":"ok" not found
Detected: EU-West
2 minutes ago

When the diagnostics heartbeat misses:

🔴 HEARTBEAT MISSED: Recon Diagnostics
Expected ping within 10 minutes — none received
Last seen: 13 minutes ago

Status page:

1. Go to Status Pages → New Status Page
2. Add your monitors
3. Share the URL with your team

What you've built

| What | How | |------|-----| | Health endpoint | Plug with Recon system, memory, and process metrics | | Keyword check | Vigilmon keyword match on "status":"ok" | | Uptime monitoring | Vigilmon HTTP monitor → /health | | Process leak detection | Process count and message queue depth | | Memory monitoring | recon_alloc.memory/1 with fragmentation ratio | | Binary leak detection | Top processes by binary memory | | Scheduled diagnostics | GenServer + recon.proc_count/2 + heartbeat | | Slack alerts | Vigilmon Slack notification channel | | Status page | Vigilmon public status page |

Recon gives you deep runtime introspection of your BEAM system. Vigilmon tells you when to pick it up.

Next steps

• Set tighter thresholds for run queue depth — a consistently non-zero run queue indicates scheduler saturation before users feel it
• Add periodic GC pressure monitoring: recon_alloc.memory(:allocated) vs memory(:used) — a large gap indicates fragmentation worth compacting
• Use Vigilmon's response time history to correlate latency spikes with GC pauses or message queue growth visible in your health response

Get started free at vigilmon.online.