How to Monitor Recon with Vigilmon

Recon is Fred Hebert's production-safe toolkit for introspecting live Erlang and Elixir nodes. It gives you process tracing, memory allocation analysis, scheduler utilisation, and safe message-count inspection — all designed to run on production nodes without causing the very incident you are trying to diagnose.

Recon is diagnostic by nature, which creates a subtle monitoring gap: you reach for Recon when something is already wrong, but external monitoring should tell you before things break. The solution is to run Recon's cheapest, safest introspection calls from your health endpoint on a schedule, surfacing degradation signals to Vigilmon before they become incidents.

This tutorial covers:

• A health endpoint powered by Recon's node introspection
• HTTP uptime monitoring with Vigilmon
• Heartbeat monitoring for periodic Recon diagnostic workers
• Alerts calibrated to Recon's memory and process metrics

Why Monitor Recon?

| Failure mode | Symptom without monitoring | |---|---| | Node memory climbing toward OOM | No alert until crash | | Process mailbox backlog building | Latency spikes; hard to diagnose retrospectively | | Binary heap growing unchecked | Gradual memory leak; invisible until GC fails | | Diagnostic worker crashes | No fresh Recon snapshots; blind during next incident | | Node partitioned from load balancer | HTTP calls time out; Recon sessions fail to connect |

Step 1: Health endpoint powered by Recon

Add Recon to your dependencies:

# mix.exs
{:recon, "~> 2.5"}

Build a health endpoint that uses Recon's safe introspection functions:

# lib/my_app_web/plugs/health_check.ex
defmodule MyAppWeb.Plugs.HealthCheck do
  import Plug.Conn

  def init(opts), do: opts

  def call(%Plug.Conn{request_path: "/health"} = conn, _opts) do
    node_stats = collect_recon_stats()
    db_ok      = check_database()
    mem_ok     = node_stats.memory_total_mb < 2_000
    proc_ok    = node_stats.top_mailbox_size < 1_000

    all_ok = db_ok and mem_ok and proc_ok
    status = if all_ok, do: 200, else: 503

    body = Jason.encode!(%{
      status: if(all_ok, do: "ok", else: "degraded"),
      checks: %{
        database:     if(db_ok,    do: "ok", else: "error"),
        memory:       if(mem_ok,   do: "ok", else: "error"),
        process_mailboxes: if(proc_ok, do: "ok", else: "error")
      },
      node: node_stats
    })

    conn
    |> put_resp_content_type("application/json")
    |> send_resp(status, body)
    |> halt()
  end

  def call(conn, _opts), do: conn

  defp collect_recon_stats do
    memory = :recon_alloc.memory(:allocated)
    proc_info = :recon.proc_count(:message_queue_len, 1)

    top_mailbox =
      case proc_info do
        [{_pid, count, _info} | _] -> count
        _ -> 0
      end

    scheduler_usage =
      case :recon.scheduler_usage(500) do
        usage when is_list(usage) -> Enum.max(Enum.map(usage, &elem(&1, 1)))
        _ -> 0.0
      end

    %{
      memory_total_mb:    Float.round(memory / 1_048_576, 1),
      top_mailbox_size:   top_mailbox,
      scheduler_max_util: Float.round(scheduler_usage * 100, 1),
      process_count:      :recon.info(:recon_lib.self(), :process_count) |> extract_count()
    }
  end

  defp extract_count({:process_count, n}) when is_integer(n), do: n
  defp extract_count(_), do: length(:erlang.processes())

  defp check_database do
    case Ecto.Adapters.SQL.query(MyApp.Repo, "SELECT 1", []) do
      {:ok, _} -> true
      _ -> false
    end
  end
end

Wire it in before your router:

# lib/my_app_web/endpoint.ex
plug MyAppWeb.Plugs.HealthCheck
plug MyAppWeb.Router

Test:

curl http://localhost:4000/health | jq .
# {
#   "status": "ok",
#   "node": {
#     "memory_total_mb": 62.4,
#     "top_mailbox_size": 0,
#     "scheduler_max_util": 3.2,
#     "process_count": 418
#   }
# }

Step 2: Set up HTTP monitoring in Vigilmon

1. Sign up at vigilmon.online.
2. Click New Monitor → HTTP.
3. Enter https://your-app.example.com/health.
4. Set check interval to 60 seconds.
5. Add a keyword check for "ok" so a "degraded" response body triggers an alert even on HTTP 200.
6. Set alert threshold to 2 consecutive failures to avoid scheduler-sampling noise.

Step 3: Periodic Recon diagnostic worker with heartbeat

Schedule a background GenServer that runs deeper Recon analysis every 30 minutes and stores a snapshot. If it stalls or crashes, Vigilmon's heartbeat monitor will alert you.

# lib/my_app/recon_diagnostics.ex
defmodule MyApp.ReconDiagnostics do
  use GenServer
  require Logger

  @interval :timer.minutes(30)

  def start_link(_), do: GenServer.start_link(__MODULE__, %{}, name: __MODULE__)

  @impl true
  def init(state) do
    schedule_run()
    {:ok, state}
  end

  @impl true
  def handle_info(:run, state) do
    snapshot = collect_snapshot()
    persist_snapshot(snapshot)
    ping_heartbeat()
    schedule_run()
    {:noreply, state}
  end

  defp collect_snapshot do
    %{
      timestamp:        System.system_time(:second),
      top_memory_procs: :recon.proc_count(:memory, 5),
      top_mailbox_procs: :recon.proc_count(:message_queue_len, 5),
      alloc_memory_mb:  Float.round(:recon_alloc.memory(:allocated) / 1_048_576, 1),
      bin_memory_mb:    Float.round(:recon_alloc.memory(:binary) / 1_048_576, 1)
    }
  end

  defp persist_snapshot(snapshot) do
    Logger.info("[ReconDiagnostics] #{inspect(snapshot, limit: 3)}")
    # Replace with your preferred store: ETS, database, telemetry event, etc.
  end

  defp ping_heartbeat do
    url = Application.get_env(:my_app, :vigilmon)[:diagnostics_heartbeat_url]
    if url do
      Task.start(fn ->
        :httpc.request(:get, {String.to_charlist(url), []}, [], [])
      end)
    end
  end

  defp schedule_run, do: Process.send_after(self(), :run, @interval)
end

Add it to your supervision tree:

children = [
  MyApp.Repo,
  MyAppWeb.Endpoint,
  MyApp.ReconDiagnostics,
]

Configure the heartbeat URL:

# config/runtime.exs
config :my_app, :vigilmon,
  diagnostics_heartbeat_url: System.get_env("VIGILMON_DIAGNOSTICS_HEARTBEAT_URL")

In Vigilmon, create a Heartbeat Monitor:

1. Click New Monitor → Heartbeat.
2. Set grace period to 40 minutes (30-min interval + 10-min buffer).
3. Copy the ping URL into VIGILMON_DIAGNOSTICS_HEARTBEAT_URL.

Step 4: Key metrics to alert on

| Metric | Alert threshold | Why | |---|---|---| | /health HTTP status | Non-200 for 2 checks | Node unreachable | | memory_total_mb | > 2 000 MB | OOM risk; Recon memory allocated exceeding safe bounds | | top_mailbox_size | > 1 000 messages | Process mailbox backlog accumulating | | scheduler_max_util | > 95% | Scheduler saturation; node at capacity | | Heartbeat silence | Grace period exceeded | Diagnostic worker crashed or node restarted |

Step 5: Status page

Add your monitors to a public status page:

1. In Vigilmon, go to Status Pages → New.
2. Add the HTTP monitor ("Node health") and the heartbeat monitor ("Diagnostic worker").
3. Share the URL with your on-call team.

During a production incident, the status page tells you instantly whether the node is up and whether the last Recon snapshot is fresh — two pieces of information that determine how you triage.

Conclusion

Recon is the toolkit you reach for when a production node is misbehaving. Vigilmon is what tells you the node is misbehaving in the first place — and what confirms it has recovered. Together they give you:

• Proactive visibility into memory, mailbox, and scheduler health via Recon-backed endpoints
• Uptime checks that detect node unavailability before your users do
• Heartbeat monitors that ensure diagnostic snapshots are being collected continuously

Start with the /health endpoint, add the diagnostic worker heartbeat, and you will have the context you need before you even open an IEx session.

Get started free at vigilmon.online.