How to Monitor Rexbug with Vigilmon

Rexbug is a production-safe Elixir wrapper around Redbug, the Erlang tracer. It lets you attach function-call, message-passing, and process-activity traces to a live BEAM node without halting the system or flooding your logs — a critical capability when you need to understand what a running node is actually doing.

But tracing a production node introduces its own risks: a runaway trace can consume memory, slow the VM, or silently stop collecting data if the node becomes overwhelmed. You need external monitoring to know your node stays healthy while Rexbug is active, and to catch cases where automated trace-collection jobs stall or error without alerting anyone.

This tutorial covers:

• A health check endpoint that exposes BEAM VM stats relevant to tracing load
• HTTP uptime monitoring with Vigilmon
• Heartbeat monitoring for automated Rexbug trace-collection scripts
• Alerts when tracing sessions cause memory or process-count spikes

Why Monitor Rexbug?

| Failure mode | Symptom without monitoring | |---|---| | BEAM node unreachable | Rexbug sessions silently fail to connect | | Trace session overwhelms memory | OOM kill with no warning | | Process count explosion from message tracing | Scheduler contention, latency spikes | | Automated trace collector crashes | No trace data collected; no alert | | Node restarted mid-session | Trace data lost; collector waits forever |

Step 1: Expose BEAM health metrics for tracing context

Add a lightweight health endpoint that surfaces the VM stats most relevant when Rexbug is active.

# lib/my_app_web/plugs/health_check.ex
defmodule MyAppWeb.Plugs.HealthCheck do
  import Plug.Conn

  def init(opts), do: opts

  def call(%Plug.Conn{request_path: "/health"} = conn, _opts) do
    checks = %{
      database: check_database(),
      memory:   check_memory(),
      processes: check_process_count()
    }

    status = if Enum.all?(checks, fn {_, v} -> v == :ok end), do: 200, else: 503

    body = Jason.encode!(%{
      status: if(status == 200, do: "ok", else: "degraded"),
      checks: checks,
      vm: vm_stats()
    })

    conn
    |> put_resp_content_type("application/json")
    |> send_resp(status, body)
    |> halt()
  end

  def call(conn, _opts), do: conn

  defp check_database do
    case Ecto.Adapters.SQL.query(MyApp.Repo, "SELECT 1", []) do
      {:ok, _} -> :ok
      _ -> :error
    end
  end

  defp check_memory do
    memory = :erlang.memory()
    total_mb = memory[:total] / 1_048_576
    if total_mb < 1_500, do: :ok, else: :error
  end

  defp check_process_count do
    count = length(:erlang.processes())
    if count < 50_000, do: :ok, else: :error
  end

  defp vm_stats do
    memory = :erlang.memory()
    %{
      process_count: length(:erlang.processes()),
      memory_total_mb: Float.round(memory[:total] / 1_048_576, 1),
      memory_processes_mb: Float.round(memory[:processes] / 1_048_576, 1),
      memory_atom_mb: Float.round(memory[:atom] / 1_048_576, 1),
      schedulers_online: :erlang.system_info(:schedulers_online)
    }
  end
end

Register the plug before your router:

# lib/my_app_web/endpoint.ex
plug MyAppWeb.Plugs.HealthCheck
plug MyAppWeb.Router

Test it:

mix phx.server
curl http://localhost:4000/health | jq .
# {
#   "status": "ok",
#   "checks": {"database": "ok", "memory": "ok", "processes": "ok"},
#   "vm": {"process_count": 312, "memory_total_mb": 48.3, ...}
# }

The vm block gives Vigilmon's keyword monitor something to diff against — you can alert if memory_total_mb crosses a threshold that would not show up as a plain 503.

Step 2: Set up HTTP monitoring in Vigilmon

1. Sign up at vigilmon.online.
2. Click New Monitor → HTTP.
3. Enter https://your-app.example.com/health.
4. Set check interval to 60 seconds.
5. Add a keyword check for "ok" — this catches degraded responses that still return HTTP 200.
6. Set alert threshold to 2 consecutive failures to avoid false positives during brief GC pauses.

Step 3: Heartbeat monitoring for automated trace collectors

Rexbug's most common production use is automated trace collection — a script or GenServer that attaches a trace, collects results for N seconds, then processes and stores them. If this collector stalls (node restart, timeout, OOM), no trace data is gathered and no one knows.

Add a heartbeat ping at the end of each successful trace run:

# lib/my_app/trace_collector.ex
defmodule MyApp.TraceCollector do
  use GenServer
  require Logger

  @interval :timer.minutes(15)
  @heartbeat_url System.get_env("VIGILMON_TRACE_HEARTBEAT_URL")

  def start_link(_), do: GenServer.start_link(__MODULE__, %{}, name: __MODULE__)

  @impl true
  def init(state) do
    schedule_run()
    {:ok, state}
  end

  @impl true
  def handle_info(:run, state) do
    case collect_trace() do
      {:ok, data} ->
        store_trace(data)
        ping_heartbeat()
      {:error, reason} ->
        Logger.error("TraceCollector failed: #{inspect(reason)}")
        # No ping → Vigilmon alerts after grace period
    end

    schedule_run()
    {:noreply, state}
  end

  defp collect_trace do
    {:ok, pid} = Rexbug.start("MyApp.SomeModule.some_function/2", time: 10_000, msgs: 100)
    receive do
      :done -> {:ok, []}
    after
      12_000 ->
        Rexbug.stop(pid)
        {:error, :timeout}
    end
  end

  defp store_trace(data), do: Logger.info("Trace: #{inspect(data)}")

  defp ping_heartbeat do
    if @heartbeat_url do
      Task.start(fn ->
        :httpc.request(:get, {String.to_charlist(@heartbeat_url), []}, [], [])
      end)
    end
  end

  defp schedule_run, do: Process.send_after(self(), :run, @interval)
end

Add to your supervision tree:

# lib/my_app/application.ex
children = [
  MyApp.Repo,
  MyAppWeb.Endpoint,
  MyApp.TraceCollector,  # ← add
]

In Vigilmon, create a Heartbeat Monitor:

1. Click New Monitor → Heartbeat.
2. Set grace period to 20 minutes (collector interval 15 min + buffer).
3. Copy the ping URL into VIGILMON_TRACE_HEARTBEAT_URL in your environment.

Step 4: Key metrics to alert on

| Metric | Alert threshold | Why | |---|---|---| | /health HTTP status | Non-200 for 2 checks | Node is unreachable | | memory_total_mb in response | > 1 500 MB | Tracing session bloating memory | | process_count in response | > 50 000 | Message tracing spawning excess processes | | Heartbeat silence | Grace period exceeded | Trace collector hung or node restarted |

Use Vigilmon's keyword check with a body match on "degraded" to alert on VM stat thresholds without requiring a full 503.

Step 5: Status page

1. In Vigilmon, go to Status Pages → New.
2. Add your HTTP monitor and heartbeat monitor.
3. Label them "Node health (web)" and "Trace collector".
4. Share the URL in your team's on-call runbook so engineers know where to look when a Rexbug session goes wrong.

Conclusion

Rexbug makes production tracing safe — but it does not make the underlying BEAM node immune to overload or silent failure. With Vigilmon you get:

• Uptime checks that detect when the node is unreachable or memory-constrained
• Heartbeat monitors that catch stalled or erroring trace-collection jobs
• Alerts before a runaway trace session becomes a production incident

Start with the /health endpoint and one heartbeat, then add VM-stat keyword checks as you grow your Rexbug usage.

Get started free at vigilmon.online.