How to Monitor observer_cli with Vigilmon

observer_cli is an interactive command-line observer for Erlang and Elixir BEAM VMs. Without a GUI connection, it provides real-time system monitoring of processes, ports, ETS tables, memory usage, message queues, and garbage collection — all from a plain terminal session into your production node. It's the fastest way to diagnose a process that's accumulating messages or a memory leak that's building toward OOM.

But observer_cli is reactive: you run it when you suspect a problem. Vigilmon is proactive: it alerts you before you know there's a problem. Together they give you both inside-out VM introspection and outside-in uptime monitoring. This tutorial covers:

• Exposing BEAM VM metrics in a health endpoint so Vigilmon can alert on thresholds
• HTTP uptime monitoring with Vigilmon
• Heartbeat monitoring for BEAM housekeeping jobs
• Slack alerts and a status page

Step 1: Add observer_cli to your project

# mix.exs
defp deps do
  [
    {:observer_cli, "~> 1.7"},
    {:req, "~> 0.4"}   # for heartbeat pings
  ]
end

Start an observer_cli session against a running node:

# Connect to your production node via IEx remote shell
iex --name debug@yourhost --cookie your_cookie --remsh app@yourhost

# Then start observer_cli:
:observer_cli.start()

You'll see a live dashboard of:

• Process count, reductions, memory
• Top processes by message queue length, heap size, and reduction rate
• Port table and ETS table sizes
• System-wide scheduler utilization

Step 2: Build a health endpoint that exposes BEAM VM metrics

observer_cli reads its data from the same BEAM introspection APIs available to your application. Expose those same metrics in a health endpoint so Vigilmon can alert on thresholds without needing a human to connect:

# lib/my_app_web/plugs/health_check.ex
defmodule MyAppWeb.Plugs.HealthCheck do
  import Plug.Conn
  require Logger

  # Alert thresholds
  @max_process_count 50_000
  @max_memory_mb 1_500
  @max_message_queue 1_000
  @max_gc_words_reclaimed_per_sec 500_000

  def init(opts), do: opts

  def call(%Plug.Conn{request_path: "/health"} = conn, _opts) do
    checks = run_checks()
    status = if checks.status == "ok", do: 200, else: 503

    conn
    |> put_resp_content_type("application/json")
    |> send_resp(status, Jason.encode!(checks))
    |> halt()
  end

  def call(conn, _opts), do: conn

  defp run_checks do
    vm = vm_check()
    memory = memory_check()
    processes = process_check()

    overall = vm.status == "ok" and memory.status == "ok" and processes.status == "ok"

    %{
      status: if(overall, do: "ok", else: "degraded"),
      vm: vm,
      memory: memory,
      processes: processes
    }
  end

  defp vm_check do
    info = :erlang.system_info(:system_version) |> to_string()
    uptime_ms = :erlang.statistics(:wall_clock) |> elem(0)

    %{
      status: "ok",
      otp_version: info,
      uptime_ms: uptime_ms,
      schedulers: :erlang.system_info(:schedulers_online)
    }
  end

  defp memory_check do
    memory = :erlang.memory()
    total_mb = div(memory[:total], 1_048_576)
    process_mb = div(memory[:processes], 1_048_576)
    atom_mb = div(memory[:atom], 1_048_576)
    binary_mb = div(memory[:binary], 1_048_576)
    ets_mb = div(memory[:ets], 1_048_576)

    status = if total_mb > @max_memory_mb, do: "degraded", else: "ok"

    if status == "degraded" do
      Logger.warning("[VM] memory usage #{total_mb}MB exceeds threshold #{@max_memory_mb}MB")
    end

    %{
      status: status,
      total_mb: total_mb,
      process_mb: process_mb,
      atom_mb: atom_mb,
      binary_mb: binary_mb,
      ets_mb: ets_mb
    }
  end

  defp process_check do
    process_count = :erlang.system_info(:process_count)
    process_limit = :erlang.system_info(:process_limit)

    # Find the longest message queue
    max_queue =
      Process.list()
      |> Enum.map(fn pid ->
        case Process.info(pid, :message_queue_len) do
          {:message_queue_len, n} -> n
          nil -> 0
        end
      end)
      |> Enum.max(fn -> 0 end)

    status =
      cond do
        process_count > @max_process_count -> "degraded"
        max_queue > @max_message_queue -> "degraded"
        true -> "ok"
      end

    if status == "degraded" do
      Logger.warning(
        "[VM] process health degraded: count=#{process_count}, max_queue=#{max_queue}"
      )
    end

    %{
      status: status,
      process_count: process_count,
      process_limit: process_limit,
      max_message_queue_len: max_queue
    }
  end
end

Register the plug:

# lib/my_app_web/endpoint.ex
plug MyAppWeb.Plugs.HealthCheck
plug MyAppWeb.Router

Test it:

mix phx.server
curl http://localhost:4000/health | jq .
# {
#   "status": "ok",
#   "vm": {
#     "status": "ok",
#     "otp_version": "Erlang/OTP 26...",
#     "uptime_ms": 12345,
#     "schedulers": 4
#   },
#   "memory": {
#     "status": "ok",
#     "total_mb": 128,
#     "process_mb": 64,
#     "atom_mb": 3,
#     "binary_mb": 22,
#     "ets_mb": 8
#   },
#   "processes": {
#     "status": "ok",
#     "process_count": 312,
#     "process_limit": 262144,
#     "max_message_queue_len": 0
#   }
# }

Step 3: Add ETS table monitoring

ETS tables can grow unboundedly if entries aren't expired. Track ETS size in your health response:

defp ets_check do
  tables =
    :ets.all()
    |> Enum.map(fn tab ->
      info = :ets.info(tab)
      %{
        name: inspect(info[:name]),
        size: info[:size],
        memory_words: info[:memory]
      }
    end)

  total_entries = Enum.sum(Enum.map(tables, & &1.size))
  large_tables = Enum.filter(tables, &(&1.size > 100_000))

  status = if large_tables == [], do: "ok", else: "degraded"

  if status == "degraded" do
    Logger.warning("[VM] large ETS tables detected: #{inspect(Enum.map(large_tables, & &1.name))}")
  end

  %{
    status: status,
    total_entries: total_entries,
    table_count: length(tables),
    large_tables: large_tables
  }
end

Add ets: ets_check() to your run_checks/0 map.

Step 4: Monitor the health endpoint with Vigilmon

1. Sign up at vigilmon.online
2. Click New Monitor → HTTP
3. Enter https://yourdomain.com/health
4. Set check interval: 1 minute
5. Enable keyword check: body must contain "status":"ok"
6. Save

When memory exceeds your threshold or a process accumulates a massive message queue, your endpoint returns "status":"degraded" even though the HTTP server is still responsive. The keyword check catches this before it escalates to an OOM crash.

Step 5: Heartbeat monitoring for BEAM housekeeping jobs

Periodic jobs that flush ETS caches, force GC, or prune dead processes need heartbeat confirmation:

# lib/my_app/workers/beam_housekeeping_worker.ex
defmodule MyApp.Workers.BeamHousekeepingWorker do
  use GenServer
  require Logger

  @interval :timer.minutes(10)

  def start_link(_), do: GenServer.start_link(__MODULE__, %{}, name: __MODULE__)

  def init(state) do
    schedule()
    {:ok, state}
  end

  def handle_info(:housekeep, state) do
    run_housekeeping()
    schedule()
    {:noreply, state}
  end

  defp run_housekeeping do
    # Force GC on all processes with large heaps
    Process.list()
    |> Enum.each(fn pid ->
      case Process.info(pid, :heap_size) do
        {:heap_size, size} when size > 100_000 ->
          :erlang.garbage_collect(pid)

        _ ->
          :ok
      end
    end)

    # Log current VM health snapshot
    process_count = :erlang.system_info(:process_count)
    memory_mb = div(:erlang.memory(:total), 1_048_576)
    Logger.info("[BeamHousekeeping] processes=#{process_count} memory=#{memory_mb}MB")

    ping_vigilmon()
  end

  defp ping_vigilmon do
    url = Application.get_env(:my_app, :vigilmon)[:heartbeat_url]

    if url do
      case Req.get(url, receive_timeout: 5_000) do
        {:ok, %{status: 200}} -> :ok
        err -> Logger.warning("[BeamHousekeeping] heartbeat ping failed: #{inspect(err)}")
      end
    end
  end

  defp schedule, do: Process.send_after(self(), :housekeep, @interval)
end

Add to your supervision tree:

# lib/my_app/application.ex
children = [
  MyApp.Repo,
  MyAppWeb.Endpoint,
  MyApp.Workers.BeamHousekeepingWorker
]

In Vigilmon, create a Heartbeat Monitor:

1. Click New Monitor → Heartbeat
2. Set expected interval: 20 minutes
3. Copy the ping URL
4. Set it as VIGILMON_HEARTBEAT_URL in your environment

Step 6: Using observer_cli for incident investigation

When Vigilmon alerts you, connect observer_cli to the running node to diagnose the cause interactively:

# SSH into the production server
ssh app@yourhost

# Attach to the running node
/app/bin/my_app remote

# In the IEx session:
:observer_cli.start()

In observer_cli, navigate to:

• [P]rocesses — sort by MSG (message queue) to find the stuck process
• [M]emory — find which process holds the most binary heap
• [E]TS — identify tables growing beyond expected bounds
• [G]C — observe GC pressure per scheduler

To inspect a specific PID from observer_cli output:

pid = :erlang.list_to_pid('<0.123.0>')
Process.info(pid, [:registered_name, :current_function, :message_queue_len, :heap_size, :backtrace])

To force GC on the suspected process:

:erlang.garbage_collect(pid)

Step 7: Slack alerts and status page

Slack alerts:

1. In Vigilmon, go to Notifications → New Channel → Slack
2. Paste your Slack incoming webhook URL
3. Enable the channel on your VM health monitor and housekeeping heartbeat

When memory exceeds the threshold:

🔴 DOWN: yourdomain.com/health
Keyword "status":"ok" not found
Detected: US-East
1 minute ago

When the housekeeping worker misses:

🔴 HEARTBEAT MISSED: BEAM Housekeeping Worker
Expected ping within 20 minutes — none received
Last seen: 24 minutes ago

Status page:

1. Go to Status Pages → New Status Page
2. Add your HTTP monitor and heartbeat monitor
3. Share with your on-call team for incident coordination

What you've built

| What | How | |------|-----| | VM metrics exposure | Process count, memory, message queue, ETS in health endpoint | | Uptime monitoring | Vigilmon HTTP monitor → /health | | Keyword check | Vigilmon keyword match on "status":"ok" | | Memory threshold alerting | total_mb > @max_memory_mb sets "status":"degraded" | | Message queue alerting | Max queue length across all processes exposed in health | | ETS table monitoring | Large table detection in health response | | Housekeeping job monitoring | Heartbeat GenServer + Vigilmon heartbeat monitor | | Incident investigation | observer_cli remote session for live BEAM introspection | | Slack alerts | Vigilmon Slack notification channel | | Status page | Vigilmon public status page |

observer_cli gives you deep BEAM introspection on demand. Vigilmon alerts you when introspection is urgently needed — so you arrive with context rather than catching the symptom after the crash.

Next steps

• Tune the alerting thresholds in your health endpoint by observing steady-state values with observer_cli during normal operation — baseline first, alert when you deviate
• Add scheduler utilization tracking using :erlang.statistics(:scheduler_wall_time) to detect CPU saturation before it manifests as latency
• Use Vigilmon's multi-region checks to confirm your BEAM node is reachable globally, since network partitions may affect only certain regions while the node itself is healthy

Get started free at vigilmon.online.