How to Monitor Firenest with Vigilmon

Firenest is the low-level distributed messaging library underlying Phoenix.PubSub. It provides a topology abstraction and point-to-point messaging between Erlang/Elixir cluster nodes. Where PubSub gives you broadcast semantics on top of a topic tree, Firenest gives you the primitives: node identity, cluster membership, and direct node-to-node message delivery. Phoenix.PubSub.PG2 and Phoenix.PubSub.Redis are both built on Firenest topologies.

When Firenest topology breaks down, the effects are felt system-wide: Phoenix.PubSub broadcast stops reaching all subscribers, LiveView updates fail to propagate across nodes, and any distributed Elixir feature that relies on cluster messaging degrades silently. Standard HTTP monitoring won't catch this — your app is up, but cluster coherence is gone.

Vigilmon heartbeat monitors let you verify that Firenest topology is connected and messages are flowing between nodes, not just that the local process is running.

Why Monitor Firenest?

Firenest is infrastructure — when it fails, everything built on top of it fails too:

• Topology startup failure — if Firenest.Topology fails to start or initialize, no distributed messaging is possible; the failure is often silent because supervisors restart it without detecting the root cause
• Node connectivity loss — a network partition, DNS failure, or Erlang cookie mismatch causes nodes to drop from the topology; Firenest.Topology.nodes/1 returns a stale or incomplete list
• Message delivery silently dropped — Firenest delivers messages best-effort; if a target node is temporarily unreachable, the message is not queued or retried; callers see no error
• Partition asymmetry — node A sees node B in its topology; node B does not see node A; messages from A reach B but not the reverse; this asymmetry is invisible to health checks that only read local state
• Clock skew causing handshake failure — Erlang distribution protocols are sensitive to extreme clock skew between nodes; if NTP drifts, nodes may fail to re-establish distribution connections after restarts

Key Metrics to Track

| Metric | What it tells you | |--------|-------------------| | Topology node count | Whether all expected cluster members are present | | Topology process memory | Accumulation of undelivered message buffers | | Broadcast round-trip success | Whether point-to-point delivery is working end-to-end | | Remote node RPC latency | Whether distribution overhead is within expected bounds | | Topology crash/restart count | How often the topology supervisor is restarting | | Message delivery failure rate | Proportion of sends to unreachable nodes |

Step 1: Configure Firenest Topology

# mix.exs
defp deps do
  [
    {:firenest, "~> 0.4"}
  ]
end

# lib/my_app/application.ex
children = [
  {Firenest.Topology, [name: MyApp.Topology, adapter: Firenest.Topology.Erlang]},
  # Phoenix.PubSub uses Firenest topology by default
  {Phoenix.PubSub, [name: MyApp.PubSub, adapter: Phoenix.PubSub.PG2]},
  MyAppWeb.Endpoint
]

Step 2: Build a Topology Health Check

# lib/my_app/firenest_health.ex
defmodule MyApp.FirenestHealth do
  @expected_min_nodes Application.compile_env(:my_app, :expected_cluster_size, 1)

  def check do
    case Process.whereis(MyApp.Topology) do
      nil ->
        %{status: :error, reason: "Firenest.Topology process not running"}

      pid ->
        memory = Process.info(pid, :memory) |> elem(1)
        nodes = Firenest.Topology.nodes(MyApp.Topology)
        node_count = length(nodes) + 1  # +1 for self

        %{
          status: topology_status(node_count),
          topology_pid: inspect(pid),
          memory_bytes: memory,
          self: to_string(Firenest.Topology.node(MyApp.Topology)),
          connected_nodes: Enum.map(nodes, &to_string/1),
          node_count: node_count,
          expected_min_nodes: @expected_min_nodes
        }
    end
  rescue
    e -> %{status: :error, reason: inspect(e)}
  end

  defp topology_status(count) when count >= @expected_min_nodes, do: :ok
  defp topology_status(_), do: :degraded
end

Step 3: Verify Point-to-Point Message Delivery

For true end-to-end verification, send a message to each peer node and verify receipt:

# lib/my_app/firenest_probe_server.ex
defmodule MyApp.FirenestProbeServer do
  use GenServer

  @probe_timeout 3_000

  def start_link(_), do: GenServer.start_link(__MODULE__, %{}, name: __MODULE__)

  def init(state), do: {:ok, state}

  def handle_info({:firenest_probe, from_pid, ref}, state) do
    send(from_pid, {:firenest_probe_ack, ref})
    {:noreply, state}
  end

  def handle_info(_, state), do: {:noreply, state}

  def probe_node(target_node) do
    ref = make_ref()
    Firenest.Topology.send(MyApp.Topology, target_node, MyApp.FirenestProbeServer, {:firenest_probe, self(), ref})

    receive do
      {:firenest_probe_ack, ^ref} -> :ok
    after
      @probe_timeout -> {:error, :timeout}
    end
  end
end

# lib/my_app/firenest_connectivity.ex
defmodule MyApp.FirenestConnectivity do
  def check do
    nodes = Firenest.Topology.nodes(MyApp.Topology)

    if nodes == [] do
      {:ok, :single_node}
    else
      results =
        Enum.map(nodes, fn node ->
          {node, MyApp.FirenestProbeServer.probe_node(node)}
        end)

      failed = Enum.filter(results, fn {_, r} -> r != :ok end)

      if failed == [] do
        {:ok, %{nodes_probed: length(nodes), all_reachable: true}}
      else
        {:degraded, %{failed: Enum.map(failed, fn {n, r} -> %{node: to_string(n), reason: inspect(r)} end)}}
      end
    end
  rescue
    e -> {:error, %{reason: inspect(e)}}
  end
end

Step 4: Expose a Health Endpoint

# lib/my_app_web/controllers/health_controller.ex
defmodule MyAppWeb.HealthController do
  use MyAppWeb, :controller

  def index(conn, _params) do
    topology = MyApp.FirenestHealth.check()
    connectivity = MyApp.FirenestConnectivity.check()

    topology_ok = topology.status == :ok
    connectivity_ok = elem(connectivity, 0) == :ok

    status = if topology_ok and connectivity_ok, do: 200, else: 503

    conn
    |> put_status(status)
    |> json(%{
      status: if(status == 200, do: "ok", else: "degraded"),
      node: Node.self() |> to_string(),
      topology: topology,
      connectivity: format_connectivity(connectivity)
    })
  end

  defp format_connectivity({:ok, :single_node}), do: %{status: "ok", mode: "single_node"}
  defp format_connectivity({:ok, data}), do: Map.merge(%{status: "ok"}, data)
  defp format_connectivity({:degraded, data}), do: Map.merge(%{status: "degraded"}, data)
  defp format_connectivity({:error, data}), do: Map.merge(%{status: "error"}, data)
end

# lib/my_app_web/router.ex
scope "/", MyAppWeb do
  get "/health", HealthController, :index
end

Step 5: Emit Telemetry Metrics

# lib/my_app/firenest_poller.ex
defmodule MyApp.FirenestPoller do
  use GenServer
  require Logger

  @interval :timer.seconds(15)

  def start_link(_), do: GenServer.start_link(__MODULE__, %{}, name: __MODULE__)

  def init(state) do
    schedule()
    {:ok, state}
  end

  def handle_info(:poll, state) do
    report()
    schedule()
    {:noreply, state}
  end

  defp report do
    pid = Process.whereis(MyApp.Topology)

    if pid do
      memory = Process.info(pid, :memory) |> elem(1)
      node_count = length(Firenest.Topology.nodes(MyApp.Topology)) + 1

      :telemetry.execute(
        [:my_app, :firenest, :topology],
        %{memory_bytes: memory, node_count: node_count},
        %{node: Node.self()}
      )
    else
      Logger.warning("FirenestPoller: MyApp.Topology process not running")
    end
  rescue
    e -> Logger.warning("FirenestPoller error: #{inspect(e)}")
  end

  defp schedule, do: Process.send_after(self(), :poll, @interval)
end

Step 6: Create Monitors in Vigilmon

HTTP monitor for the health endpoint:

1. Sign in at vigilmon.online
2. Click New Monitor → HTTP
3. URL: https://your-app.example.com/health
4. Interval: 60 seconds
5. Alert condition: non-200 or body contains "topology":{"status":"degraded"}

Heartbeat monitor for topology liveness:

# lib/my_app/firenest_heartbeat.ex
defmodule MyApp.FirenestHeartbeat do
  use GenServer

  @interval :timer.minutes(1)

  def start_link(_), do: GenServer.start_link(__MODULE__, %{}, name: __MODULE__)

  def init(state) do
    schedule()
    {:ok, state}
  end

  def handle_info(:ping, state) do
    health = MyApp.FirenestHealth.check()

    if health.status == :ok do
      url = System.get_env("VIGILMON_FIRENEST_HEARTBEAT_URL")

      if url do
        :httpc.request(:get, {String.to_charlist(url), []}, [{:timeout, 5_000}], [])
      end
    end

    schedule()
    {:noreply, state}
  end

  defp schedule, do: Process.send_after(self(), :ping, @interval)
end

In Vigilmon, create a Heartbeat monitor with a 3-minute expected interval.

Step 7: Alerting

In Vigilmon, go to Notifications → New Channel:

Slack for topology degradation:

🔴 DOWN: Firenest Topology — MyApp
Monitor: /health returning topology.status = "degraded"
Node: myapp@10.0.0.1
Action: Check node connectivity, Erlang cookie, and distribution port reachability

PagerDuty for heartbeat miss — a missed Firenest heartbeat means the topology process is down or point-to-point messaging has stalled, which disables Phoenix.PubSub broadcast cluster-wide.

Common Firenest Issues and Fixes

Nodes missing from topology after restart:

# Verify Erlang distribution is running and cookie matches
iex> Node.list()
# If empty and nodes are reachable, check:
# 1. --sname / --name flags in release config
# 2. RELEASE_COOKIE / --cookie are consistent across nodes
# 3. Port 4369 (epmd) is reachable between nodes

Asymmetric topology (A sees B but B doesn't see A):

# From each node's IEx, check topology.nodes
iex(node_a@host)> Firenest.Topology.nodes(MyApp.Topology)
[:"node_b@host"]

iex(node_b@host)> Firenest.Topology.nodes(MyApp.Topology)
[]  # <-- asymmetric; trigger reconnect

# Force reconnect attempt
iex(node_b@host)> Node.connect(:"node_a@host")

High topology process memory:

# Inspect topology process state
iex> :sys.get_state(MyApp.Topology)
# Look for pending message queues to unreachable nodes
# Topology buffers messages for nodes it believes are temporarily unreachable

What You've Built

| What | How | |------|-----| | Topology process check | Verify Firenest.Topology process alive and node count | | Point-to-point probe | Send and receive messages to each peer node | | Structured health endpoint | JSON health response with topology and connectivity status | | Node count monitoring | Telemetry on connected node count every 15s | | Liveness heartbeat | GenServer pinging Vigilmon when topology is healthy | | Real-time alerting | Vigilmon HTTP monitor + PagerDuty on heartbeat miss |

Firenest is the messaging foundation for Phoenix.PubSub and distributed Elixir clustering. Vigilmon makes sure the foundation is intact.

Next Steps

• Add a monitor per cluster region if you run multi-region Elixir deployments on Fly.io or Render
• Alert on node count dropping below half the expected cluster size to catch split-brain scenarios
• Monitor Phoenix.PubSub broadcast latency using Firenest point-to-point probes as a baseline
• Use Vigilmon's downtime history to correlate Firenest topology gaps with user-reported issues

Get started free at vigilmon.online.