How to Monitor Phoenix.Tracker with Vigilmon

Phoenix.Tracker is the distributed presence tracking system built into Phoenix. It uses CRDTs (Conflict-free Replicated Data Types) to maintain a consistent view of connected entities — typically online users, connected devices, or active sessions — across every node in your cluster. Unlike Phoenix.Presence (a higher-level abstraction), Tracker gives you direct access to the CRDT-based state store, making it suitable for custom distributed tracking scenarios.

When Tracker's CRDT synchronization breaks down, your application's view of who is "online" becomes inconsistent. Node A might show 1,500 connected users while node B shows 800. Users who disconnected are still listed as present. New connections don't appear cluster-wide. These failures are silent from the perspective of HTTP monitoring — your application continues to serve responses while distributed presence is silently wrong.

Vigilmon heartbeat monitors let you verify that Tracker state is consistent and synchronized across your cluster, not just that the process is alive.

Why Monitor Phoenix.Tracker?

Tracker failures are subtle and often manifested as data inconsistency rather than errors:

• CRDT sync lag — after a node restart or network hiccup, the Tracker CRDT state on the rejoining node may be stale for several seconds while delta syncs propagate
• Ghost presences — a node crash without graceful shutdown leaves tracked entries in the CRDT that other nodes believe are still valid; they persist until the crashed node's clock delta expires
• Topic explosion — dynamically created tracker topics (e.g., one per chat room) that are never cleaned up accumulate in memory and slow down CRDT merge operations
• Delta flood — a node rejoining after a long partition sends a large CRDT delta; if not rate-limited, this can saturate the PubSub channel used for Tracker sync
• Partition asymmetry — during a network partition, each side maintains its own CRDT independently; after reconnection, entries from both sides merge, which can temporarily inflate presence counts
• Metadata bloat — tracking large metadata payloads (e.g., full user records instead of IDs) per presence increases CRDT merge cost and memory usage

Key Metrics to Track

| Metric | What it tells you | |--------|-------------------| | Tracked entry count per topic | Current presence count; spikes may indicate ghost presences | | CRDT sync delta size | Sync traffic; spikes after partitions indicate large state deltas | | Tracker process memory | Growing memory indicates topic or entry accumulation | | Cross-node count consistency | Whether all nodes agree on the same presence count | | Track/untrack rate | Healthy churn vs. spike indicating mass reconnect or disconnect | | Replica count per entry | Whether entries are replicated across expected nodes |

Step 1: Add a Custom Tracker Module

# lib/my_app/presence_tracker.ex
defmodule MyApp.PresenceTracker do
  use Phoenix.Tracker,
    name: __MODULE__,
    pubsub_server: MyApp.PubSub

  def start_link(opts) do
    opts = Keyword.merge([name: __MODULE__], opts)
    Phoenix.Tracker.start_link(__MODULE__, opts, opts)
  end

  @impl Phoenix.Tracker
  def init(opts) do
    server = Keyword.fetch!(opts, :pubsub_server)
    {:ok, %{pubsub_server: server}}
  end

  @impl Phoenix.Tracker
  def handle_diff(diff, state) do
    for {topic, {joins, leaves}} <- diff do
      for {key, meta} <- joins do
        :telemetry.execute(
          [:my_app, :tracker, :join],
          %{count: 1},
          %{topic: topic_class(topic)}
        )

        _ = {key, meta}
      end

      for {key, meta} <- leaves do
        :telemetry.execute(
          [:my_app, :tracker, :leave],
          %{count: 1},
          %{topic: topic_class(topic)}
        )

        _ = {key, meta}
      end
    end

    {:ok, state}
  end

  defp topic_class(topic) do
    topic |> String.split(":") |> List.first()
  end
end

# lib/my_app/application.ex
children = [
  {Phoenix.PubSub, name: MyApp.PubSub},
  MyApp.PresenceTracker,
  MyAppWeb.Endpoint
]

Step 2: Build a Tracker Health Check

# lib/my_app/tracker_health.ex
defmodule MyApp.TrackerHealth do
  @moduledoc """
  Checks Phoenix.Tracker health: process status, memory, and entry counts.
  """

  @health_topic "__tracker_health__"
  @health_key "probe"

  def check do
    case Process.whereis(MyApp.PresenceTracker) do
      nil ->
        %{status: :error, reason: "Tracker process not running"}

      pid ->
        memory = Process.info(pid, :memory) |> elem(1)
        entry_count = count_all_entries()

        track_result = probe_track()

        %{
          status: if(track_result == :ok, do: :ok, else: :degraded),
          tracker_pid: inspect(pid),
          memory_bytes: memory,
          total_entries: entry_count,
          probe: track_result
        }
    end
  rescue
    e -> %{status: :error, reason: inspect(e)}
  end

  defp probe_track do
    meta = %{health_probe: true, node: Node.self()}
    result = Phoenix.Tracker.track(MyApp.PresenceTracker, self(), @health_topic, @health_key, meta)

    case result do
      {:ok, _ref} ->
        list = Phoenix.Tracker.list(MyApp.PresenceTracker, @health_topic)
        Phoenix.Tracker.untrack(MyApp.PresenceTracker, self(), @health_topic, @health_key)

        if Enum.any?(list, fn {key, _} -> key == @health_key end) do
          :ok
        else
          :entry_not_found
        end

      {:error, reason} ->
        reason
    end
  end

  defp count_all_entries do
    # List entries across known topic classes
    # In production, maintain a registry of active topics
    try do
      Phoenix.Tracker.list(MyApp.PresenceTracker, "user")
      |> length()
    rescue
      _ -> 0
    end
  end
end

Step 3: Monitor Cross-Node Consistency

For clustered deployments, verify that all nodes agree on the same presence state:

# lib/my_app/tracker_consistency.ex
defmodule MyApp.TrackerConsistency do
  @health_topic "__tracker_health__"
  @tolerance 0.05

  def check_consistency do
    nodes = [Node.self() | Node.list()]

    if length(nodes) < 2 do
      {:ok, :single_node}
    else
      local_count = Phoenix.Tracker.list(MyApp.PresenceTracker, @health_topic) |> length()

      remote_counts =
        Enum.map(Node.list(), fn node ->
          case :rpc.call(node, Phoenix.Tracker, :list, [MyApp.PresenceTracker, @health_topic], 3000) do
            {:badrpc, _} -> nil
            list -> length(list)
          end
        end)
        |> Enum.reject(&is_nil/1)

      all_counts = [local_count | remote_counts]
      max_count = Enum.max(all_counts)
      min_count = Enum.min(all_counts)

      drift =
        if max_count > 0,
          do: (max_count - min_count) / max_count,
          else: 0.0

      if drift <= @tolerance do
        {:ok, %{drift: drift, counts: all_counts}}
      else
        {:degraded, %{drift: drift, counts: all_counts, max: max_count, min: min_count}}
      end
    end
  end
end

Step 4: Add a Health Endpoint

# lib/my_app_web/controllers/health_controller.ex
defmodule MyAppWeb.HealthController do
  use MyAppWeb, :controller

  def index(conn, _params) do
    tracker_check = MyApp.TrackerHealth.check()
    consistency_check = MyApp.TrackerConsistency.check_consistency()

    tracker_ok = tracker_check.status == :ok
    consistency_ok = elem(consistency_check, 0) in [:ok]

    status = if tracker_ok and consistency_ok, do: 200, else: 503

    conn
    |> put_status(status)
    |> json(%{
      status: if(status == 200, do: "ok", else: "degraded"),
      node: Node.self() |> to_string(),
      tracker: tracker_check,
      consistency: format_consistency(consistency_check)
    })
  end

  defp format_consistency({:ok, :single_node}), do: %{status: "ok", mode: "single_node"}
  defp format_consistency({:ok, data}), do: Map.merge(%{status: "ok"}, data)
  defp format_consistency({:degraded, data}), do: Map.merge(%{status: "degraded"}, data)
end

# lib/my_app_web/router.ex
scope "/", MyAppWeb do
  get "/health", HealthController, :index
end

Step 5: Instrument Tracker with Telemetry

# lib/my_app/tracker_poller.ex
defmodule MyApp.TrackerPoller do
  use GenServer
  require Logger

  @poll_interval :timer.seconds(30)
  @topics_to_monitor ["user", "room", "device"]

  def start_link(_), do: GenServer.start_link(__MODULE__, %{}, name: __MODULE__)

  def init(state) do
    schedule()
    {:ok, state}
  end

  def handle_info(:poll, state) do
    report_metrics()
    schedule()
    {:noreply, state}
  end

  defp report_metrics do
    case Process.whereis(MyApp.PresenceTracker) do
      nil ->
        Logger.warning("Phoenix.Tracker process not found")

      pid ->
        memory = Process.info(pid, :memory) |> elem(1)

        :telemetry.execute(
          [:my_app, :tracker, :memory],
          %{bytes: memory},
          %{}
        )

        Enum.each(@topics_to_monitor, fn topic_prefix ->
          count =
            Phoenix.Tracker.list(MyApp.PresenceTracker, topic_prefix)
            |> length()

          :telemetry.execute(
            [:my_app, :tracker, :entries],
            %{count: count},
            %{topic: topic_prefix}
          )
        end)
    end
  rescue
    e -> Logger.warning("TrackerPoller error: #{inspect(e)}")
  end

  defp schedule, do: Process.send_after(self(), :poll, @poll_interval)
end

Define Telemetry.Metrics:

[
  counter("my_app.tracker.join.count", tags: [:topic]),
  counter("my_app.tracker.leave.count", tags: [:topic]),
  last_value("my_app.tracker.entries.count", tags: [:topic]),
  last_value("my_app.tracker.memory.bytes")
]

Step 6: Create Monitors in Vigilmon

HTTP monitor for the health endpoint:

1. Sign in at vigilmon.online
2. Click New Monitor → HTTP
3. Set URL to https://your-app.example.com/health
4. Set interval to 60 seconds
5. Alert condition: status 200 and body contains "tracker":{"status":"ok"}

Heartbeat monitor for Tracker liveness:

# lib/my_app/tracker_heartbeat.ex
defmodule MyApp.TrackerHeartbeat do
  use GenServer

  @heartbeat_url System.get_env("VIGILMON_TRACKER_HEARTBEAT_URL")
  @interval :timer.minutes(1)

  def start_link(_), do: GenServer.start_link(__MODULE__, %{}, name: __MODULE__)

  def init(state) do
    schedule()
    {:ok, state}
  end

  def handle_info(:ping, state) do
    check = MyApp.TrackerHealth.check()

    if check.status == :ok do
      send_heartbeat()
    end

    schedule()
    {:noreply, state}
  end

  defp send_heartbeat do
    if @heartbeat_url do
      :httpc.request(:get, {String.to_charlist(@heartbeat_url), []}, [{:timeout, 3000}], [])
    end
  end

  defp schedule, do: Process.send_after(self(), :ping, @interval)
end

Create a Heartbeat monitor in Vigilmon with a 3-minute expected interval. A missed heartbeat means the Tracker process is down or the track/untrack cycle is failing.

Step 7: Alerting

In Vigilmon, configure Notifications → New Channel:

Slack for Tracker degradation:

🔴 DOWN: Phoenix.Tracker — MyApp
Monitor: /health returning tracker.status = "degraded"
Node: myapp@10.0.0.1
Action: Check Tracker process, PubSub connectivity, and CRDT delta sync logs

PagerDuty for heartbeat miss:

Configure a high-priority alert — if the Tracker heartbeat stops, distributed presence is failing and users may see stale online/offline states.

Common Phoenix.Tracker Issues and Fixes

Ghost presences after node crash:

# Configure a shorter tombstone expiry so crashed node entries
# are cleaned up faster (default is 60s in Phoenix 1.7+)
use Phoenix.Tracker,
  name: __MODULE__,
  pubsub_server: MyApp.PubSub,
  presence_timeout: 15_000  # ms; lower = faster cleanup after crash

High memory from topic accumulation:

# Untrack explicitly when a session ends
@impl true
def handle_info({:DOWN, _ref, :process, pid, _reason}, state) do
  Phoenix.Tracker.untrack(MyApp.PresenceTracker, pid)
  {:noreply, state}
end

Consistency drift after partition:

# From each node's IEx, compare list counts
iex(node1)> Phoenix.Tracker.list(MyApp.PresenceTracker, "user") |> length()
1543
iex(node2)> Phoenix.Tracker.list(MyApp.PresenceTracker, "user") |> length()
800  # <-- drift; wait for CRDT delta sync or force reconnect

What You've Built

| What | How | |------|-----| | Tracker process check | Verify process alive and track/untrack cycle works | | Cross-node consistency check | RPC comparison of entry counts across cluster nodes | | Structured health endpoint | JSON health response with tracker and consistency status | | Entry count monitoring | Telemetry on per-topic entry counts for leak detection | | Liveness heartbeat | GenServer pinging Vigilmon when Tracker is healthy | | Real-time alerting | Vigilmon HTTP monitor + PagerDuty on heartbeat miss |

Phoenix.Tracker makes distributed presence possible in Phoenix clusters. Vigilmon makes sure the presence data stays accurate when it matters.

Next Steps

• Add Grafana panels for entry count by topic to identify which rooms or user segments are growing
• Set up alerts on consistency drift exceeding 10% to catch partition aftereffects before users notice
• Monitor CRDT delta sizes in PubSub messages to catch sync storms after large partitions
• Use Vigilmon's downtime history to measure Tracker availability for SLA reporting

Get started free at vigilmon.online.