How to Monitor Swarm (Elixir) with Vigilmon

Swarm is a distributed process registry and process grouping library for Elixir clusters. It provides automatic process distribution across cluster nodes, handles process handoff when nodes join or leave, and offers global process registration so any node can look up and communicate with any registered process regardless of where it runs.

Unlike Horde's CRDT approach, Swarm uses a consistent hashing ring and a leader-coordinated protocol for process distribution. When a node joins the cluster, Swarm calculates which processes should migrate to it and performs handoff. When a node leaves, its processes are redistributed to the remaining nodes. This makes Swarm's failure modes distinct: handoff stalls, ring inconsistencies, and leader election failures each produce different symptoms.

Vigilmon lets you expose Swarm process distribution health through a health endpoint and alert when the registry or handoff mechanism degrades.

Why Monitor Swarm?

Swarm failures affect process availability across the cluster:

• Handoff timeout — when a node joins, Swarm migrates processes to it; if handoff takes too long or times out, processes may be duplicated or lost during the transition
• Ring inconsistency — if nodes have different views of cluster membership, the consistent hash ring produces different routing decisions; processes are started on the wrong node or looked up incorrectly
• Leader election failure — Swarm uses a leader to coordinate registration; if the leader crashes during an election, registration requests queue until a new leader is elected
• Registration collision — two nodes register the same name simultaneously; Swarm resolves this, but the resolution involves terminating one of the processes
• Process not found after handoff — a process that was being handed off may be briefly unavailable; callers see {:error, :not_registered} during the window
• Swarm tracker crash — the Swarm.Tracker process coordinates global state; its crash triggers a restart with full ring recalculation, which is expensive in large clusters

Key Metrics to Track

| Metric | What it tells you | |--------|-------------------| | Registered process count | Total globally registered processes in the cluster | | Handoff event rate | How frequently processes migrate between nodes | | Registration latency | Time to register a process and retrieve it from another node | | Node ring membership | Whether the consistent hash ring includes all cluster nodes | | Swarm tracker uptime | Whether the local Swarm tracker process is alive | | Process lookup latency | Time to resolve a registered name to a PID | | Heartbeat from registered process | Whether a specific critical process is globally reachable |

Step 1: Add Swarm to Your Application

# mix.exs
defp deps do
  [
    {:swarm, "~> 3.4"},
    {:libcluster, "~> 3.3"},  # recommended for cluster formation
    # ... other deps
  ]
end

Swarm starts automatically as part of its application. Configure it:

# config/runtime.exs
config :swarm,
  node_blacklist: [],
  debug: false

Register processes globally:

# lib/my_app/session_worker.ex
defmodule MyApp.SessionWorker do
  use GenServer

  def start_link({user_id, opts}) do
    name = {:via, :swarm, {__MODULE__, user_id}}
    GenServer.start_link(__MODULE__, %{user_id: user_id}, name: name)
  end

  def init(state), do: {:ok, state}

  # Swarm calls this before handing off to another node
  def handle_call({:swarm, :begin_handoff}, _from, state) do
    {:reply, {:resume, state}, state}
  end

  # Swarm calls this with the state from the previous node
  def handle_cast({:swarm, :end_handoff, state}, _old_state) do
    {:noreply, state}
  end

  # Swarm calls this on node disconnect
  def handle_info({:swarm, :die}, state) do
    {:stop, :normal, state}
  end
end

Start processes through Swarm:

# Start a globally distributed process
{:ok, pid} = Swarm.register_name({MyApp.SessionWorker, user_id}, MyApp.SessionWorker, :start_link, [{user_id, []}])

# Look up a process from any node
case Swarm.whereis_name({MyApp.SessionWorker, user_id}) do
  :undefined -> {:error, :not_found}
  pid -> {:ok, pid}
end

Step 2: Monitor Swarm Registry Health

# lib/my_app/swarm_monitor.ex
defmodule MyApp.SwarmMonitor do
  use GenServer
  require Logger

  @poll_interval :timer.seconds(30)

  def start_link(_), do: GenServer.start_link(__MODULE__, %{}, name: __MODULE__)

  def init(state) do
    schedule()
    {:ok, state}
  end

  def handle_info(:poll, state) do
    report_registry_health()
    report_tracker_health()
    schedule()
    {:noreply, state}
  end

  defp report_registry_health do
    # Get all registered names and their pids
    registered = Swarm.registered()
    count = length(registered)

    # Check how many registered processes are local vs remote
    local_count = Enum.count(registered, fn {_name, pid} -> node(pid) == node() end)
    remote_count = count - local_count

    :telemetry.execute(
      [:my_app, :swarm, :registry],
      %{total_count: count, local_count: local_count, remote_count: remote_count},
      %{node: node()}
    )

    Logger.debug("[SwarmMonitor] registered: #{count} total (#{local_count} local, #{remote_count} remote)")
  rescue
    e -> Logger.warning("[SwarmMonitor] registry check failed: #{inspect(e)}")
  end

  defp report_tracker_health do
    tracker_alive =
      case Process.whereis(:swarm_tracker) do
        pid when is_pid(pid) -> Process.alive?(pid)
        nil -> false
      end

    :telemetry.execute(
      [:my_app, :swarm, :tracker],
      %{alive: if(tracker_alive, do: 1, else: 0)},
      %{node: node()}
    )

    unless tracker_alive do
      Logger.error("[SwarmMonitor] Swarm tracker is NOT alive on #{node()}")
    end
  end

  defp schedule, do: Process.send_after(self(), :poll, @poll_interval)
end

Step 3: Measure Registration Round-Trip Latency

# lib/my_app/swarm_probe.ex
defmodule MyApp.SwarmProbe do
  require Logger

  @probe_name {MyApp.SwarmProbe, :health_probe}

  defmodule ProbeWorker do
    use GenServer

    def start_link(args), do: GenServer.start_link(__MODULE__, args)
    def init(state), do: {:ok, state}

    def handle_call({:swarm, :begin_handoff}, _from, state), do: {:reply, :ignore, state}
    def handle_info({:swarm, :die}, state), do: {:stop, :normal, state}
  end

  @doc "Register a probe process and verify it can be looked up. Returns latency_ms or error."
  def roundtrip do
    start = System.monotonic_time(:millisecond)

    result =
      with {:ok, pid} <-
             Swarm.register_name(
               @probe_name,
               GenServer,
               :start_link,
               [ProbeWorker, %{}, []]
             ),
           found_pid when is_pid(found_pid) <- Swarm.whereis_name(@probe_name),
           true <- Process.alive?(found_pid) do
        latency_ms = System.monotonic_time(:millisecond) - start
        {:ok, latency_ms}
      else
        :undefined -> {:error, "probe process not found after registration"}
        false -> {:error, "probe process registered but not alive"}
        {:error, {:already_registered, _}} -> {:ok, 0}  # probe from previous check still alive
        error -> {:error, inspect(error)}
      end

    # Clean up
    Swarm.unregister_name(@probe_name)

    result
  rescue
    e ->
      Logger.warning("[SwarmProbe] exception: #{inspect(e)}")
      {:error, inspect(e)}
  end
end

Step 4: Build a Swarm Health Endpoint

# lib/my_app_web/controllers/health_controller.ex
defmodule MyAppWeb.HealthController do
  use MyAppWeb, :controller

  def index(conn, _params) do
    tracker_check = check_tracker()
    registry_check = check_registry()
    probe_check = check_probe()

    all_ok =
      tracker_check.status == "ok" and
        registry_check.status == "ok" and
        probe_check.status == "ok"

    http_status = if all_ok, do: 200, else: 503

    conn
    |> put_status(http_status)
    |> json(%{
      status: if(all_ok, do: "ok", else: "degraded"),
      node: node(),
      checks: %{
        swarm_tracker: tracker_check,
        registry: registry_check,
        probe: probe_check
      }
    })
  end

  defp check_tracker do
    case Process.whereis(:swarm_tracker) do
      pid when is_pid(pid) ->
        %{status: if(Process.alive?(pid), do: "ok", else: "error")}

      nil ->
        %{status: "error", reason: "swarm_tracker process not found"}
    end
  end

  defp check_registry do
    registered = Swarm.registered()
    count = length(registered)
    %{status: "ok", registered_count: count}
  rescue
    e -> %{status: "error", reason: inspect(e)}
  end

  defp check_probe do
    case MyApp.SwarmProbe.roundtrip() do
      {:ok, latency_ms} -> %{status: "ok", latency_ms: latency_ms}
      {:error, reason} -> %{status: "error", reason: reason}
    end
  end
end

Register the route:

# lib/my_app_web/router.ex
scope "/health", MyAppWeb do
  get "/", HealthController, :index
end

Step 5: Track Handoff Events

Wrap your Swarm process starts to capture handoff telemetry:

# lib/my_app/swarm_registry.ex
defmodule MyApp.SwarmRegistry do
  require Logger

  def start_worker(name, module, args) do
    start = System.monotonic_time(:millisecond)

    result = Swarm.register_name(name, module, :start_link, [args])

    duration_ms = System.monotonic_time(:millisecond) - start

    outcome = case result do
      {:ok, _pid} -> "ok"
      {:error, {:already_registered, _}} -> "already_registered"
      {:error, reason} -> "error:#{inspect(reason)}"
    end

    :telemetry.execute(
      [:my_app, :swarm, :register],
      %{duration_ms: duration_ms},
      %{name: name, outcome: outcome, node: node()}
    )

    if duration_ms > 1_000 do
      Logger.warning("[SwarmRegistry] slow registration for #{inspect(name)}: #{duration_ms}ms")
    end

    result
  end
end

Step 6: Set Up Monitoring in Vigilmon

HTTP Monitor (Swarm health endpoint)

1. Sign in at vigilmon.online
2. Click New Monitor → HTTP
3. Set URL to https://your-app.example.com/health
4. Expected status: 200
5. Body must contain: "status":"ok"
6. Check interval: 60 seconds

Heartbeat Monitor (Swarm registry liveness)

# lib/my_app/swarm_heartbeat.ex
defmodule MyApp.SwarmHeartbeat do
  use GenServer
  require Logger

  @interval :timer.minutes(1)

  def start_link(_), do: GenServer.start_link(__MODULE__, %{}, name: __MODULE__)

  def init(state) do
    schedule()
    {:ok, state}
  end

  def handle_info(:ping, state) do
    url = System.get_env("VIGILMON_SWARM_HEARTBEAT_URL")

    case MyApp.SwarmProbe.roundtrip() do
      {:ok, _latency_ms} ->
        if url do
          :httpc.request(:get, {String.to_charlist(url), []}, [{:timeout, 5_000}], [])
          Logger.debug("[SwarmHeartbeat] ping sent from #{node()}")
        end

      {:error, reason} ->
        Logger.error("[SwarmHeartbeat] skipping ping — probe failed: #{reason}")
    end

    schedule()
    {:noreply, state}
  end

  defp schedule, do: Process.send_after(self(), :ping, @interval)
end

In Vigilmon:

1. Click New Monitor → Heartbeat
2. Name: Swarm Registry Liveness — [node]
3. Expected interval: 2 minutes
4. Copy the URL → set as VIGILMON_SWARM_HEARTBEAT_URL

Step 7: Alerting

In Vigilmon, configure Notifications → New Channel:

Slack for Swarm health failure:

🔴 DOWN: Swarm Registry — MyApp
Monitor: /health returned swarm_tracker.status = "error" or probe.status = "error"
Check: Process.whereis(:swarm_tracker) in remsh on the affected node
Action: if tracker is dead, check OTP supervisor tree — Swarm should restart it automatically

PagerDuty for heartbeat miss:

A missed Swarm heartbeat means the registry probe failed — processes can't be registered or looked up globally. Configure high-priority PagerDuty for this monitor.

Registration latency alert:

Use the HTTP monitor with a response time threshold:

Alert: response time > 2000ms
Meaning: Swarm probe registration is slow — possible leader election or ring rebalancing in progress

Common Swarm Issues and Fixes

Slow registration during node join:

# Swarm rebalances the ring during node joins — registration calls may queue
# Increase the registration timeout in callers:
Swarm.register_name(name, module, :start_link, [args], timeout: 10_000)

Process not found immediately after registration:

# Swarm's global registry may have propagation delay on large clusters
# Add a brief retry with backoff in lookup code:
defp find_process(name, retries \\ 3) do
  case Swarm.whereis_name(name) do
    :undefined when retries > 0 ->
      Process.sleep(50)
      find_process(name, retries - 1)
    pid -> pid
  end
end

Swarm tracker restart loop:

# Check logs for repeated "swarm_tracker" restarts
grep "swarm_tracker" /var/log/myapp.log | tail -20
# If restarting frequently, check for OOM or large cluster ring recalculation overhead

Handoff state loss:

# Ensure your GenServer implements begin_handoff and end_handoff correctly
# {:reply, {:resume, state}, state} — continues process on new node with original state
# {:reply, :ignore, state} — drops the process on the new node (it won't restart)
# {:reply, :restart, state} — restarts the process fresh on the new node

What You've Built

| What | How | |------|-----| | Swarm tracker monitoring | Check Process.whereis(:swarm_tracker) on each poll | | Registry count polling | Swarm.registered/0 with local vs remote breakdown | | Registration latency probe | Register/lookup round-trip with latency measurement | | Registration telemetry | Telemetry on every Swarm.register_name/4 call | | Swarm health endpoint | /health aggregates tracker, registry, and probe checks | | Liveness heartbeat | GenServer pinging Vigilmon only on successful probe | | HTTP and heartbeat monitors | Vigilmon checks endpoint + heartbeat liveness per node |

Swarm handles automatic process distribution and handoff across your cluster. Vigilmon makes sure the distribution is working when your application depends on globally registered processes.

Next Steps

• Export Swarm telemetry to Grafana for registration latency history and handoff event rate dashboards
• Add per-process heartbeat monitors for critical singletons (e.g., a global rate-limiter or leader process)
• Use Vigilmon incident history to correlate Swarm degradation with rolling deploy events or cluster size changes
• Consider monitoring handoff duration by timing the gap between begin_handoff and end_handoff callbacks

Get started free at vigilmon.online.