How to Monitor Partisan with Vigilmon

Partisan is a high-performance, pluggable distributed communication library for Erlang and Elixir. It replaces the default Erlang distribution (dist) with a purpose-built overlay network, giving you control over peer-to-peer topology strategies — full mesh, HyParView, client-server, and more. Partisan removes the single-channel bottleneck of Erlang distribution by supporting multiple channels per peer connection and separating the overlay connection layer from message delivery.

When Partisan's overlay breaks down, your distributed application loses the ability to route messages across nodes. Unlike a simple process crash, Partisan failures can manifest as silent message loss, asymmetric connectivity, or topology divergence — where node A believes it is connected to node B, but node B does not reciprocate. Standard HTTP uptime monitors won't catch any of this.

Vigilmon heartbeat monitors let you verify that Partisan's overlay is actually delivering messages across your cluster, not just that the process is running.

Why Monitor Partisan?

Partisan failures are subtle and often invisible to application-level health checks:

• Topology divergence — the local node believes it has a full-mesh overlay, but some peers have drifted out of the membership view due to a missed gossip round
• Channel saturation — one of Partisan's named channels fills its send buffer; messages queued on that channel stall while other channels continue flowing
• Membership oscillation — nodes rapidly join and leave the overlay due to misconfigured failure detection timers, causing instability without a clean partition
• Silent message loss — Partisan's partisan_default_channel drops messages when the send buffer is full and send_after_connect is disabled
• Overlay not starting — if the Partisan application is not in your supervision tree or starts before the network interface is ready, the overlay starts with no peers
• HyParView active view undersize — in large clusters using the HyParView topology, the active view can shrink below the target size after churn, reducing overlay coverage

Key Metrics to Track

| Metric | What it tells you | |--------|-------------------| | Peer connection count | Whether the overlay has expected number of connected peers | | Membership view size | Active view size vs. expected cluster size | | Channel queue depth | Backpressure on each named channel | | Message delivery latency | Round-trip time for test messages across the overlay | | Join/leave rate | Membership churn indicating instability | | Channel error count | Failed sends per channel |

Step 1: Add Partisan to Your Application

# mix.exs
defp deps do
  [
    {:partisan, "~> 5.0"}
  ]
end

# config/config.exs
config :partisan,
  peer_service_manager: :partisan_full_membership_strategy,
  parallelism: 4,
  channels: [
    %{channel: :default, parallelism: 4},
    %{channel: :data, parallelism: 2},
    %{channel: :control, parallelism: 1}
  ],
  # Failure detection
  failure_detector: true,
  gossip_interval: 10_000,
  # Bind port
  peer_port: 10100

# lib/my_app/application.ex
children = [
  {Partisan, []},
  # rest of children
]

Step 2: Build a Peer Connectivity Health Check

# lib/my_app/partisan_health.ex
defmodule MyApp.PartisanHealth do
  @moduledoc """
  Checks Partisan overlay health: membership view size and peer reachability.
  """

  @expected_cluster_size Application.compile_env(:my_app, :expected_cluster_size, 1)

  def check do
    members = :partisan_peer_service.members()

    case members do
      {:ok, member_set} ->
        member_list = MapSet.to_list(member_set)
        count = length(member_list)

        %{
          status: if(count >= @expected_cluster_size - 1, do: :ok, else: :degraded),
          peer_count: count,
          expected: @expected_cluster_size - 1,
          peers: Enum.map(member_list, &extract_node_name/1)
        }

      {:error, reason} ->
        %{status: :error, reason: inspect(reason)}
    end
  rescue
    e -> %{status: :error, reason: inspect(e)}
  end

  defp extract_node_name(%{name: name}), do: name
  defp extract_node_name(node) when is_atom(node), do: node
  defp extract_node_name(other), do: inspect(other)
end

Step 3: Build a Message Delivery Probe

Peer count alone doesn't tell you messages are flowing. Build an end-to-end delivery test using Partisan's message passing:

# lib/my_app/partisan_probe.ex
defmodule MyApp.PartisanProbe do
  use GenServer
  require Logger

  @probe_topic :partisan_health_probe
  @timeout 3_000

  def start_link(_), do: GenServer.start_link(__MODULE__, %{}, name: __MODULE__)

  def probe_local do
    probe_id = :crypto.strong_rand_bytes(8) |> Base.encode16()
    GenServer.call(__MODULE__, {:probe, probe_id}, @timeout + 500)
  end

  def probe_remote(target_node) do
    probe_id = :crypto.strong_rand_bytes(8) |> Base.encode16()
    start = System.monotonic_time(:millisecond)

    :partisan.cast_message(target_node, MyApp.PartisanProbe, {:echo_probe, probe_id, node()})

    receive do
      {:probe_echo, ^probe_id} ->
        latency = System.monotonic_time(:millisecond) - start
        {:ok, latency}
    after
      @timeout -> {:error, :timeout}
    end
  end

  @impl GenServer
  def init(state) do
    :partisan_peer_service.add_sup_callback(fn msg ->
      send(__MODULE__, {:partisan_msg, msg})
    end)

    {:ok, state}
  end

  @impl GenServer
  def handle_call({:probe, probe_id}, from, state) do
    {:noreply, Map.put(state, probe_id, {from, System.monotonic_time(:millisecond)})}
  end

  @impl GenServer
  def handle_info({:partisan_msg, {:probe, probe_id, from_node}}, state) do
    :partisan.cast_message(from_node, MyApp.PartisanProbe, {:probe_ack, probe_id})
    {:noreply, state}
  end

  def handle_info({:partisan_msg, {:probe_ack, probe_id}}, state) do
    case Map.pop(state, probe_id) do
      {{from, start_ms}, new_state} ->
        latency = System.monotonic_time(:millisecond) - start_ms
        GenServer.reply(from, {:ok, latency})
        {:noreply, new_state}

      {nil, state} ->
        {:noreply, state}
    end
  end

  def handle_info({:echo_probe, probe_id, reply_to}, state) do
    :partisan.cast_message(reply_to, MyApp.PartisanProbe, {:probe_echo, probe_id})
    {:noreply, state}
  end

  def handle_info(_, state), do: {:noreply, state}
end

Step 4: Add a Health Endpoint

# lib/my_app_web/controllers/health_controller.ex
defmodule MyAppWeb.HealthController do
  use MyAppWeb, :controller

  def index(conn, _params) do
    partisan_check = MyApp.PartisanHealth.check()
    db_ok = db_ok?()

    status =
      if partisan_check.status == :ok and db_ok, do: 200, else: 503

    conn
    |> put_status(status)
    |> json(%{
      status: if(status == 200, do: "ok", else: "degraded"),
      node: Node.self() |> to_string(),
      partisan: partisan_check,
      database: if(db_ok, do: "ok", else: "error")
    })
  end

  defp db_ok? do
    case Ecto.Adapters.SQL.query(MyApp.Repo, "SELECT 1", []) do
      {:ok, _} -> true
      _ -> false
    end
  rescue
    _ -> false
  end
end

# lib/my_app_web/router.ex
scope "/", MyAppWeb do
  get "/health", HealthController, :index
end

Step 5: Instrument Partisan with Telemetry

# lib/my_app/partisan_poller.ex
defmodule MyApp.PartisanPoller do
  use GenServer
  require Logger

  @poll_interval :timer.seconds(30)

  def start_link(_), do: GenServer.start_link(__MODULE__, %{}, name: __MODULE__)

  def init(state) do
    schedule()
    {:ok, state}
  end

  def handle_info(:poll, state) do
    report_metrics()
    schedule()
    {:noreply, state}
  end

  defp report_metrics do
    case :partisan_peer_service.members() do
      {:ok, member_set} ->
        count = MapSet.size(member_set)

        :telemetry.execute(
          [:my_app, :partisan, :membership],
          %{peer_count: count},
          %{node: node()}
        )

      {:error, reason} ->
        Logger.warning("Partisan membership check failed: #{inspect(reason)}")

        :telemetry.execute(
          [:my_app, :partisan, :membership],
          %{peer_count: 0},
          %{node: node(), error: inspect(reason)}
        )
    end

    report_channel_stats()
  end

  defp report_channel_stats do
    try do
      channels = [:default, :data, :control]

      Enum.each(channels, fn channel ->
        queue_size = get_channel_queue_size(channel)

        :telemetry.execute(
          [:my_app, :partisan, :channel],
          %{queue_size: queue_size},
          %{channel: channel, node: node()}
        )
      end)
    rescue
      _ -> :ok
    end
  end

  defp get_channel_queue_size(channel) do
    case :partisan_util.process_identifier(:partisan_default_channel) do
      {:ok, pid} ->
        {:message_queue_len, len} = Process.info(pid, :message_queue_len)
        len

      _ ->
        0
    end
  rescue
    _ -> 0
  end

  defp schedule, do: Process.send_after(self(), :poll, @poll_interval)
end

Define Telemetry.Metrics:

# lib/my_app/telemetry.ex
[
  last_value("my_app.partisan.membership.peer_count",
    tags: [:node]
  ),
  last_value("my_app.partisan.channel.queue_size",
    tags: [:channel, :node]
  )
]

Step 6: Create Monitors in Vigilmon

HTTP monitor for the health endpoint:

1. Sign in at vigilmon.online
2. Click New Monitor → HTTP
3. Set URL to https://your-app.example.com/health
4. Set interval to 60 seconds
5. Alert condition: status 200 and body contains "partisan":{"status":"ok"}

Heartbeat monitor for overlay liveness:

# lib/my_app/partisan_heartbeat.ex
defmodule MyApp.PartisanHeartbeat do
  use GenServer

  @heartbeat_url System.get_env("VIGILMON_PARTISAN_HEARTBEAT_URL")
  @interval :timer.minutes(1)

  def start_link(_), do: GenServer.start_link(__MODULE__, %{}, name: __MODULE__)

  def init(state) do
    schedule()
    {:ok, state}
  end

  def handle_info(:ping, state) do
    check = MyApp.PartisanHealth.check()

    if check.status == :ok do
      send_heartbeat()
    end

    schedule()
    {:noreply, state}
  end

  defp send_heartbeat do
    if @heartbeat_url do
      :httpc.request(:get, {String.to_charlist(@heartbeat_url), []}, [{:timeout, 3000}], [])
    end
  end

  defp schedule, do: Process.send_after(self(), :ping, @interval)
end

Create a Heartbeat monitor in Vigilmon with a 3-minute expected interval. A missed heartbeat means Partisan's overlay has too few peers or message routing has failed.

Step 7: Alerting

In Vigilmon, configure Notifications → New Channel:

Slack for peer count degradation:

🔴 DOWN: Partisan Overlay — MyApp
Monitor: /health returning partisan.status = "degraded"
Node: myapp@10.0.0.1
Action: Check Partisan peer port (10100), network routes between nodes, and gossip interval config

PagerDuty for heartbeat miss:

Configure a high-priority alert — if the heartbeat stops, nodes have lost overlay connectivity and distributed message routing is broken.

Common Partisan Issues and Fixes

No peers after startup:

# Ensure Partisan starts after the network is ready
# and seeds peers explicitly if using static topology
config :partisan,
  peer_service_manager: :partisan_full_membership_strategy,
  seeds: [
    %{name: :"myapp@10.0.0.2", listen_addrs: [%{ip: {10, 0, 0, 2}, port: 10100}]},
    %{name: :"myapp@10.0.0.3", listen_addrs: [%{ip: {10, 0, 0, 3}, port: 10100}]}
  ]

Channel send buffer full:

# Reduce channel parallelism to avoid overwhelming receivers,
# or increase buffer sizes for high-throughput channels
config :partisan,
  channels: [
    %{channel: :data, parallelism: 4, buffer_size: 1024}
  ]

Topology divergence after node restart:

# Verify membership from each node's IEx console
iex> :partisan_peer_service.members()
{:ok, #MapSet<[...]>}

# If views diverge, trigger a manual sync
iex> :partisan_peer_service.sync_join(peer_node_spec)

What You've Built

| What | How | |------|-----| | Membership view check | Peer count vs. expected cluster size | | Message delivery probe | GenServer round-trip probe via Partisan channels | | Structured health endpoint | JSON health response with Partisan overlay status | | Channel queue monitoring | Telemetry on per-channel backpressure | | Liveness heartbeat | GenServer pinging Vigilmon when overlay is healthy | | Real-time alerting | Vigilmon HTTP monitor + PagerDuty on heartbeat miss |

Partisan gives your Elixir cluster flexible, high-performance distributed communication. Vigilmon makes sure the overlay stays connected when it matters.

Next Steps

• Add Grafana panels for channel queue depth to catch backpressure before it causes message loss
• Set up separate heartbeats per named channel to isolate which channel is failing
• Monitor topology-specific metrics (HyParView active/passive view sizes) for large clusters
• Use Vigilmon's downtime history to measure overlay availability for SLA reporting

Get started free at vigilmon.online.