tutorial

How to Monitor Syn (Elixir) with Vigilmon

Monitor Syn's global process registry health, cluster membership synchronization, and cross-node process discovery in your Erlang/Elixir cluster using Vigilmon.

How to Monitor Syn (Elixir) with Vigilmon

Syn is a global process registry and process group manager for Erlang/Elixir clusters. It maintains a distributed, eventually-consistent registry that maps names to PIDs across all nodes in your cluster. You can register a process on node A and look it up by name from node B without any coordination. Syn also supports process groups — named collections of processes that can be messaged broadcast-style cluster-wide.

When Syn's registry loses consistency, process discovery breaks silently. A call to Syn.lookup/2 returns :undefined for a process that is running but registered on a partitioned node. Group messages silently miss members on disconnected nodes. These failures don't raise exceptions unless you're explicitly handling :undefined returns — most code assumes successful lookup.

Vigilmon heartbeat monitors let you verify that Syn's registry is operational and synchronized, not just that the local Syn process is alive.

Why Monitor Syn?

Syn failures manifest as silent incorrect behavior rather than crashes:

  • Registry divergence after partition — each partition maintains its own registry; when healed, Syn merges by last-write-wins; processes that registered on a partitioned node may be silently overwritten or duplicated
  • Process registered but unreachable — a process dies without calling Syn.unregister; its entry persists in the registry until the node's heartbeat timeout expires; lookups return a stale PID that is no longer alive
  • Scope synchronization lag — Syn uses scopes to namespace registries; after a new node joins, it must synchronize the scope state from peers; during this sync window, the new node has an incomplete view of registered processes
  • Meta mismatch — Syn.register/4 accepts metadata that is replicated cluster-wide; if a re-registration races with the original on two nodes, the winning metadata may not match what either caller expected
  • Group broadcast partial failure — Syn.publish/3 sends messages to all group members cluster-wide; if one node is unreachable, members on that node silently miss the message
  • Registry memory growth — stale entries from crashed processes accumulate if the node monitoring is misconfigured or if Syn.unregister_on_process_exit is not used

Key Metrics to Track

| Metric | What it tells you | |--------|-------------------| | Registry entry count by scope | Total registered processes; unexpected spikes indicate stale entries | | Syn process memory | Growing memory indicates stale entry accumulation | | Cross-node registry count agreement | Whether all nodes see the same number of registered entries | | Lookup success rate | Proportion of lookups returning a live PID vs. :undefined | | Group member count per group | Whether group membership is consistent across nodes | | Node sync status | Whether new nodes have completed scope synchronization |

Step 1: Add Syn to Your Application

# mix.exs
defp deps do
  [
    {:syn, "~> 3.3"}
  ]
end

# lib/my_app/application.ex
children = [
  :syn,
  # ... rest of children
]

# config/config.exs
config :syn,
  scopes: [:registry, :groups]

Register processes with metadata:

# Register a named process in the :registry scope
:ok = Syn.register(:registry, {:user, user_id}, self(), %{
  node: Node.self(),
  started_at: System.system_time(:second)
})

# Join a group
:ok = Syn.join(:groups, :background_workers, self(), %{node: Node.self()})

Step 2: Build a Syn Health Check

# lib/my_app/syn_health.ex
defmodule MyApp.SynHealth do
  @probe_name {:__health_probe__, Node.self()}
  @probe_scope :registry

  def check do
    syn_pid = Process.whereis(:syn)

    if is_nil(syn_pid) do
      %{status: :error, reason: "Syn process not running"}
    else
      memory = Process.info(syn_pid, :memory) |> elem(1)
      entry_count = Syn.registry_count(@probe_scope)

      register_result = probe_register()

      %{
        status: if(register_result == :ok, do: :ok, else: :error),
        syn_pid: inspect(syn_pid),
        memory_bytes: memory,
        registry_entry_count: entry_count,
        register_probe: register_result,
        node: Node.self() |> to_string()
      }
    end
  rescue
    e -> %{status: :error, reason: inspect(e)}
  end

  defp probe_register do
    Syn.unregister(@probe_scope, @probe_name)

    result = Syn.register(@probe_scope, @probe_name, self(), %{probe: true})

    case result do
      :ok ->
        case Syn.lookup(@probe_scope, @probe_name) do
          {pid, _meta} when pid == self() ->
            Syn.unregister(@probe_scope, @probe_name)
            :ok

          _ ->
            :lookup_mismatch
        end

      {:error, reason} ->
        {:registration_failed, reason}
    end
  rescue
    e -> {:error, inspect(e)}
  end
end

Step 3: Check Cross-Node Registry Consistency

# lib/my_app/syn_consistency.ex
defmodule MyApp.SynConsistency do
  @scope :registry
  @tolerance 0.05

  def check do
    nodes = [Node.self() | Node.list()]

    if length(nodes) < 2 do
      {:ok, :single_node}
    else
      local_count = Syn.registry_count(@scope)

      remote_counts =
        Enum.map(Node.list(), fn node ->
          case :rpc.call(node, Syn, :registry_count, [@scope], 3_000) do
            {:badrpc, _} -> nil
            count -> count
          end
        end)
        |> Enum.reject(&is_nil/1)

      all_counts = [local_count | remote_counts]
      max_count = Enum.max(all_counts)
      min_count = Enum.min(all_counts)

      drift =
        if max_count > 0,
          do: (max_count - min_count) / max_count,
          else: 0.0

      if drift <= @tolerance do
        {:ok, %{drift: Float.round(drift, 4), counts: all_counts, nodes: length(nodes)}}
      else
        {:degraded, %{drift: Float.round(drift, 4), counts: all_counts, max: max_count, min: min_count}}
      end
    end
  rescue
    e -> {:error, %{reason: inspect(e)}}
  end
end

Step 4: Expose a Health Endpoint

# lib/my_app_web/controllers/health_controller.ex
defmodule MyAppWeb.HealthController do
  use MyAppWeb, :controller

  def index(conn, _params) do
    syn = MyApp.SynHealth.check()
    consistency = MyApp.SynConsistency.check()

    syn_ok = syn.status == :ok
    consistency_ok = elem(consistency, 0) == :ok

    status = if syn_ok and consistency_ok, do: 200, else: 503

    conn
    |> put_status(status)
    |> json(%{
      status: if(status == 200, do: "ok", else: "degraded"),
      node: Node.self() |> to_string(),
      syn: syn,
      consistency: format_consistency(consistency)
    })
  end

  defp format_consistency({:ok, :single_node}), do: %{status: "ok", mode: "single_node"}
  defp format_consistency({:ok, data}), do: Map.merge(%{status: "ok"}, data)
  defp format_consistency({:degraded, data}), do: Map.merge(%{status: "degraded"}, data)
  defp format_consistency({:error, data}), do: Map.merge(%{status: "error"}, data)
end

# lib/my_app_web/router.ex
scope "/", MyAppWeb do
  get "/health", HealthController, :index
end

Step 5: Emit Telemetry Metrics

# lib/my_app/syn_poller.ex
defmodule MyApp.SynPoller do
  use GenServer
  require Logger

  @interval :timer.seconds(30)
  @scopes [:registry, :groups]
  @groups_to_monitor [:background_workers, :api_handlers]

  def start_link(_), do: GenServer.start_link(__MODULE__, %{}, name: __MODULE__)

  def init(state) do
    schedule()
    {:ok, state}
  end

  def handle_info(:poll, state) do
    report()
    schedule()
    {:noreply, state}
  end

  defp report do
    syn_pid = Process.whereis(:syn)

    if syn_pid do
      memory = Process.info(syn_pid, :memory) |> elem(1)

      :telemetry.execute(
        [:my_app, :syn, :process],
        %{memory_bytes: memory},
        %{node: Node.self()}
      )

      Enum.each(@scopes, fn scope ->
        count = Syn.registry_count(scope)

        :telemetry.execute(
          [:my_app, :syn, :registry],
          %{count: count},
          %{scope: scope}
        )
      end)

      Enum.each(@groups_to_monitor, fn group ->
        members = Syn.members(:groups, group) |> length()

        :telemetry.execute(
          [:my_app, :syn, :group],
          %{member_count: members},
          %{group: group}
        )
      end)
    else
      Logger.warning("SynPoller: Syn process not running")
    end
  rescue
    e -> Logger.warning("SynPoller error: #{inspect(e)}")
  end

  defp schedule, do: Process.send_after(self(), :poll, @interval)
end

Step 6: Create Monitors in Vigilmon

HTTP monitor for the health endpoint:

  1. Sign in at vigilmon.online
  2. Click New Monitor → HTTP
  3. URL: https://your-app.example.com/health
  4. Interval: 60 seconds
  5. Alert condition: non-200 or body contains "syn":{"status":"error"}

Heartbeat monitor for Syn registry liveness:

# lib/my_app/syn_heartbeat.ex
defmodule MyApp.SynHeartbeat do
  use GenServer

  @interval :timer.minutes(1)

  def start_link(_), do: GenServer.start_link(__MODULE__, %{}, name: __MODULE__)

  def init(state) do
    schedule()
    {:ok, state}
  end

  def handle_info(:ping, state) do
    check = MyApp.SynHealth.check()

    if check.status == :ok do
      url = System.get_env("VIGILMON_SYN_HEARTBEAT_URL")

      if url do
        :httpc.request(:get, {String.to_charlist(url), []}, [{:timeout, 5_000}], [])
      end
    end

    schedule()
    {:noreply, state}
  end

  defp schedule, do: Process.send_after(self(), :ping, @interval)
end

In Vigilmon, create a Heartbeat monitor with a 3-minute expected interval. Set VIGILMON_SYN_HEARTBEAT_URL from the Vigilmon dashboard.

Step 7: Alerting

In Vigilmon, go to Notifications → New Channel:

Slack for registry degradation:

🔴 DOWN: Syn Registry — MyApp
Monitor: /health returning syn.status = "error"
Node: myapp@10.0.0.1
Action: Check Syn process, cluster connectivity, and scope sync status

PagerDuty for heartbeat miss — a missed heartbeat means Syn is not processing register/lookup cycles, which breaks all named process discovery in the cluster.

Common Syn Issues and Fixes

Stale entries from crashed processes:

# Always register with process monitoring so Syn unregisters on exit
# Syn 3.x does this automatically when you register with the calling PID.
# For external PIDs, monitor manually:
def register_and_monitor(name, pid, meta) do
  :ok = Syn.register(:registry, name, pid, meta)
  Process.monitor(pid)
end

def handle_info({:DOWN, _ref, :process, pid, _reason}, state) do
  # Syn unregisters automatically for the process's own registrations.
  # For registrations you made on behalf of others, clean up explicitly:
  Syn.unregister(:registry, {:user, state.user_id_for_pid[pid]})
  {:noreply, state}
end

Registry count divergence after partition:

# From each node's IEx
iex(node_a@host)> Syn.registry_count(:registry)
1542

iex(node_b@host)> Syn.registry_count(:registry)
843  # <-- diverged; wait for reconnect and Syn merge

# After the partition heals, Syn merges using last-write-wins.
# Monitor the count on both nodes until they converge.

Lookup returns :undefined for a running process:

# Verify the process is registered from the registering node
iex(registering_node)> Syn.lookup(:registry, {:user, 123})
# If registered, check from the querying node:
iex(querying_node)> Syn.lookup(:registry, {:user, 123})
# :undefined means either:
# 1. Syn hasn't synced the entry yet (wait a few seconds after node join)
# 2. The registering node is partitioned from the querying node

What You've Built

| What | How | |------|-----| | Syn process check | Verify Syn process alive and register/lookup cycle works | | Cross-node count check | RPC comparison of registry entry counts across nodes | | Structured health endpoint | JSON health response with Syn and consistency status | | Registry and group metrics | Telemetry on entry counts and group sizes every 30s | | Liveness heartbeat | GenServer pinging Vigilmon when Syn is healthy | | Real-time alerting | Vigilmon HTTP monitor + PagerDuty on heartbeat miss |

Syn makes cluster-wide process discovery possible without coordination. Vigilmon makes sure discovery is actually working across all your nodes.

Next Steps

  • Add per-scope monitors if your application uses multiple Syn scopes for different process types
  • Alert on registry count dropping unexpectedly to catch mass process crashes before they cascade
  • Monitor group member counts per node to detect partition-induced membership drift
  • Use Vigilmon's response time history to track lookup latency growth as your registry scales

Get started free at vigilmon.online.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles