How to Monitor Horde with Vigilmon

Horde is a distributed process registry and supervisor library for Elixir, built on δ-CRDTs (conflict-free replicated data types). It enables globally distributed process management across cluster nodes: processes registered in Horde.Registry are visible from every node, and processes supervised by Horde.DynamicSupervisor can fail over to other nodes when their host node crashes.

Because Horde uses CRDT synchronization to achieve eventual consistency, its failure modes are unlike those of local supervisors. A network partition can produce temporary inconsistency between nodes — both might believe they own a process. A slow CRDT sync can delay failover. A node that joins the cluster without being added to the Horde members list won't participate in distribution. None of these appear as HTTP errors unless you explicitly instrument them.

Vigilmon lets you expose Horde cluster membership and process distribution health through a health endpoint and alert when distributed process management degrades.

Why Monitor Horde?

Horde failures affect distributed process availability, often silently:

• Membership divergence — if nodes don't agree on the Horde member list, CRDT sync is incomplete and registry state diverges between nodes
• Process duplication after partition — both sides of a network partition may spawn a process for the same key; on heal, Horde terminates duplicates, but the termination itself can disrupt in-flight work
• Failover delay — a node crash triggers Horde to redistribute its processes to other nodes; if the redistribution takes too long, dependent services see missing processes
• Registry lookup failure — a process registered on a crashed node remains in the registry briefly; lookups during the CRDT propagation delay return stale results
• Horde supervisor overload — when many processes fail over simultaneously (mass node loss), the receiving supervisors may be overwhelmed by simultaneous starts
• Missing member after deploy — a rolling deploy temporarily reduces the Horde member list; if the deploy stalls, reduced membership persists

Key Metrics to Track

| Metric | What it tells you | |--------|-------------------| | Horde members per node | Whether all nodes agree on cluster membership | | Registry size per node | Whether registry state is consistent across nodes | | Supervisor child count | How many distributed processes are running | | Process registration latency | Time to register and look up a process | | Failover time | How long process redistribution takes after a node loss | | CRDT sync lag | Difference in registry size across nodes (approximation) | | Heartbeat from registered process | Whether a specific critical process is alive |

Step 1: Add Horde to Your Application

# mix.exs
defp deps do
  [
    {:horde, "~> 0.9"},
    {:libcluster, "~> 3.3"},  # recommended for cluster formation
    # ... other deps
  ]
end

Define your Horde registry and supervisor:

# lib/my_app/distributed_registry.ex
defmodule MyApp.DistributedRegistry do
  use Horde.Registry

  def start_link(_) do
    Horde.Registry.start_link(__MODULE__, [keys: :unique], name: __MODULE__)
  end

  def init(init_arg) do
    [members: members()]
    |> Keyword.merge(init_arg)
    |> Horde.Registry.init()
  end

  defp members do
    [Node.self() | Node.list()]
    |> Enum.map(&{__MODULE__, &1})
  end
end

# lib/my_app/distributed_supervisor.ex
defmodule MyApp.DistributedSupervisor do
  use Horde.DynamicSupervisor

  def start_link(_) do
    Horde.DynamicSupervisor.start_link(__MODULE__, [strategy: :one_for_one], name: __MODULE__)
  end

  def init(init_arg) do
    [members: members(), distribution_strategy: Horde.UniformDistribution]
    |> Keyword.merge(init_arg)
    |> Horde.DynamicSupervisor.init()
  end

  defp members do
    [Node.self() | Node.list()]
    |> Enum.map(&{__MODULE__, &1})
  end
end

Handle cluster membership changes by updating Horde members when nodes join or leave:

# lib/my_app/horde_connector.ex
defmodule MyApp.HordeConnector do
  use GenServer
  require Logger

  def start_link(_), do: GenServer.start_link(__MODULE__, %{}, name: __MODULE__)

  def init(state) do
    :net_kernel.monitor_nodes(true, node_type: :visible)
    {:ok, state}
  end

  def handle_info({:nodeup, _node, _opts}, state) do
    set_horde_members()
    {:noreply, state}
  end

  def handle_info({:nodedown, _node, _opts}, state) do
    set_horde_members()
    {:noreply, state}
  end

  defp set_horde_members do
    members = [Node.self() | Node.list()]

    registry_members = Enum.map(members, &{MyApp.DistributedRegistry, &1})
    supervisor_members = Enum.map(members, &{MyApp.DistributedSupervisor, &1})

    Horde.Cluster.set_members(MyApp.DistributedRegistry, registry_members)
    Horde.Cluster.set_members(MyApp.DistributedSupervisor, supervisor_members)

    Logger.info("[HordeConnector] updated members: #{inspect(members)}")
  end
end

Step 2: Instrument Horde Operations with Telemetry

# lib/my_app/horde_monitor.ex
defmodule MyApp.HordeMonitor do
  use GenServer
  require Logger

  @poll_interval :timer.seconds(30)

  def start_link(_), do: GenServer.start_link(__MODULE__, %{}, name: __MODULE__)

  def init(state) do
    schedule()
    {:ok, state}
  end

  def handle_info(:poll, state) do
    report_registry_health()
    report_supervisor_health()
    schedule()
    {:noreply, state}
  end

  defp report_registry_health do
    local_size = Horde.Registry.count(MyApp.DistributedRegistry)
    members = Horde.Cluster.members(MyApp.DistributedRegistry)

    :telemetry.execute(
      [:my_app, :horde, :registry],
      %{local_size: local_size, member_count: length(members)},
      %{node: node(), members: members}
    )

    Logger.debug("[HordeMonitor] registry size: #{local_size}, members: #{length(members)}")
  rescue
    e ->
      Logger.warning("[HordeMonitor] registry check failed: #{inspect(e)}")
  end

  defp report_supervisor_health do
    children = Horde.DynamicSupervisor.which_children(MyApp.DistributedSupervisor)
    child_count = length(children)
    members = Horde.Cluster.members(MyApp.DistributedSupervisor)

    :telemetry.execute(
      [:my_app, :horde, :supervisor],
      %{child_count: child_count, member_count: length(members)},
      %{node: node(), members: members}
    )

    Logger.debug("[HordeMonitor] supervisor children: #{child_count}, members: #{length(members)}")
  rescue
    e ->
      Logger.warning("[HordeMonitor] supervisor check failed: #{inspect(e)}")
  end

  defp schedule, do: Process.send_after(self(), :poll, @poll_interval)
end

Step 3: Measure Registry Round-Trip Latency

# lib/my_app/horde_probe.ex
defmodule MyApp.HordeProbe do
  require Logger

  @probe_key :horde_health_probe
  @probe_name {MyApp.DistributedRegistry, @probe_key}

  @doc "Register a probe entry and immediately look it up. Returns latency in ms or an error."
  def roundtrip do
    start = System.monotonic_time(:millisecond)

    registration_result =
      Horde.Registry.register(MyApp.DistributedRegistry, @probe_key, :probe)

    lookup_result =
      case Horde.Registry.lookup(MyApp.DistributedRegistry, @probe_key) do
        [{_pid, :probe}] -> :ok
        [] -> {:error, "probe not found after registration"}
        other -> {:error, "unexpected lookup result: #{inspect(other)}"}
      end

    latency_ms = System.monotonic_time(:millisecond) - start

    # Clean up (unregister by exiting the registered process scope)
    case registration_result do
      {:ok, _} -> Horde.Registry.unregister(MyApp.DistributedRegistry, @probe_key)
      _ -> :ok
    end

    case lookup_result do
      :ok ->
        :telemetry.execute(
          [:my_app, :horde, :probe],
          %{latency_ms: latency_ms},
          %{result: "ok", node: node()}
        )
        {:ok, latency_ms}

      {:error, reason} ->
        Logger.error("[HordeProbe] roundtrip failed: #{reason}")
        {:error, reason}
    end
  rescue
    e ->
      Logger.error("[HordeProbe] exception: #{inspect(e)}")
      {:error, inspect(e)}
  end
end

Step 4: Build a Horde Health Endpoint

# lib/my_app_web/controllers/health_controller.ex
defmodule MyAppWeb.HealthController do
  use MyAppWeb, :controller

  def index(conn, _params) do
    registry_check = check_registry()
    supervisor_check = check_supervisor()
    probe_check = check_probe()

    all_ok =
      registry_check.status == "ok" and
        supervisor_check.status == "ok" and
        probe_check.status == "ok"

    http_status = if all_ok, do: 200, else: 503

    conn
    |> put_status(http_status)
    |> json(%{
      status: if(all_ok, do: "ok", else: "degraded"),
      node: node(),
      checks: %{
        registry: registry_check,
        supervisor: supervisor_check,
        probe: probe_check
      }
    })
  end

  defp check_registry do
    members = Horde.Cluster.members(MyApp.DistributedRegistry)
    size = Horde.Registry.count(MyApp.DistributedRegistry)

    %{
      status: if(length(members) > 0, do: "ok", else: "error"),
      member_count: length(members),
      registry_size: size
    }
  rescue
    e -> %{status: "error", reason: inspect(e)}
  end

  defp check_supervisor do
    members = Horde.Cluster.members(MyApp.DistributedSupervisor)
    children = Horde.DynamicSupervisor.which_children(MyApp.DistributedSupervisor)

    %{
      status: if(length(members) > 0, do: "ok", else: "error"),
      member_count: length(members),
      child_count: length(children)
    }
  rescue
    e -> %{status: "error", reason: inspect(e)}
  end

  defp check_probe do
    case MyApp.HordeProbe.roundtrip() do
      {:ok, latency_ms} -> %{status: "ok", latency_ms: latency_ms}
      {:error, reason} -> %{status: "error", reason: reason}
    end
  end
end

Register the route:

# lib/my_app_web/router.ex
scope "/health", MyAppWeb do
  get "/", HealthController, :index
end

Step 5: Heartbeat Monitor

# lib/my_app/horde_heartbeat.ex
defmodule MyApp.HordeHeartbeat do
  use GenServer
  require Logger

  @interval :timer.minutes(1)

  def start_link(_), do: GenServer.start_link(__MODULE__, %{}, name: __MODULE__)

  def init(state) do
    schedule()
    {:ok, state}
  end

  def handle_info(:ping, state) do
    url = System.get_env("VIGILMON_HORDE_HEARTBEAT_URL")

    case MyApp.HordeProbe.roundtrip() do
      {:ok, _latency_ms} ->
        if url do
          :httpc.request(:get, {String.to_charlist(url), []}, [{:timeout, 5_000}], [])
        end

      {:error, reason} ->
        Logger.error("[HordeHeartbeat] skipping ping — registry unhealthy: #{reason}")
    end

    schedule()
    {:noreply, state}
  end

  defp schedule, do: Process.send_after(self(), :ping, @interval)
end

Step 6: Set Up Monitoring in Vigilmon

HTTP Monitor (Horde health endpoint)

1. Sign in at vigilmon.online
2. Click New Monitor → HTTP
3. Set URL to https://your-app.example.com/health
4. Expected status: 200
5. Body must contain: "status":"ok"
6. Check interval: 60 seconds

Add monitors per node for multi-region deployments:

https://myapp.fly.dev/health           # primary
https://lhr.myapp.fly.dev/health       # secondary

Heartbeat Monitor (Horde registry liveness)

1. Click New Monitor → Heartbeat
2. Name: Horde Registry Liveness — [node]
3. Expected interval: 2 minutes
4. Copy the URL → set as VIGILMON_HORDE_HEARTBEAT_URL in your environment

A missed heartbeat means the Horde registry can't complete a register/lookup round-trip — distributed process discovery is broken on this node.

Step 7: Alerting

In Vigilmon, configure Notifications → New Channel:

Slack for registry degradation:

🔴 DOWN: Horde Registry — MyApp
Monitor: /health returned registry.status = "error"
Check: member_count in response, run Horde.Cluster.members/1 in remsh
Action: verify libcluster connected all nodes before investigating Horde members

PagerDuty for heartbeat miss:

A missed Horde heartbeat means distributed process routing is broken. Configure high-priority PagerDuty for this monitor.

Common Horde Issues and Fixes

Registry entries not visible from all nodes:

# Ensure HordeConnector has updated members after cluster formation
# Check current members from any node:
Horde.Cluster.members(MyApp.DistributedRegistry)
# If a node is missing, call set_horde_members() manually or restart HordeConnector

Process duplication after partition heal:

# Horde handles this via CRDT convergence — duplicates are terminated automatically
# Monitor logs for "[Horde.DynamicSupervisor] terminating duplicate" entries
# These are expected on heal; alert only if they persist more than a few seconds

Slow CRDT sync on large clusters:

# Reduce sync interval (default is 5s in Horde)
# In Horde.Registry init:
def init(init_arg) do
  [members: members(), delta_crdt_options: [sync_interval: 2_000]]
  |> Keyword.merge(init_arg)
  |> Horde.Registry.init()
end

Horde members not updated after node join:

# Verify HordeConnector is in the supervision tree and monitoring nodes
:net_kernel.monitor_nodes(true)
# Test with: send(MyApp.HordeConnector, {:nodeup, :test@host, []})

What You've Built

| What | How | |------|-----| | Node membership tracking | HordeConnector updates members on every node change | | Registry health polling | GenServer monitoring member count and registry size | | Supervisor health polling | GenServer monitoring child count and member list | | Registry round-trip probe | Register/lookup cycle with latency measurement | | Horde health endpoint | /health aggregates all checks with structured JSON | | Liveness heartbeat | GenServer pinging Vigilmon only on successful probe | | HTTP and heartbeat monitors | Vigilmon checks endpoint + heartbeat liveness per node |

Horde gives your Elixir cluster globally consistent process distribution. Vigilmon makes sure that distribution is working when your application depends on it.

Next Steps

• Export Horde telemetry to Grafana for registry size history and sync lag dashboards
• Add per-process heartbeat monitors for critical singleton processes (e.g., a global rate-limiter or session manager)
• Use Vigilmon incident history to correlate Horde degradation with deploy events or network incidents
• Consider Horde.Registry.select/2 in your health probe for more realistic query-path testing

Get started free at vigilmon.online.