tutorial

How to Monitor libcluster with Vigilmon

Track libcluster node formation, topology changes, and cluster membership in your Elixir application, and use Vigilmon to alert when cluster nodes fail to connect or drop unexpectedly.

How to Monitor libcluster with Vigilmon

libcluster is the de-facto Elixir library for automatic cluster node discovery and formation. It supports multiple topology strategies — Gossip UDP, DNS polling, Kubernetes endpoints, EC2 tags, and custom strategies — and ensures that nodes joining or leaving your distributed Elixir application are automatically connected to the cluster without manual intervention.

Because libcluster operates at the networking and process group layer, its failures are silent from the outside. A node that fails to connect due to a misconfigured strategy continues running normally; only the cluster-aware features break. Distributed GenServers can't locate processes on other nodes. Horde registries diverge. Phoenix PubSub messages don't reach disconnected nodes. None of this surfaces as an HTTP error unless you explicitly instrument cluster state.

Vigilmon lets you expose libcluster formation health through a health endpoint and alert when node membership deviates from what you expect.

Why Monitor libcluster?

Cluster formation failures produce distributed application failures, not application errors:

  • Node connection failure — a misconfigured topology strategy silently fails; the node runs but is isolated from the cluster
  • Split cluster — a network partition disconnects a subset of nodes; distributed state diverges silently
  • Topology strategy crash — the libcluster strategy process itself can crash; without monitoring its supervisor tree, you won't know until a scheduled reconnect attempt fails
  • DNS TTL mismatch — in Kubernetes, a pod IP changes between DNS TTL cycles; libcluster can lose a node reference until the next poll
  • Expected node count drift — rolling deploys or autoscaling events should produce transient membership changes; prolonged drift indicates a failed pod or misconfigured strategy
  • Cookie mismatch — a node with the wrong Erlang cookie silently fails to authenticate to the cluster; libcluster marks it disconnected but reports no explicit error

Key Metrics to Track

| Metric | What it tells you | |--------|-------------------| | Connected node count | How many nodes are currently in the cluster | | Expected vs actual node count | Whether autoscaling or deploys left nodes unreachable | | Topology strategy uptime | Whether the libcluster strategy process is running | | Node connect/disconnect events | Rate of cluster membership changes | | Heartbeat from each node | Whether each node can reach Vigilmon independently | | Health endpoint response | Whether the app on each node is healthy |

Step 1: Add libcluster to Your Application

# mix.exs
defp deps do
  [
    {:libcluster, "~> 3.3"},
    # ... other deps
  ]
end

Configure your topology strategy in config/runtime.exs:

# config/runtime.exs
config :libcluster,
  topologies: [
    my_app: [
      strategy: Cluster.Strategy.Gossip,
      config: [
        secret: System.get_env("LIBCLUSTER_SECRET", "dev-secret"),
        multicast_addr: {230, 1, 1, 251},
        port: 45892,
        if_addr: {0, 0, 0, 0}
      ]
    ]
  ]

For Kubernetes:

# config/runtime.exs
config :libcluster,
  topologies: [
    my_app_k8s: [
      strategy: Cluster.Strategy.Kubernetes,
      config: [
        mode: :dns,
        kubernetes_node_basename: System.get_env("K8S_NODE_BASENAME", "myapp"),
        kubernetes_selector: System.get_env("K8S_SELECTOR", "app=myapp"),
        kubernetes_namespace: System.get_env("K8S_NAMESPACE", "default"),
        polling_interval: 10_000
      ]
    ]
  ]

Add libcluster to your supervision tree:

# lib/my_app/application.ex
def start(_type, _args) do
  topologies = Application.get_env(:libcluster, :topologies, [])

  children = [
    {Cluster.Supervisor, [topologies, [name: MyApp.ClusterSupervisor]]},
    MyApp.Repo,
    MyAppWeb.Endpoint,
    MyApp.ClusterMonitor
  ]

  opts = [strategy: :one_for_one, name: MyApp.Supervisor]
  Supervisor.start_link(children, opts)
end

Step 2: Track Cluster Membership with Telemetry

Subscribe to :net_kernel node events and emit telemetry on every membership change:

# lib/my_app/cluster_monitor.ex
defmodule MyApp.ClusterMonitor do
  use GenServer
  require Logger

  @poll_interval :timer.seconds(30)
  @expected_node_count String.to_integer(System.get_env("CLUSTER_EXPECTED_NODES", "1"))

  def start_link(_), do: GenServer.start_link(__MODULE__, %{}, name: __MODULE__)

  def init(state) do
    :net_kernel.monitor_nodes(true)
    schedule()
    {:ok, state}
  end

  # Node joined the cluster
  def handle_info({:nodeup, node}, state) do
    Logger.info("[Cluster] node UP: #{node}")

    :telemetry.execute(
      [:my_app, :cluster, :node_up],
      %{count: 1},
      %{node: node, connected: length(Node.list())}
    )

    {:noreply, state}
  end

  # Node left the cluster
  def handle_info({:nodedown, node}, state) do
    Logger.warning("[Cluster] node DOWN: #{node}")

    :telemetry.execute(
      [:my_app, :cluster, :node_down],
      %{count: 1},
      %{node: node, connected: length(Node.list())}
    )

    {:noreply, state}
  end

  # Periodic cluster membership report
  def handle_info(:poll, state) do
    connected_nodes = Node.list()
    node_count = length(connected_nodes) + 1  # +1 for self

    :telemetry.execute(
      [:my_app, :cluster, :membership],
      %{node_count: node_count, expected: @expected_node_count},
      %{nodes: connected_nodes, self: node()}
    )

    if node_count < @expected_node_count do
      Logger.warning(
        "[Cluster] only #{node_count}/#{@expected_node_count} nodes connected: #{inspect(connected_nodes)}"
      )
    end

    schedule()
    {:noreply, state}
  end

  defp schedule, do: Process.send_after(self(), :poll, @poll_interval)
end

Attach a handler to log and forward telemetry:

# lib/my_app/cluster_telemetry.ex
defmodule MyApp.ClusterTelemetry do
  require Logger

  def attach do
    events = [
      [:my_app, :cluster, :node_up],
      [:my_app, :cluster, :node_down],
      [:my_app, :cluster, :membership]
    ]

    :telemetry.attach_many("my-app-cluster", events, &handle_event/4, nil)
  end

  def handle_event([:my_app, :cluster, :node_up], %{count: _}, meta, _) do
    Logger.info("[ClusterTelemetry] node joined: #{meta.node}, total: #{meta.connected + 1}")
  end

  def handle_event([:my_app, :cluster, :node_down], %{count: _}, meta, _) do
    Logger.warning("[ClusterTelemetry] node left: #{meta.node}, remaining: #{meta.connected + 1}")
  end

  def handle_event([:my_app, :cluster, :membership], metrics, meta, _) do
    Logger.debug(
      "[ClusterTelemetry] cluster size: #{metrics.node_count}/#{metrics.expected} — #{inspect(meta.nodes)}"
    )
  end
end

Call MyApp.ClusterTelemetry.attach() in your Application.start/2.

Step 3: Build a Cluster Health Endpoint

# lib/my_app_web/controllers/health_controller.ex
defmodule MyAppWeb.HealthController do
  use MyAppWeb, :controller

  @expected_node_count String.to_integer(System.get_env("CLUSTER_EXPECTED_NODES", "1"))

  def index(conn, _params) do
    connected = Node.list()
    node_count = length(connected) + 1
    quorum = node_count > div(@expected_node_count, 2)

    cluster_healthy = node_count >= @expected_node_count

    checks = %{
      cluster: %{
        status: if(cluster_healthy, do: "ok", else: "degraded"),
        node_count: node_count,
        expected: @expected_node_count,
        connected_nodes: connected,
        quorum: quorum
      },
      libcluster: %{
        status: check_libcluster_supervisor()
      }
    }

    http_status = if cluster_healthy, do: 200, else: 503

    conn
    |> put_status(http_status)
    |> json(%{
      status: if(http_status == 200, do: "ok", else: "degraded"),
      node: node(),
      checks: checks
    })
  end

  defp check_libcluster_supervisor do
    case Process.whereis(MyApp.ClusterSupervisor) do
      pid when is_pid(pid) -> "ok"
      nil -> "error"
    end
  end
end

Register the route:

# lib/my_app_web/router.ex
scope "/health", MyAppWeb do
  get "/", HealthController, :index
end

Step 4: Heartbeat Monitor per Node

In a distributed cluster, each node should send its own heartbeat to Vigilmon. Add a heartbeat GenServer:

# lib/my_app/cluster_heartbeat.ex
defmodule MyApp.ClusterHeartbeat do
  use GenServer
  require Logger

  @interval :timer.minutes(1)

  def start_link(_), do: GenServer.start_link(__MODULE__, %{}, name: __MODULE__)

  def init(state) do
    schedule()
    {:ok, state}
  end

  def handle_info(:ping, state) do
    url = System.get_env("VIGILMON_CLUSTER_HEARTBEAT_URL")

    if url do
      case :httpc.request(
             :get,
             {String.to_charlist(url), []},
             [{:timeout, 5_000}],
             []
           ) do
        {:ok, _} ->
          Logger.debug("[ClusterHeartbeat] ping sent from #{node()}")

        {:error, reason} ->
          Logger.warning("[ClusterHeartbeat] ping failed: #{inspect(reason)}")
      end
    end

    schedule()
    {:noreply, state}
  end

  defp schedule, do: Process.send_after(self(), :ping, @interval)
end

Add to your supervision tree so every node runs its own heartbeat process.

Step 5: Set Up Monitoring in Vigilmon

HTTP Monitor (cluster health endpoint)

  1. Sign in at vigilmon.online
  2. Click New Monitor → HTTP
  3. Set URL to https://your-app.example.com/health
  4. Expected status: 200
  5. Body must contain: "status":"ok"
  6. Check interval: 60 seconds

For multi-node deployments on Fly.io or Kubernetes, add a separate HTTP monitor per node:

https://myapp.fly.dev/health           # primary region
https://lhr.myapp.fly.dev/health       # secondary region

Each monitor alerts independently, letting you pinpoint whether a cluster failure is node-specific or global.

Heartbeat Monitor (per-node liveness)

  1. Click New Monitor → Heartbeat
  2. Name: Cluster Node Liveness — [node name]
  3. Expected interval: 2 minutes
  4. Copy the URL → set as VIGILMON_CLUSTER_HEARTBEAT_URL in your environment

A missed heartbeat on any node means that node is either down or isolated from the internet — even if the other nodes are healthy.

Step 6: Alerting

In Vigilmon, configure Notifications → New Channel:

Slack for cluster degradation:

🔴 DOWN: Cluster Health — MyApp
Monitor: /health returned cluster.status = "degraded"
Check: node_count vs expected in the response body
Action: kubectl get pods / fly status to identify disconnected nodes

PagerDuty for heartbeat miss:

A missed heartbeat means a node is completely unreachable. Configure a high-priority PagerDuty integration in Vigilmon for heartbeat monitors — these warrant immediate response.

Quorum alert:

Configure a second HTTP monitor that checks the response body for "quorum":true:

Alert: body does not contain '"quorum":true'
Meaning: fewer than half of expected cluster nodes are reachable — distributed features are unavailable

Common libcluster Issues and Fixes

Gossip strategy not discovering nodes on the same host:

# Use epmd strategy for local multi-node development
config :libcluster,
  topologies: [
    dev: [
      strategy: Cluster.Strategy.Epmd,
      config: [
        hosts: [:"node1@127.0.0.1", :"node2@127.0.0.1"]
      ]
    ]
  ]

Kubernetes DNS strategy not finding pods:

# Verify the DNS headless service resolves
kubectl exec -it <pod> -- nslookup myapp.default.svc.cluster.local
# Should return one IP per pod — if not, check headless service config

Cookie mismatch preventing connection:

# In releases, set the Erlang cookie via env var
# rel/env.sh.eex
export RELEASE_COOKIE="${RELEASE_COOKIE:-dev-cookie}"

libcluster supervisor not starting:

# Verify topologies are configured before the supervisor starts
topologies = Application.get_env(:libcluster, :topologies, [])
IO.inspect(topologies, label: "libcluster topologies")
# Empty list = no strategies configured = nodes never connect

What You've Built

| What | How | |------|-----| | Node event tracking | :net_kernel.monitor_nodes/1 with telemetry on join/leave | | Periodic membership report | GenServer polling Node.list() vs expected count | | Cluster health endpoint | /health reporting node count, quorum, libcluster supervisor status | | Per-node heartbeat | GenServer pinging Vigilmon every minute from each node | | HTTP monitor | Vigilmon checks endpoint for "status":"ok" | | Heartbeat monitor | Vigilmon detects silent node failure via missed heartbeat | | Quorum alerting | Body check for "quorum":true catches split-cluster scenarios |

libcluster handles the mechanics of connecting your cluster. Vigilmon tells you when the result isn't what you expected.

Next Steps

  • Add per-strategy health checks (e.g., verify the Kubernetes DNS endpoint resolves before reporting cluster healthy)
  • Export telemetry to Grafana for node membership history and event rate dashboards
  • Use Vigilmon incident history to correlate cluster events with deploy timestamps
  • Consider adding a monitor for your Erlang Port Mapper Daemon (epmd) endpoint to catch port conflicts on cluster formation

Get started free at vigilmon.online.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles