tutorial

How to Monitor Mnesia with Vigilmon

Track Mnesia table availability, transaction throughput, replication lag, and cluster node membership in your Erlang/Elixir application, and use Vigilmon to alert when the distributed database degrades.

How to Monitor Mnesia with Vigilmon

Mnesia is a distributed real-time database built directly into the Erlang/OTP runtime. Unlike external databases, Mnesia runs inside your BEAM application process, supports both in-memory and disk-based storage, provides ACID transactions, and replicates data across cluster nodes without a separate database server. It is widely used in high-availability Erlang systems — including Ejabberd, RabbitMQ, and OpenAMF — and is accessible from Elixir applications that need low-latency, co-located data storage with automatic replication.

Because Mnesia runs inside your OTP application, its failures don't produce external connection errors that traditional uptime monitors catch. A split-brain condition that partitions your cluster into inconsistent halves continues to serve reads from each partition. A table that becomes unavailable on some nodes silently rejects writes only to those nodes. A schema change that fails on one node creates an inconsistency that persists until manually resolved. None of these appear as HTTP errors unless you explicitly instrument and expose them.

Vigilmon lets you expose Mnesia cluster health through a health endpoint and alert when table availability or replication state degrades.

Why Monitor Mnesia?

Mnesia failures are silent from the outside and complex from the inside:

  • Table unavailability — a ram_copies or disc_copies table becomes unavailable on a node if the node that holds the master copy is unreachable; writes to that node fail with {:aborted, {:no_exists, TableName}}
  • Split-brain partition — a network partition causes nodes to diverge; each continues to accept writes; on heal, Mnesia must reconcile diverged data, which it cannot do automatically for most conflict types
  • Dirty operations bypassing consistency — mnesia:dirty_read and mnesia:dirty_write skip transaction overhead but also skip lock checks; mixed use of dirty and transactional operations in the same table can produce inconsistent views
  • Transaction queue buildup — under high write contention, Mnesia serializes conflicting transactions; a slow transaction holds locks and causes others to queue, degrading throughput for all writers
  • Schema divergence on cluster join — when a node rejoins after a crash, its Mnesia schema may differ from the running cluster; failing to reconcile produces inconsistent table definitions across nodes
  • Disk log corruption — Mnesia's disc_copies mode writes to disk log files; a disk error that corrupts these files causes table loading failures on restart

Key Metrics to Track

| Metric | What it tells you | |--------|-------------------| | Table availability per node | Whether each table is accessible on each cluster node | | Transaction commit rate | Write throughput through the Mnesia transaction manager | | Transaction abort rate | Lock contention, dirty/transactional conflicts, or schema issues | | Connected cluster nodes | Whether all expected nodes are members of the Mnesia cluster | | Replication lag | Difference in table sizes across nodes (approximation) | | Table record count | Unexpected drops or growth in specific tables | | Heartbeat freshness | Whether Mnesia can complete a read/write cycle on this node |

Step 1: Verify Mnesia is Started and Configured

Mnesia must be started before your application connects to it. In Elixir, add it to your application dependencies:

# mix.exs
def application do
  [
    extra_applications: [:logger, :mnesia],
    mod: {MyApp.Application, []}
  ]
end

Create the schema and tables once, during initial setup (not on every start):

# lib/my_app/mnesia_setup.ex
defmodule MyApp.MnesiaSetup do
  require Logger

  def init(nodes \\ [node()]) do
    case :mnesia.create_schema(nodes) do
      :ok ->
        Logger.info("[Mnesia] Schema created on #{inspect(nodes)}")

      {:error, {_, {:already_exists, _}}} ->
        Logger.debug("[Mnesia] Schema already exists")

      {:error, reason} ->
        Logger.error("[Mnesia] Schema creation failed: #{inspect(reason)}")
        {:error, reason}
    end

    :mnesia.start()

    create_tables(nodes)
  end

  defp create_tables(nodes) do
    tables = [
      {:sessions,
       attributes: [:id, :user_id, :data, :inserted_at],
       disc_copies: nodes,
       type: :set},
      {:rate_limits,
       attributes: [:key, :count, :window_start],
       ram_copies: nodes,
       type: :set}
    ]

    for {name, opts} <- tables do
      case :mnesia.create_table(name, opts) do
        {:atomic, :ok} ->
          Logger.info("[Mnesia] Table #{name} created")

        {:aborted, {:already_exists, ^name}} ->
          Logger.debug("[Mnesia] Table #{name} already exists")

        {:aborted, reason} ->
          Logger.error("[Mnesia] Table #{name} creation failed: #{inspect(reason)}")
      end
    end

    :mnesia.wait_for_tables([:sessions, :rate_limits], 10_000)
  end
end

Step 2: Wrap Transactions with Telemetry

Instrument all Mnesia operations through a wrapper module:

# lib/my_app/mnesia_store.ex
defmodule MyApp.MnesiaStore do
  require Logger

  @doc """
  Executes an Mnesia transaction and emits telemetry.
  Returns `{:ok, result}` or `{:error, reason}`.
  """
  def transaction(operation, label) do
    start = System.monotonic_time(:millisecond)

    result =
      case :mnesia.transaction(operation) do
        {:atomic, value} ->
          {:ok, value}

        {:aborted, reason} ->
          Logger.warning("[Mnesia] transaction #{label} aborted: #{inspect(reason)}")
          {:error, reason}
      end

    duration_ms = System.monotonic_time(:millisecond) - start
    outcome = if match?({:ok, _}, result), do: "ok", else: "aborted"

    :telemetry.execute(
      [:my_app, :mnesia, :transaction],
      %{duration_ms: duration_ms},
      %{label: label, outcome: outcome}
    )

    result
  end

  @doc "Read a record by key — wraps :mnesia.read in a transaction."
  def read(table, key) do
    transaction(fn -> :mnesia.read(table, key) end, "read:#{table}")
  end

  @doc "Write a record — wraps :mnesia.write in a transaction."
  def write(table, record) do
    transaction(fn -> :mnesia.write(table, record, :write) end, "write:#{table}")
  end

  @doc "Delete a record by key."
  def delete(table, key) do
    transaction(fn -> :mnesia.delete(table, key, :write) end, "delete:#{table}")
  end
end

Attach a telemetry handler:

# lib/my_app/application.ex
def start(_type, _args) do
  :telemetry.attach(
    "my-app-mnesia-handler",
    [:my_app, :mnesia, :transaction],
    &MyApp.MnesiaTelemetry.handle_event/4,
    nil
  )

  # ... rest of start/2
end

# lib/my_app/mnesia_telemetry.ex
defmodule MyApp.MnesiaTelemetry do
  require Logger

  def handle_event([:my_app, :mnesia, :transaction], %{duration_ms: ms}, meta, _) do
    case meta.outcome do
      "ok" -> Logger.debug("[Mnesia] #{meta.label} ok in #{ms}ms")
      "aborted" -> Logger.warning("[Mnesia] #{meta.label} ABORTED after #{ms}ms")
    end
  end
end

Step 3: Poll Mnesia Cluster Health

Build a poller that checks table availability and cluster membership on each node:

# lib/my_app/mnesia_poller.ex
defmodule MyApp.MnesiaPoller do
  use GenServer
  require Logger

  @poll_interval :timer.seconds(30)
  @monitored_tables [:sessions, :rate_limits]
  @expected_nodes [node()]

  def start_link(_), do: GenServer.start_link(__MODULE__, %{}, name: __MODULE__)

  def init(state) do
    schedule()
    {:ok, state}
  end

  def handle_info(:poll, state) do
    report_cluster_health()
    report_table_health()
    schedule()
    {:noreply, state}
  end

  defp report_cluster_health do
    running_nodes = :mnesia.system_info(:running_db_nodes)
    all_nodes = :mnesia.system_info(:db_nodes)
    missing_nodes = @expected_nodes -- running_nodes

    :telemetry.execute(
      [:my_app, :mnesia, :cluster],
      %{running_count: length(running_nodes), all_count: length(all_nodes)},
      %{running_nodes: running_nodes, missing_nodes: missing_nodes}
    )

    if missing_nodes != [] do
      Logger.warning("[Mnesia] nodes not in cluster: #{inspect(missing_nodes)}")
    end
  end

  defp report_table_health do
    for table <- @monitored_tables do
      info = :mnesia.table_info(table, :all)
      active_replicas = Keyword.get(info, :active_replicas, [])
      record_count = :mnesia.table_info(table, :size)

      :telemetry.execute(
        [:my_app, :mnesia, :table],
        %{active_replicas: length(active_replicas), record_count: record_count},
        %{table: table}
      )

      if active_replicas == [] do
        Logger.error("[Mnesia] table #{table} has NO active replicas — writes will fail")
      end
    end
  rescue
    e -> Logger.warning("[MnesiaPoller] error: #{inspect(e)}")
  end

  defp schedule, do: Process.send_after(self(), :poll, @poll_interval)
end

Step 4: Build a Mnesia Health Endpoint

# lib/my_app_web/controllers/health_controller.ex
defmodule MyAppWeb.HealthController do
  use MyAppWeb, :controller

  @test_table :rate_limits
  @test_key :health_probe

  def index(conn, _params) do
    mnesia_check = check_mnesia_roundtrip()
    cluster_check = check_cluster()

    checks = %{
      mnesia: format_check(mnesia_check),
      cluster: cluster_check
    }

    status = if match?({:ok, _}, mnesia_check), do: 200, else: 503

    conn
    |> put_status(status)
    |> json(%{
      status: if(status == 200, do: "ok", else: "degraded"),
      checks: checks
    })
  end

  defp check_mnesia_roundtrip do
    start = System.monotonic_time(:millisecond)

    record = {@test_table, @test_key, 0, :os.system_time(:second)}

    result =
      case :mnesia.transaction(fn ->
             :mnesia.write(@test_table, record, :write)
             :mnesia.read(@test_table, @test_key)
           end) do
        {:atomic, [_]} ->
          {:ok, System.monotonic_time(:millisecond) - start}

        {:atomic, []} ->
          {:error, "wrote but read returned empty"}

        {:aborted, reason} ->
          {:error, inspect(reason)}
      end

    result
  rescue
    e -> {:error, inspect(e)}
  end

  defp check_cluster do
    running = :mnesia.system_info(:running_db_nodes)
    all = :mnesia.system_info(:db_nodes)
    %{running_nodes: running, configured_nodes: all, quorum: length(running) > div(length(all), 2)}
  end

  defp format_check({:ok, latency_ms}), do: %{status: "ok", latency_ms: latency_ms}
  defp format_check({:error, reason}), do: %{status: "error", reason: reason}
end

Register the route:

# lib/my_app_web/router.ex
scope "/health", MyAppWeb do
  get "/", HealthController, :index
end

Step 5: Liveness Heartbeat

# lib/my_app/mnesia_heartbeat.ex
defmodule MyApp.MnesiaHeartbeat do
  use GenServer
  require Logger

  @heartbeat_url System.get_env("VIGILMON_MNESIA_HEARTBEAT_URL")
  @interval :timer.minutes(1)
  @test_table :rate_limits
  @test_key :heartbeat_probe

  def start_link(_), do: GenServer.start_link(__MODULE__, %{}, name: __MODULE__)

  def init(state) do
    schedule()
    {:ok, state}
  end

  def handle_info(:check, state) do
    record = {@test_table, @test_key, 0, :os.system_time(:second)}

    case :mnesia.transaction(fn ->
           :mnesia.write(@test_table, record, :write)
           :mnesia.read(@test_table, @test_key)
         end) do
      {:atomic, [_]} ->
        ping_vigilmon()

      {:aborted, reason} ->
        Logger.error("[MnesiaHeartbeat] health check failed: #{inspect(reason)}")
    end

    schedule()
    {:noreply, state}
  end

  defp ping_vigilmon do
    if @heartbeat_url do
      :httpc.request(
        :get,
        {String.to_charlist(@heartbeat_url), []},
        [{:timeout, 5_000}],
        []
      )
    end
  end

  defp schedule, do: Process.send_after(self(), :check, @interval)
end

Add to your supervision tree:

# lib/my_app/application.ex
children = [
  MyApp.Repo,
  MyAppWeb.Endpoint,
  MyApp.MnesiaPoller,
  MyApp.MnesiaHeartbeat
]

Step 6: Set Up Monitoring in Vigilmon

HTTP Monitor (Mnesia health endpoint)

  1. Sign in at vigilmon.online
  2. Click New Monitor → HTTP
  3. Set URL to https://your-app.example.com/health
  4. Expected status: 200
  5. Body must contain: "mnesia":{"status":"ok"}
  6. Check interval: 60 seconds

Heartbeat Monitor (Mnesia liveness)

  1. Click New Monitor → Heartbeat
  2. Name: Mnesia DB Liveness
  3. Expected interval: 2 minutes
  4. Copy the URL → set as VIGILMON_MNESIA_HEARTBEAT_URL in your environment

A missed heartbeat means Mnesia cannot complete a transactional write/read cycle on this node — any feature that writes to Mnesia is broken.

Step 7: Alerting

In Vigilmon, configure Notifications → New Channel:

Slack for Mnesia health failure:

🔴 DOWN: Mnesia — MyApp
Monitor: /health returned mnesia.status = "error"
Action: Check :mnesia.system_info(:running_db_nodes), table active_replicas, and disc log integrity

PagerDuty for heartbeat miss:

A missed heartbeat on Mnesia liveness in a distributed cluster may indicate a split-brain or node loss. Configure a high-priority PagerDuty integration in Vigilmon's notification settings.

Cluster quorum alert via HTTP monitor:

Configure a second Vigilmon HTTP monitor that checks the /health body for "quorum":true:

Alert: body does not contain '"quorum":true'
Meaning: fewer than half of configured nodes are running — the cluster may be partitioned

Common Mnesia Issues and Fixes

Table unavailable after node restart:

# Wait for tables before starting application children
# Increase timeout for disc_copies tables with large datasets
:mnesia.wait_for_tables([:sessions, :rate_limits], 30_000)

Split-brain partition recovery:

# After a network partition heals, check for inconsistent nodes
inconsistent = :mnesia.system_info(:inconsistent_database)
if inconsistent != [] do
  Logger.error("[Mnesia] inconsistent nodes detected: #{inspect(inconsistent)}")
  # Manual intervention required: decide authoritative node and force merge
  # :mnesia.set_master_nodes(table, [authoritative_node])
end

Transaction abort from lock timeout:

# Reduce transaction scope to minimize lock contention
# Instead of a single large transaction, use smaller focused ones
:mnesia.transaction(fn ->
  # Only lock what you need, not the entire table
  :mnesia.read(:sessions, specific_key)
end)

Schema inconsistency on cluster join:

%% In Erlang shell on the joining node — force schema sync from running cluster
mnesia:change_config(extra_db_nodes, [RunningNode]).
mnesia:change_table_copy_type(schema, node(), disc_copies).

Disk log corruption preventing table load:

# Backup and rebuild from another node's copy
# In production: always replicate disc_copies to at least 2 nodes
# for recovery without data loss

What You've Built

| What | How | |------|-----| | Transaction telemetry | Telemetry counter with outcome and label tags on every operation | | Cluster health polling | Running vs configured node count via mnesia.system_info | | Table health polling | Active replicas and record count per monitored table | | Mnesia health endpoint | /health runs a transactional write/read round-trip | | Liveness heartbeat | GenServer pinging Vigilmon only on successful transaction | | HTTP and heartbeat monitors | Vigilmon checks endpoint + heartbeat liveness | | Quorum monitoring | HTTP body check for "quorum":true in the health response |

Mnesia gives you a battle-hardened distributed database inside your OTP application. Vigilmon makes sure it stays consistent and available when your cluster depends on it.

Next Steps

  • Export Mnesia telemetry to Grafana for per-table transaction throughput and abort rate dashboards
  • Add per-table heartbeat monitors for your highest-stakes tables (e.g., sessions, rate_limits)
  • Set up Vigilmon incident history to correlate Mnesia degradation events with node additions or network events
  • Consider mnesia:subscribe/1 for schema change events as an internal alerting mechanism alongside Vigilmon

Get started free at vigilmon.online.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles