tutorial

How to Monitor EventStoreDB Uptime and Health with Vigilmon

EventStoreDB cluster leader elections, projection failures, subscription group lag, and truncated checkpoint files are hard to detect from outside the process. Learn how to monitor EventStoreDB externally with Vigilmon HTTP probes and heartbeat monitors for event-sourced aggregate rebuild jobs and projection catch-up tasks.

EventStoreDB is the purpose-built database for event sourcing — append-only streams, persistent subscriptions, server-side projections, and catch-up subscriptions are its core primitives. But "append-only" does not mean "failure-free." An EventStoreDB cluster undergoing a leader election enters a brief read-only window during which all writes return NotLeader errors; applications that do not handle leader re-election gracefully will drop events. A server-side projection stuck in a Faulted state continues to report itself as running until you inspect the projection status API. A persistent subscription consumer group that falls behind accumulates unacknowledged events silently — the stream grows, memory pressure builds, and the application starts delivering stale read-model state without raising an alert. And EventStoreDB's embedded HTTP interface, used for stats and gossip, can become unresponsive on nodes with exhausted file descriptor limits long before the gRPC API stops accepting connections.

Vigilmon gives you external visibility into EventStoreDB health through HTTP probe monitoring and heartbeat monitoring for aggregate rebuild jobs and projection catch-up tasks. This tutorial walks through both.

Why EventStoreDB Monitoring Needs More Than Node Stats

EventStoreDB exposes a /health/live endpoint and a gossip API at /gossip that report node state. They cannot tell you:

  • Whether write operations are actually succeeding from your application servers, or whether leader election is in progress and writes are being silently rejected
  • Whether a server-side projection is Faulted — the projection API marks itself Running during brief fault windows and only transitions to Faulted after a checkpoint write failure
  • Whether a persistent subscription consumer group has accumulated excessive unacknowledged events and is delivering stale events to your read-model projectors
  • Whether catch-up subscriptions established by your service have dropped their connection and are operating in a disconnected state
  • Whether a scheduled aggregate rebuild job — which replays a full event stream to reconstruct a read model — has stalled or failed partway through
  • Whether the truncation/scavenging process has corrupted or dropped a chunk file that your subscriptions depend on

These failure modes cause silent event loss, stale read models, and write outages that your application logs capture but EventStoreDB's own dashboard does not surface as actionable alerts. External monitoring through Vigilmon catches them by probing the actual write and subscription paths your application depends on.

Step 1: Build an EventStoreDB Health Endpoint

EventStoreDB exposes a built-in HTTP API and a gRPC API. For health monitoring, the most reliable approach is to deploy a thin sidecar service that connects to EventStoreDB using the official client, performs a probe write and read, and exposes the result via HTTP.

Node.js / Express Example

// health/eventstoredb.js
const { EventStoreDBClient, jsonEvent, FORWARDS, START } = require('@eventstore/db-client');
const express = require('express');
const { randomUUID } = require('crypto');

const app = express();

const client = EventStoreDBClient.connectionString(
  process.env.ESDB_CONNECTION_STRING
  // e.g. esdb://user:pass@localhost:2113?tls=true
  //   or esdb+discover://user:pass@cluster-node1:2113,node2:2113,node3:2113?tls=true
);

const HEALTH_STREAM = process.env.ESDB_HEALTH_STREAM || '$vigilmon-health';

app.get('/health/eventstoredb', async (req, res) => {
  const correlationId = randomUUID();
  try {
    // Append a sentinel event — verifies write path and leader availability
    const writeStart = Date.now();
    await client.appendToStream(HEALTH_STREAM, [
      jsonEvent({ type: 'HealthCheck', data: { correlationId, probedAt: new Date().toISOString() } }),
    ]);
    const writeMs = Date.now() - writeStart;

    // Read the last event back — verifies read path
    const readStart = Date.now();
    const events = [];
    for await (const resolvedEvent of client.readStream(HEALTH_STREAM, {
      direction: FORWARDS,
      fromRevision: START,
      maxCount: 1,
      resolveLinkTos: false,
    })) {
      events.push(resolvedEvent);
    }
    const readMs = Date.now() - readStart;

    return res.status(200).json({ status: 'ok', write_ms: writeMs, read_ms: readMs });
  } catch (err) {
    const code = err.constructor?.name || 'UnknownError';
    // NotLeaderError means a leader election is in progress
    // StreamDeletedException means the health stream was deleted
    return res.status(503).json({ status: 'down', error: err.message, error_type: code });
  }
});

// Optional: check projection status
app.get('/health/eventstoredb/projection', async (req, res) => {
  const projectionName = req.query.name || process.env.ESDB_PROJECTION_NAME;
  if (!projectionName) {
    return res.status(400).json({ error: 'Missing projection name' });
  }
  try {
    const status = await client.getProjectionStatus(projectionName);
    const isFaulted = status.projectionStatus === 'Faulted';
    const isStopped = status.projectionStatus === 'Stopped';
    if (isFaulted || isStopped) {
      return res.status(503).json({
        status: 'degraded',
        projection: projectionName,
        projection_status: status.projectionStatus,
        reason: status.reason,
      });
    }
    return res.status(200).json({
      status: 'ok',
      projection: projectionName,
      projection_status: status.projectionStatus,
    });
  } catch (err) {
    return res.status(503).json({ status: 'down', error: err.message });
  }
});

const PORT = process.env.PORT || 8080;
app.listen(PORT, () => console.log(`EventStoreDB health service on :${PORT}`));

Python / FastAPI Example

# health_eventstoredb.py
import os
import time
import uuid
from fastapi import FastAPI
from fastapi.responses import JSONResponse
from esdbclient import EventStoreDBClient, NewEvent, StreamState
from esdbclient.exceptions import NotFound, GrpcError

app = FastAPI()

ESDB_URI = os.environ["ESDB_CONNECTION_STRING"]
# e.g. "esdb://user:pass@localhost:2113?tls=false"
HEALTH_STREAM = os.environ.get("ESDB_HEALTH_STREAM", "$vigilmon-health")

client = EventStoreDBClient(uri=ESDB_URI)

@app.get("/health/eventstoredb")
async def eventstoredb_health():
    correlation_id = str(uuid.uuid4())
    try:
        # Append sentinel event
        write_start = time.monotonic()
        client.append_to_stream(
            HEALTH_STREAM,
            current_version=StreamState.ANY,
            events=[
                NewEvent(
                    type="HealthCheck",
                    data={"correlationId": correlation_id, "probedAt": time.time()},
                )
            ],
        )
        write_ms = int((time.monotonic() - write_start) * 1000)

        # Read last event
        read_start = time.monotonic()
        events = list(client.get_stream(HEALTH_STREAM, limit=1))
        read_ms = int((time.monotonic() - read_start) * 1000)

        return {"status": "ok", "write_ms": write_ms, "read_ms": read_ms}

    except GrpcError as e:
        return JSONResponse(
            status_code=503,
            content={"status": "down", "error": str(e), "error_type": type(e).__name__},
        )

Verify the endpoint before wiring up Vigilmon:

curl -i http://your-health-service.internal/health/eventstoredb
# HTTP/1.1 200 OK
# {"status":"ok","write_ms":12,"read_ms":8}

Step 2: Configure Vigilmon HTTP Monitor for EventStoreDB

  1. Log in to vigilmon.online and go to Monitors → New Monitor
  2. Choose HTTP / HTTPS
  3. Set the URL to your health endpoint: http://your-health-service.internal/health/eventstoredb
  4. Set the check interval to 1 minute
  5. Under Expected response, configure:
    • Status code: 200
    • Response body contains: "status":"ok"
    • Response time threshold: 5000ms
  6. Under Alert channels, assign your Slack or PagerDuty channel
  7. Save the monitor

Vigilmon probes from multiple geographic regions simultaneously, requiring multi-region consensus before opening an incident. This eliminates false alarms from transient single-probe blips and distinguishes genuine cluster leader elections from local connectivity issues.

Monitoring Projection Health Separately

Server-side projections that power your read models should be monitored independently from the core write path:

  • [esdb-core] /health/eventstoredb — P1 page on failure; all event writes are blocked
  • [esdb-projection-orders] /health/eventstoredb/projection?name=orders-read-model — P2 Slack alert; read model is stale

Create one Vigilmon monitor per critical projection. A faulted projection that goes undetected delivers stale query results to your application with no error surfaced to end users.

Step 3: Heartbeat Monitoring for Aggregate Rebuild Jobs and Catch-Up Tasks

Event-sourced applications periodically need to replay entire event streams to rebuild a read model from scratch — after a projection schema migration, a code bug that corrupted a projection checkpoint, or a data reprocessing requirement. These rebuild jobs can run for hours and fail at any point without the orchestrator knowing. Separately, catch-up subscriptions established during service startup can silently disconnect and restart from a stale checkpoint.

Vigilmon heartbeat monitors detect silent stalls: your rebuild job pings Vigilmon on successful completion, and your catch-up subscription service pings on each successful batch of events processed. If pings stop arriving within the expected window, Vigilmon fires an alert.

Set Up the Heartbeat Monitor

  1. In Vigilmon, go to Monitors → New Monitor → Heartbeat
  2. Set the name: esdb-aggregate-rebuild (or esdb-catchup-subscription-orders)
  3. Set the expected interval to match your job schedule (e.g. 6 hours for a periodic rebuild; 5 minutes for a catch-up subscription processor)
  4. Set the grace period: 30 minutes for rebuild jobs; 10 minutes for subscriptions
  5. Save — copy the unique heartbeat URL, e.g. https://vigilmon.online/heartbeat/abc123xyz

Node.js Heartbeat Example for Aggregate Rebuild Jobs

// rebuild-read-model.js — run as a scheduled job or one-off task
const { EventStoreDBClient, FORWARDS, START } = require('@eventstore/db-client');
const axios = require('axios');

const client = EventStoreDBClient.connectionString(process.env.ESDB_CONNECTION_STRING);
const HEARTBEAT_URL = process.env.VIGILMON_HEARTBEAT_URL;

async function rebuildOrdersReadModel() {
  const readModelStore = require('./read-model-store'); // your DB adapter
  const STREAM_NAME = 'orders';

  console.log('Starting orders read-model rebuild...');
  let eventCount = 0;

  // Clear existing read model before replay
  await readModelStore.truncate('orders');

  for await (const resolvedEvent of client.readStream(STREAM_NAME, {
    direction: FORWARDS,
    fromRevision: START,
    resolveLinkTos: true,
  })) {
    const { event } = resolvedEvent;
    if (!event) continue;

    // Apply each event to the read model
    await readModelStore.applyEvent('orders', event.type, event.data);
    eventCount++;

    if (eventCount % 1000 === 0) {
      console.log(`Replayed ${eventCount} events...`);
    }
  }

  console.log(`Rebuild complete — replayed ${eventCount} events`);

  // Ping Vigilmon — rebuild completed successfully
  await axios.get(HEARTBEAT_URL, { timeout: 5000 });
  console.log('Heartbeat sent to Vigilmon');
}

rebuildOrdersReadModel().catch(err => {
  console.error('Rebuild failed:', err);
  // No heartbeat ping — Vigilmon will alert after grace period
  process.exit(1);
});

Node.js Heartbeat Example for Persistent Subscription Consumers

// subscription-consumer.js — long-running service
const { EventStoreDBClient, persistentSubscriptionToStreamSettingsFromDefaults } = require('@eventstore/db-client');
const axios = require('axios');

const client = EventStoreDBClient.connectionString(process.env.ESDB_CONNECTION_STRING);
const HEARTBEAT_URL = process.env.VIGILMON_HEARTBEAT_URL;
const STREAM = 'orders';
const GROUP = 'orders-read-model-projector';

let lastHeartbeatAt = Date.now();
const HEARTBEAT_INTERVAL_MS = 4 * 60 * 1000; // 4 minutes (Vigilmon grace: 10 min)

async function runSubscription() {
  const subscription = client.subscribeToPersistentSubscriptionToStream(STREAM, GROUP);

  for await (const resolvedEvent of subscription) {
    const { event } = resolvedEvent;
    if (!event) continue;

    try {
      // Process event (apply to read model)
      await processEvent(event);
      await subscription.ack(resolvedEvent);

      // Ping Vigilmon periodically to prove the subscription is alive and making progress
      if (Date.now() - lastHeartbeatAt >= HEARTBEAT_INTERVAL_MS) {
        await axios.get(HEARTBEAT_URL, { timeout: 5000 });
        lastHeartbeatAt = Date.now();
      }
    } catch (err) {
      console.error(`Failed to process event ${event.id}:`, err);
      await subscription.nack('park', err.message, resolvedEvent);
    }
  }
}

async function processEvent(event) {
  // your read model application logic
}

runSubscription().catch(err => {
  console.error('Subscription died:', err);
  // No heartbeat — Vigilmon alerts after grace period
  process.exit(1);
});

Step 4: Alert Routing and Response Time Thresholds

Configure alert routing in Vigilmon to match the severity of each EventStoreDB failure mode:

| Monitor | Alert Channel | Priority | |---|---|---| | EventStoreDB core /health/eventstoredb | Slack + PagerDuty | P1 | | Projection status /health/eventstoredb/projection?name=* | Slack | P2 | | Heartbeat: aggregate rebuild job | Slack + email | P2 | | Heartbeat: persistent subscription consumer | Slack | P2 |

Set response time thresholds as early warnings before full availability failures:

  • Alert at 1000ms for the health endpoint — an EventStoreDB append + read inside the same cluster should complete in under 50ms; latency above 1 second indicates leader election in progress, I/O saturation, or a scavenging run impacting read performance
  • Alert at 5000ms as a second threshold — signals the cluster is operating in a degraded state
  • Set a separate P1 threshold at 10000ms to page on-call before Vigilmon marks the monitor fully down — gives the team time to investigate whether a leader re-election is completing or a node has crashed

Summary

EventStoreDB failures surface in subtle ways — silent leader elections dropping writes, faulted projections delivering stale read models, consumer group lag, and failed rebuild jobs — long before your application surfaces them as user-visible errors. Vigilmon gives you external visibility across the full failure surface:

| Monitor Type | What It Covers | |---|---| | HTTP monitor on /health/eventstoredb | Write path availability, leader election detection, gRPC connectivity | | HTTP monitor per projection | Server-side projection faults and stale checkpoint detection | | Heartbeat monitor: aggregate rebuild | Silent mid-stream rebuild failures and checkpoint corruption | | Heartbeat monitor: subscription consumer | Disconnected catch-up subscriptions and consumer group stalls |

Get started free at vigilmon.online — your first EventStoreDB monitor is running in under two minutes.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles