How to Monitor Apache BookKeeper Uptime and Health with Vigilmon

Apache BookKeeper is a distributed, fault-tolerant write-ahead logging service used as the storage backbone for Apache Pulsar, Twitter's DistributedLog, and high-throughput financial messaging systems. BookKeeper's ensemble-based write quorum model provides strong durability guarantees — but when enough bookies (storage nodes) become unavailable, the ensemble can no longer form write quorums and your dependent systems silently stop accepting writes, with no obvious error to callers.

Vigilmon gives you external visibility into BookKeeper availability through HTTP probe monitoring and heartbeat monitoring for BookKeeper-dependent producers. This tutorial walks through both.

Why Apache BookKeeper Monitoring Matters

BookKeeper's architecture introduces failure modes that process-level monitoring cannot detect:

• Quorum write failures occur when an ensemble loses enough bookies that the required write quorum (Qw) cannot be formed — the cluster appears to be running but all new writes are rejected
• Ledger recovery storms happen after multiple bookie failures, where the remaining bookies spend resources fencing and recovering ledgers, causing elevated latency for live writes
• ZooKeeper connectivity loss disconnects bookies from the metadata store — they continue serving reads but cannot participate in new ledger creates
• Disk full on a bookie causes the node to enter read-only mode silently — writes reroute to other bookies, which may already be at capacity
• The BookKeeper HTTP admin server (if enabled) can be reachable while the bookie service itself is rejecting write requests on port 3181

External monitoring through Vigilmon verifies whether your BookKeeper cluster is actually accepting writes — not just whether the JVM process is alive.

Step 1: Build a BookKeeper Health Endpoint

BookKeeper exposes an optional HTTP admin server that can be enabled in bk_server.conf:

# Enable the HTTP admin endpoint (BookKeeper 4.7+)
httpServerEnabled=true
httpServerPort=8080

Once enabled, test the native health endpoint:

curl -i http://your-bookie-host:8080/api/v1/bookie/state
# HTTP/1.1 200 OK
# {"running":true,"readOnly":false,"shuttingDown":false,"availableForHighPriorityWrites":true}

For a thorough write-path check, build a sidecar that opens a real ledger:

Node.js Sidecar Health Endpoint

// bookkeeper-health.js — tests the full write path
const express = require('express');
const { BookKeeper } = require('bookkeeper-client'); // hypothetical JS client

const app = express();
const BK_CONNECT = process.env.BK_ZOOKEEPER_CONNECT || 'localhost:2181';

app.get('/health/bookkeeper', async (req, res) => {
  let client;
  let ledger;
  try {
    // Connect to BookKeeper via ZooKeeper
    client = await BookKeeper.create({ zkServers: BK_CONNECT, timeout: 10000 });

    // Create a probe ledger (ensemble=3, writeQuorum=2, ackQuorum=2)
    ledger = await client.createLedger({
      ensembleSize: parseInt(process.env.BK_ENSEMBLE || '3'),
      writeQuorumSize: parseInt(process.env.BK_WRITE_QUORUM || '2'),
      ackQuorumSize: parseInt(process.env.BK_ACK_QUORUM || '2'),
      digestType: 'CRC32',
      password: Buffer.from('health'),
    });

    // Write a probe entry
    await ledger.addEntry(Buffer.from('vigilmon-probe'));

    // Close cleanly — this fences the ledger
    await ledger.close();
    await client.close();

    return res.status(200).json({ status: 'ok', write: 'ok' });
  } catch (err) {
    try { if (ledger) await ledger.close(); } catch (_) {}
    try { if (client) await client.close(); } catch (_) {}
    return res.status(503).json({ status: 'down', error: err.message });
  }
});

app.listen(3001);

Python Sidecar Health Endpoint

# bookkeeper_health.py — FastAPI sidecar for BookKeeper health
import os
import httpx
from fastapi import FastAPI
from fastapi.responses import JSONResponse

app = FastAPI()
BOOKIE_ADMIN_URL = os.environ.get("BOOKIE_ADMIN_URL", "http://localhost:8080")

@app.get("/health/bookkeeper")
async def bookkeeper_health():
    async with httpx.AsyncClient(timeout=10.0) as client:
        # Check bookie state via HTTP admin endpoint
        try:
            r = await client.get(f"{BOOKIE_ADMIN_URL}/api/v1/bookie/state")
            state = r.json()
            if not state.get("running"):
                return JSONResponse(status_code=503, content={"status": "down", "state": state})
            if state.get("readOnly"):
                return JSONResponse(status_code=503, content={"status": "degraded", "reason": "read_only", "state": state})
        except Exception as e:
            return JSONResponse(status_code=503, content={"status": "down", "check": "admin", "error": str(e)})

        # Check cluster info — number of available bookies
        try:
            r = await client.get(f"{BOOKIE_ADMIN_URL}/api/v1/bookkeeper/cluster_info")
            cluster = r.json()
            available = cluster.get("availableBookiesNum", 0)
            min_bookies = int(os.environ.get("BK_MIN_BOOKIES", "3"))
            if available < min_bookies:
                return JSONResponse(status_code=503, content={
                    "status": "degraded",
                    "reason": "insufficient_bookies",
                    "available": available,
                    "required": min_bookies,
                })
        except Exception as e:
            return JSONResponse(status_code=503, content={"status": "degraded", "check": "cluster_info", "error": str(e)})

    return {"status": "ok", "state": "running", "available_bookies": available}

Deploy and verify:

curl -i https://your-app.example.com/health/bookkeeper
# HTTP/1.1 200 OK
# {"status":"ok","state":"running","available_bookies":3}

Step 2: Configure a Vigilmon HTTP Monitor for BookKeeper

1. Log in to vigilmon.online and go to Monitors → New Monitor
2. Choose HTTP / HTTPS
3. Set the URL to your health endpoint: https://your-app.example.com/health/bookkeeper
4. Set the check interval to 1 minute
5. Under Expected response, configure:
   • Status code: 200
   • Response body contains: "status":"ok"
   • Response time threshold: 10000ms (ledger create involves ZooKeeper round-trips)
6. Under Alert channels, assign your Slack or email channel
7. Save the monitor

Vigilmon probes from multiple geographic regions. A single-region transient failure will not page you — Vigilmon requires multi-region consensus before opening an incident.

For a lighter-weight check using the native bookie state endpoint directly:

1. Create a second monitor pointing to http://bookie-1:8080/api/v1/bookie/state
2. Set the expected status code to 200
3. Add a response body check: "running":true
4. Add a response body check: "readOnly":false

Failure Coverage

| Failure Mode | Process Monitor | Vigilmon | |---|---|---| | Bookie JVM crash | ✓ | ✓ | | Bookie in read-only mode (disk full) | ✗ | ✓ | | Quorum write failure (too few bookies) | ✗ | ✓ | | ZooKeeper connectivity loss | ✗ | ✓ | | Ledger recovery storm (high latency) | ✗ | ✓ |

Step 3: Heartbeat Monitoring for BookKeeper-Dependent Producers

Applications that write to Apache Pulsar (which uses BookKeeper for storage) or directly to BookKeeper ledgers will fail silently when the write quorum cannot be formed. Pulsar producers will block indefinitely waiting for acknowledgement from bookies that are no longer available. Vigilmon's heartbeat monitors detect this silent stall.

Set Up the Heartbeat Monitor

1. In Vigilmon, go to Monitors → New Monitor → Heartbeat
2. Set the name: bookkeeper-producer
3. Set the expected interval to match your write frequency (e.g., 5 minutes)
4. Set the grace period: 10 minutes
5. Save — copy the heartbeat URL, e.g. https://vigilmon.online/heartbeat/abc123xyz

Wire It Into Your Pulsar Producer (Java)

// VigilmonHeartbeatInterceptor.java — Pulsar producer interceptor
import org.apache.pulsar.client.api.*;
import java.net.HttpURLConnection;
import java.net.URL;
import java.util.concurrent.atomic.AtomicLong;

public class VigilmonHeartbeatInterceptor<T> implements ProducerInterceptor<T> {
    private final String heartbeatUrl = System.getenv("VIGILMON_HEARTBEAT_URL");
    private final AtomicLong successCount = new AtomicLong(0);

    @Override
    public void onSendAcknowledgement(Producer<T> producer, Message<T> message,
                                       MessageId msgId, Throwable exception) {
        if (exception != null) return;
        // Ping every 100 successful sends (adjust to your throughput)
        if (successCount.incrementAndGet() % 100 == 0) {
            pingVigilmon();
        }
    }

    private void pingVigilmon() {
        try {
            HttpURLConnection conn = (HttpURLConnection) new URL(heartbeatUrl).openConnection();
            conn.setRequestMethod("GET");
            conn.setConnectTimeout(3000);
            conn.setReadTimeout(3000);
            conn.getResponseCode();
            conn.disconnect();
        } catch (Exception ignored) {}
    }

    @Override public Message<T> beforeSend(Producer<T> producer, Message<T> message) { return message; }
    @Override public void close() {}
}

Register the interceptor:

PulsarClient pulsarClient = PulsarClient.builder()
    .serviceUrl("pulsar://localhost:6650")
    .build();

Producer<byte[]> producer = pulsarClient.newProducer()
    .topic("persistent://public/default/my-topic")
    .intercept(new VigilmonHeartbeatInterceptor<>())
    .create();

Direct BookKeeper Write (Python)

import requests, os

HEARTBEAT_URL = os.environ["VIGILMON_HEARTBEAT_URL"]
write_count = 0

def write_to_bookkeeper(ledger, data: bytes):
    global write_count
    ledger.add_entry(data)
    write_count += 1
    if write_count % 50 == 0:
        requests.get(HEARTBEAT_URL, timeout=5)

Step 4: Monitor Multiple Bookies

Each bookie in the ensemble is a potential single point of failure. Monitor them individually:

[bookkeeper-bookie-1] :8080/api/v1/bookie/state
[bookkeeper-bookie-2] :8080/api/v1/bookie/state
[bookkeeper-bookie-3] :8080/api/v1/bookie/state
[bookkeeper-cluster]  /health/bookkeeper (sidecar)

The sidecar cluster monitor checks whether the ensemble has enough bookies to form write quorums. Individual bookie monitors detect which specific node entered read-only mode or became unreachable.

Name monitors with the bookie number in brackets so alert notifications immediately identify which node to investigate.

Group all BookKeeper monitors into a single Vigilmon Status Page alongside any dependent Pulsar broker monitors.

Step 5: Alert Routing for Write-Ahead Log Outages

BookKeeper outages propagate upward through any system that depends on it. Pulsar cannot accept new messages; DistributedLog streams stall; financial transaction logs stop recording. The blast radius is large and the failure is invisible to end users until their transactions time out.

Configure alert channels in Vigilmon:

1. Cluster sidecar health monitor → immediate Slack + PagerDuty (P1 — write quorum at risk)
2. Individual bookie state monitors → Slack (P2 — reduced redundancy)
3. Producer heartbeat monitors → Slack + PagerDuty (P1 — active data loss)

Set response time thresholds carefully:

• Alert at 5000ms on the cluster health monitor — BookKeeper write latency spikes signal a bookie under disk pressure before it goes full read-only
• Alert at 2000ms on individual bookie state endpoints — a slow admin response is an early indicator of GC pressure or disk I/O contention

Summary

Apache BookKeeper failures are silent and cascading — ensemble degradation leads to write quorum failure, which halts every system depending on it for durability. Vigilmon monitors the full path.

| Monitor Type | What It Covers | |---|---| | HTTP monitor on sidecar /health/bookkeeper | Write quorum viability, available bookie count | | HTTP monitor on /api/v1/bookie/state | Individual bookie health, read-only detection | | Heartbeat monitor | Producer/writer liveness and active throughput |

Get started free at vigilmon.online — your first BookKeeper monitor takes under two minutes to configure.