tutorial

How to Monitor Apache ZooKeeper Health and Session State with Vigilmon

ZooKeeper failures cascade silently into every distributed system that depends on it. Learn how to monitor ZooKeeper node health, leader election state, and client session health with Vigilmon HTTP probes and heartbeat monitors.

Apache ZooKeeper is the coordination backbone of distributed systems — Kafka, HBase, Hadoop, and dozens of other services rely on it for leader election, distributed locks, and service discovery. When ZooKeeper degrades, the services that depend on it degrade silently: brokers lose partition leadership, HBase region servers fail to register, and distributed jobs stall waiting for locks that never release.

Vigilmon gives you external visibility into ZooKeeper health through HTTP probe monitoring and heartbeat monitors for the applications that depend on ZooKeeper coordination. This tutorial covers both.

Why ZooKeeper Needs External Monitoring

ZooKeeper's own four-letter commands (mntr, ruok, stat) expose rich internal metrics — but only if you're actively polling them. External monitoring with Vigilmon adds:

  • Proactive alerting when a ZooKeeper node stops responding or loses quorum
  • Leader election visibility so you know immediately when the ensemble re-elects and services pause
  • Session expiry detection via a health sidecar that tracks active client session counts
  • Heartbeat monitoring for dependent applications that use ZooKeeper for coordination locks

These layers work together: a ZooKeeper node can be technically running while serving no sessions, and an ensemble can maintain quorum while one node falls so far behind it causes split-brain reads.

Step 1: Build a ZooKeeper Health Endpoint

ZooKeeper exposes health via four-letter commands over a raw TCP socket — not HTTP. You need a lightweight HTTP sidecar that wraps the ZooKeeper admin interface.

Node.js Health Sidecar

// health/zookeeper.js
const express = require('express');
const net = require('net');

const app = express();

function sendFourLetterCommand(host, port, command) {
  return new Promise((resolve, reject) => {
    const client = net.createConnection({ host, port }, () => {
      client.write(command);
    });
    let data = '';
    client.on('data', chunk => { data += chunk; });
    client.on('end', () => resolve(data));
    client.on('error', reject);
    setTimeout(() => { client.destroy(); reject(new Error('timeout')); }, 3000);
  });
}

app.get('/health/zookeeper', async (req, res) => {
  const host = process.env.ZK_HOST || 'localhost';
  const port = parseInt(process.env.ZK_PORT || '2181');

  try {
    // ruok returns "imok" if the server is running
    const ruok = await sendFourLetterCommand(host, port, 'ruok');
    if (ruok.trim() !== 'imok') {
      return res.status(503).json({ status: 'down', reason: 'ruok_failed', response: ruok.trim() });
    }

    // mntr gives detailed stats including leader/follower mode and session count
    const mntr = await sendFourLetterCommand(host, port, 'mntr');
    const lines = mntr.split('\n').filter(Boolean);
    const stats = {};
    for (const line of lines) {
      const [key, value] = line.split('\t');
      if (key) stats[key.trim()] = value ? value.trim() : '';
    }

    const mode = stats['zk_server_state'] || 'unknown';
    const sessions = parseInt(stats['zk_num_alive_connections'] || '0');
    const pendingQueue = parseInt(stats['zk_outstanding_requests'] || '0');
    const avgLatency = parseFloat(stats['zk_avg_latency'] || '0');

    if (mode === 'unknown') {
      return res.status(503).json({ status: 'degraded', reason: 'cannot_determine_mode' });
    }

    if (pendingQueue > 100) {
      return res.status(503).json({
        status: 'degraded',
        reason: 'high_outstanding_requests',
        outstanding_requests: pendingQueue,
        mode,
        sessions,
      });
    }

    return res.status(200).json({
      status: 'ok',
      mode,
      sessions,
      avg_latency_ms: avgLatency,
      outstanding_requests: pendingQueue,
    });
  } catch (err) {
    return res.status(503).json({ status: 'down', error: err.message });
  }
});

app.get('/health/zookeeper/quorum', async (req, res) => {
  // Check all nodes in the ensemble
  const nodes = (process.env.ZK_NODES || 'localhost:2181').split(',');
  const results = await Promise.allSettled(
    nodes.map(async node => {
      const [host, port] = node.split(':');
      const ruok = await sendFourLetterCommand(host, parseInt(port || '2181'), 'ruok');
      return { node, ok: ruok.trim() === 'imok' };
    })
  );

  const nodeResults = results.map(r => r.status === 'fulfilled' ? r.value : { node: 'unknown', ok: false });
  const healthy = nodeResults.filter(r => r.ok).length;
  const quorum = Math.floor(nodes.length / 2) + 1;

  if (healthy < quorum) {
    return res.status(503).json({
      status: 'down',
      reason: 'quorum_lost',
      healthy_nodes: healthy,
      total_nodes: nodes.length,
      quorum_required: quorum,
      nodes: nodeResults,
    });
  }

  return res.status(200).json({
    status: 'ok',
    healthy_nodes: healthy,
    total_nodes: nodes.length,
    quorum_required: quorum,
    nodes: nodeResults,
  });
});

app.listen(3006);

Python Health Sidecar

# health_zookeeper.py
import socket, os
from flask import Flask, jsonify

app = Flask(__name__)

def four_letter_command(host, port, cmd, timeout=3):
    with socket.create_connection((host, port), timeout=timeout) as s:
        s.sendall(cmd.encode())
        data = b''
        while True:
            chunk = s.recv(4096)
            if not chunk:
                break
            data += chunk
        return data.decode()

@app.route('/health/zookeeper')
def health():
    host = os.environ.get('ZK_HOST', 'localhost')
    port = int(os.environ.get('ZK_PORT', '2181'))
    try:
        ruok = four_letter_command(host, port, 'ruok')
        if ruok.strip() != 'imok':
            return jsonify({'status': 'down', 'reason': 'ruok_failed'}), 503

        mntr = four_letter_command(host, port, 'mntr')
        stats = {}
        for line in mntr.splitlines():
            if '\t' in line:
                k, v = line.split('\t', 1)
                stats[k.strip()] = v.strip()

        mode = stats.get('zk_server_state', 'unknown')
        sessions = int(stats.get('zk_num_alive_connections', 0))
        outstanding = int(stats.get('zk_outstanding_requests', 0))

        if outstanding > 100:
            return jsonify({
                'status': 'degraded',
                'reason': 'high_outstanding_requests',
                'outstanding_requests': outstanding,
                'mode': mode,
            }), 503

        return jsonify({'status': 'ok', 'mode': mode, 'sessions': sessions})
    except Exception as e:
        return jsonify({'status': 'down', 'error': str(e)}), 503

if __name__ == '__main__':
    app.run(host='0.0.0.0', port=3006)

Using the ZooKeeper Admin Server (ZooKeeper 3.5+)

ZooKeeper 3.5+ includes a built-in admin HTTP server on port 8080:

# Check server health
curl http://localhost:8080/commands/ruok

# Get detailed stats
curl http://localhost:8080/commands/mntr

# Check leader status
curl http://localhost:8080/commands/leader

Point your Vigilmon monitor directly at these endpoints if you run ZooKeeper 3.5+.

Step 2: Configure Vigilmon HTTP Monitor for ZooKeeper

  1. Log in to vigilmon.online and go to Monitors → New Monitor
  2. Choose HTTP / HTTPS
  3. Set the URL to your ZooKeeper health endpoint: https://your-app.example.com/health/zookeeper
  4. Set the check interval to 1 minute
  5. Under Expected response, configure:
    • Status code: 200
    • Response body contains: "status":"ok"
    • Response time threshold: 3000ms
  6. Under Alert channels, assign your Slack or PagerDuty channel
  7. Save the monitor

Add a second monitor for quorum health:

  • URL: https://your-app.example.com/health/zookeeper/quorum
  • Expected: 200, body contains "status":"ok"
  • Interval: 2 minutes (quorum loss is slower-moving)
  • Alert channel: P1 pager for quorum failures — all dependent services will be impacted

Vigilmon's multi-region probe consensus filters transient network blips from genuine ZooKeeper failures.

Step 3: Heartbeat Monitoring for ZooKeeper-Dependent Applications

ZooKeeper health alone isn't enough. Applications that rely on ZooKeeper for distributed locks or leader election can be stuck in an election loop, holding expired sessions, or blocked waiting for a znode that was never created — while ZooKeeper itself appears healthy.

Vigilmon heartbeat monitors detect these silent application stalls: your service pings Vigilmon after each successful coordination operation. If pings stop, Vigilmon alerts.

Set Up the Heartbeat Monitor

  1. In Vigilmon, go to Monitors → New Monitor → Heartbeat
  2. Set the name: my-service-zk-leader-election
  3. Set the expected interval: 5 minutes
  4. Set the grace period: 10 minutes
  5. Save — copy the heartbeat URL, e.g. https://vigilmon.online/heartbeat/abc123xyz

Wire It Into a Java ZooKeeper Client (Apache Curator)

// ZooKeeperHealthBeater.java
import org.apache.curator.framework.CuratorFramework;
import org.apache.curator.framework.recipes.leader.LeaderLatch;
import java.net.http.HttpClient;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse;
import java.net.URI;

public class ZooKeeperHealthBeater {
    private final CuratorFramework client;
    private final LeaderLatch leaderLatch;
    private final String vigilmonHeartbeatUrl;
    private final HttpClient http = HttpClient.newHttpClient();

    public ZooKeeperHealthBeater(CuratorFramework client, String path, String heartbeatUrl) throws Exception {
        this.client = client;
        this.leaderLatch = new LeaderLatch(client, path, "my-service");
        this.vigilmonHeartbeatUrl = heartbeatUrl;
        leaderLatch.start();
    }

    public void doLeaderWork() throws Exception {
        if (leaderLatch.hasLeadership()) {
            // ... perform leader-only work ...

            // Ping Vigilmon to confirm election and work completed successfully
            http.send(
                HttpRequest.newBuilder(URI.create(vigilmonHeartbeatUrl)).GET().build(),
                HttpResponse.BodyHandlers.discarding()
            );
        }
    }
}

Wire It Into a Python ZooKeeper Client (kazoo)

from kazoo.client import KazooClient
from kazoo.recipe.election import Election
import requests, os, time

zk = KazooClient(hosts=os.environ['ZK_HOSTS'])
zk.start()

heartbeat_url = os.environ['VIGILMON_HEARTBEAT_URL']

def my_leader_work():
    while True:
        # Perform leader work
        process_coordination_task()

        # Signal Vigilmon the leader is alive and working
        try:
            requests.get(heartbeat_url, timeout=5)
        except Exception:
            pass

        time.sleep(60)

election = Election(zk, '/my-service/leader')
election.run(my_leader_work)

Step 4: Alert Routing for ZooKeeper Failures

ZooKeeper failures cascade fast — a lost quorum affects every service that depends on it within seconds. Route alerts by severity:

| Monitor | Alert Channel | Priority | |---|---|---| | ZooKeeper node /health/zookeeper | Slack + PagerDuty | P1 | | Ensemble quorum /health/zookeeper/quorum | Slack + PagerDuty (all oncall) | P1 | | Heartbeat: leader election service | Slack + email | P2 | | Heartbeat: distributed lock service | Slack | P2 |

Set response time thresholds as early warning signals:

  • Alert at 1000ms for the ZooKeeper health endpoint (elevated latency precedes session timeouts)
  • Alert at 2000ms for quorum check (slow node responses indicate ensemble stress)

For production ensembles, set up a status page in Vigilmon showing ZooKeeper node health alongside the dependent services — this makes cascade root-cause analysis instant during incidents.

Summary

ZooKeeper failures are coordination failures — they don't throw 500 errors, they cause dependent services to stall, re-elect, or fail silently. External monitoring catches these before they become incidents:

| Monitor Type | What It Covers | |---|---| | HTTP monitor on /health/zookeeper | Node liveness, server mode, session count, outstanding requests | | HTTP monitor on /health/zookeeper/quorum | Ensemble quorum status across all nodes | | Heartbeat monitor | Leader election completion, distributed lock acquisition liveness |

Get started free at vigilmon.online — your first ZooKeeper monitor is running in under two minutes.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles