tutorial

How to Monitor Apache Curator Health and ZooKeeper Recipe State with Vigilmon

Apache Curator wraps ZooKeeper coordination but its recipe failures are invisible to standard monitoring. Learn how to monitor Curator client health, leader election liveness, and distributed lock state with Vigilmon HTTP probes and heartbeat monitors.

Apache Curator is the standard Java client library for Apache ZooKeeper — it wraps raw ZooKeeper sessions with retry policies, leader election, distributed locks, service discovery, and other high-level coordination recipes. But when a Curator recipe stalls — a leader latch that never acquires leadership, a distributed lock stuck in a wait loop, a service discovery cache that stops refreshing — nothing throws an exception. The application just sits there, waiting indefinitely, while downstream callers time out or hang.

Vigilmon gives you external visibility into Curator-dependent application health through HTTP probe monitoring and heartbeat monitors tied directly to recipe lifecycle events. This tutorial covers both.

Why Curator Needs External Monitoring

Curator handles the hard parts of ZooKeeper correctly — retries, session recovery, watcher re-registration — but those same automatic recovery behaviors make failures invisible:

  • Leader election stalls silently: a LeaderLatch may never call hasLeadership() if the ensemble is degraded, with no exception surfaced
  • Distributed lock timeouts accumulate: InterProcessMutex.acquire() blocks until a timeout you may have set too high
  • Service discovery caches go stale when ZooKeeper sessions expire and the cache stops refreshing
  • Connection state changes (SUSPENDED → LOST) are events, not exceptions — code that doesn't explicitly handle them silently operates on a stale view

External monitoring with Vigilmon adds:

  • Proactive alerting when a Curator-based service stops completing its coordination cycles
  • Heartbeat monitoring so you know immediately when a leader election or lock acquisition loop stops progressing
  • HTTP probe monitoring for Curator client connection state exposed via a health endpoint
  • Multi-region availability checking from outside your network perimeter

Step 1: Build a Curator Health Endpoint

Curator doesn't expose an HTTP health interface — you need to add one to your application that reflects client connection state and recipe liveness.

Java Spring Boot Health Endpoint (Actuator)

// CuratorHealthIndicator.java
import org.apache.curator.framework.CuratorFramework;
import org.apache.curator.framework.imps.CuratorFrameworkState;
import org.apache.curator.framework.state.ConnectionState;
import org.springframework.boot.actuate.health.Health;
import org.springframework.boot.actuate.health.HealthIndicator;
import org.springframework.stereotype.Component;

@Component
public class CuratorHealthIndicator implements HealthIndicator {

    private final CuratorFramework curator;

    public CuratorHealthIndicator(CuratorFramework curator) {
        this.curator = curator;
    }

    @Override
    public Health health() {
        CuratorFrameworkState state = curator.getState();
        if (state != CuratorFrameworkState.STARTED) {
            return Health.down()
                .withDetail("reason", "curator_not_started")
                .withDetail("state", state.name())
                .build();
        }

        // Check ZooKeeper connection state via the underlying client
        // CuratorFramework.getZookeeperClient().isConnected() checks the raw session
        boolean connected = curator.getZookeeperClient().isConnected();
        if (!connected) {
            return Health.down()
                .withDetail("reason", "zookeeper_not_connected")
                .withDetail("state", state.name())
                .build();
        }

        return Health.up()
            .withDetail("state", state.name())
            .withDetail("zookeeper_connected", true)
            .build();
    }
}

With Spring Boot Actuator, this exposes /actuator/health/curator automatically. Point Vigilmon directly at this endpoint.

Node.js / Express Health Sidecar (for non-Java services wrapping Curator output)

// health/curator.js
const express = require('express');
const { execSync } = require('child_process');

const app = express();

// This sidecar reads a shared state file written by your Curator-dependent Java app
// The Java app writes { "connected": true, "last_election_ms": 1234567890123 } periodically
const STATE_FILE = process.env.CURATOR_STATE_FILE || '/var/run/curator-health.json';
const STALE_THRESHOLD_MS = parseInt(process.env.STALE_THRESHOLD_MS || '120000'); // 2 minutes

app.get('/health/curator', (req, res) => {
  try {
    const raw = require('fs').readFileSync(STATE_FILE, 'utf8');
    const state = JSON.parse(raw);
    const age = Date.now() - (state.written_at_ms || 0);

    if (!state.connected) {
      return res.status(503).json({ status: 'down', reason: 'zookeeper_disconnected', state });
    }

    if (age > STALE_THRESHOLD_MS) {
      return res.status(503).json({
        status: 'degraded',
        reason: 'stale_state_file',
        age_ms: age,
        threshold_ms: STALE_THRESHOLD_MS,
      });
    }

    return res.status(200).json({ status: 'ok', age_ms: age, state });
  } catch (err) {
    return res.status(503).json({ status: 'down', error: err.message });
  }
});

app.listen(3007);

Java: Writing the State File

// CuratorStateWriter.java — run as a scheduled task every 30s
import org.apache.curator.framework.CuratorFramework;
import com.fasterxml.jackson.databind.ObjectMapper;
import java.io.*;
import java.util.*;

@Component
public class CuratorStateWriter {

    private final CuratorFramework curator;
    private final ObjectMapper mapper = new ObjectMapper();

    public CuratorStateWriter(CuratorFramework curator) {
        this.curator = curator;
    }

    @Scheduled(fixedDelay = 30000)
    public void writeState() throws Exception {
        Map<String, Object> state = new HashMap<>();
        state.put("connected", curator.getZookeeperClient().isConnected());
        state.put("framework_state", curator.getState().name());
        state.put("written_at_ms", System.currentTimeMillis());

        File file = new File(System.getenv().getOrDefault("CURATOR_STATE_FILE", "/var/run/curator-health.json"));
        mapper.writeValue(file, state);
    }
}

Python Health Sidecar (kazoo-based, equivalent recipes)

# health_curator.py
import os, json, time
from flask import Flask, jsonify

app = Flask(__name__)

STATE_FILE = os.environ.get('CURATOR_STATE_FILE', '/var/run/curator-health.json')
STALE_THRESHOLD_MS = int(os.environ.get('STALE_THRESHOLD_MS', '120000'))

@app.route('/health/curator')
def health():
    try:
        with open(STATE_FILE) as f:
            state = json.load(f)

        age_ms = int(time.time() * 1000) - state.get('written_at_ms', 0)

        if not state.get('connected'):
            return jsonify({'status': 'down', 'reason': 'zookeeper_disconnected'}), 503

        if age_ms > STALE_THRESHOLD_MS:
            return jsonify({'status': 'degraded', 'reason': 'stale_state_file', 'age_ms': age_ms}), 503

        return jsonify({'status': 'ok', 'age_ms': age_ms})
    except Exception as e:
        return jsonify({'status': 'down', 'error': str(e)}), 503

if __name__ == '__main__':
    app.run(host='0.0.0.0', port=3007)

Step 2: Configure Vigilmon HTTP Monitor for Curator

  1. Log in to vigilmon.online and go to Monitors → New Monitor
  2. Choose HTTP / HTTPS
  3. Set the URL to your Curator health endpoint: https://your-app.example.com/actuator/health/curator (or https://your-app.example.com/health/curator for the sidecar)
  4. Set the check interval to 1 minute
  5. Under Expected response, configure:
    • Status code: 200
    • Response body contains: "status":"UP" (Actuator) or "status":"ok" (sidecar)
    • Response time threshold: 5000ms
  6. Under Alert channels, assign your Slack or PagerDuty channel
  7. Save the monitor

Vigilmon's multi-region probe consensus means a single network blip won't page your team — only sustained failures trigger alerts.

Step 3: Heartbeat Monitoring for Curator Recipe Liveness

Connection state monitoring catches disconnects, but not recipe stalls. A Curator LeaderLatch can be "connected" while stuck waiting for an ensemble with too many failed nodes to grant leadership. A InterProcessMutex can be held by a dead process while your application queues indefinitely.

Vigilmon heartbeat monitors catch these: your application pings Vigilmon after each successful recipe cycle. If pings stop, Vigilmon alerts.

Set Up the Heartbeat Monitor

  1. In Vigilmon, go to Monitors → New Monitor → Heartbeat
  2. Set the name: my-service-curator-leader
  3. Set the expected interval: 5 minutes
  4. Set the grace period: 10 minutes
  5. Save — copy the heartbeat URL, e.g. https://vigilmon.online/heartbeat/abc123xyz

Wire It Into a LeaderLatch

// LeaderElectionService.java
import org.apache.curator.framework.CuratorFramework;
import org.apache.curator.framework.recipes.leader.LeaderLatch;
import org.apache.curator.framework.recipes.leader.LeaderLatchListener;
import java.net.http.*;
import java.net.URI;

@Service
public class LeaderElectionService implements LeaderLatchListener {

    private final LeaderLatch leaderLatch;
    private final HttpClient http = HttpClient.newHttpClient();
    private final String heartbeatUrl;

    public LeaderElectionService(CuratorFramework curator,
                                  @Value("${vigilmon.heartbeat.url}") String heartbeatUrl) throws Exception {
        this.heartbeatUrl = heartbeatUrl;
        this.leaderLatch = new LeaderLatch(curator, "/my-service/leader", "node-1");
        this.leaderLatch.addListener(this);
        this.leaderLatch.start();
    }

    @Override
    public void isLeader() {
        // We just became leader — signal Vigilmon that election completed
        try {
            http.sendAsync(
                HttpRequest.newBuilder(URI.create(heartbeatUrl)).GET().build(),
                HttpResponse.BodyHandlers.discarding()
            );
        } catch (Exception ignored) {}

        // Perform leader work
        performLeaderWork();
    }

    @Override
    public void notLeader() {
        // Leadership lost — do not ping heartbeat here
    }

    @Scheduled(fixedDelay = 60000)
    public void heartbeatIfLeader() {
        if (leaderLatch.hasLeadership()) {
            // Confirm we're still active as leader
            try {
                http.sendAsync(
                    HttpRequest.newBuilder(URI.create(heartbeatUrl)).GET().build(),
                    HttpResponse.BodyHandlers.discarding()
                );
            } catch (Exception ignored) {}
        }
    }
}

Wire It Into an InterProcessMutex Lock Loop

// DistributedLockService.java
import org.apache.curator.framework.recipes.locks.InterProcessMutex;
import java.util.concurrent.TimeUnit;
import java.net.http.*;
import java.net.URI;

@Service
public class DistributedLockService {

    private final InterProcessMutex mutex;
    private final HttpClient http = HttpClient.newHttpClient();
    private final String heartbeatUrl;

    public DistributedLockService(CuratorFramework curator,
                                   @Value("${vigilmon.heartbeat.url}") String heartbeatUrl) {
        this.mutex = new InterProcessMutex(curator, "/my-service/lock");
        this.heartbeatUrl = heartbeatUrl;
    }

    @Scheduled(fixedDelay = 30000)
    public void processWithLock() throws Exception {
        boolean acquired = mutex.acquire(10, TimeUnit.SECONDS);
        if (!acquired) {
            log.warn("Could not acquire distributed lock within 10 seconds");
            return;
        }
        try {
            // Perform protected work
            doProtectedWork();

            // Signal Vigilmon that lock acquisition and work completed
            http.sendAsync(
                HttpRequest.newBuilder(URI.create(heartbeatUrl)).GET().build(),
                HttpResponse.BodyHandlers.discarding()
            );
        } finally {
            mutex.release();
        }
    }
}

Step 4: Alert Routing for Curator Failures

Curator failures range from momentary session suspension (recoverable in seconds) to permanent quorum loss (requires operator action). Route alerts by severity:

| Monitor | Alert Channel | Priority | |---|---|---| | HTTP: /actuator/health/curator | Slack + PagerDuty | P1 | | Heartbeat: leader election | Slack + PagerDuty | P1 | | Heartbeat: distributed lock loop | Slack + email | P2 | | HTTP: Curator state file staleness | Slack | P2 |

Set response time thresholds as early warning signals:

  • Alert at 2000ms for health endpoint (elevated latency indicates ZooKeeper session pressure)
  • Alert at 4000ms for ZooKeeper admin endpoints that the sidecar queries

For production clusters, expose a status page in Vigilmon showing Curator-dependent service health alongside the underlying ZooKeeper ensemble — this makes it immediately clear whether an application recipe stall is caused by the application or by the ensemble.

Summary

Curator failures are coordination failures — they don't throw 500 errors, they cause distributed elections and locks to stall silently. External monitoring catches these before they become incidents:

| Monitor Type | What It Covers | |---|---| | HTTP monitor on /actuator/health/curator | Client connection state, session liveness | | HTTP monitor on state file endpoint | State file freshness, background thread health | | Heartbeat monitor on leader election | LeaderLatch acquisition and leader work completion | | Heartbeat monitor on lock loop | InterProcessMutex acquisition and protected work completion |

Get started free at vigilmon.online — your first Curator monitor is running in under two minutes.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles