tutorial

Monitoring Hazelcast with Vigilmon: In-Memory Computing Availability, Cluster Membership, and External Uptime Checks

Practical guide to uptime and health monitoring for Hazelcast — HTTP health endpoints that probe real cluster connectivity, member health, and independent external monitoring with Vigilmon.

Hazelcast is an open-source in-memory computing platform that provides distributed data structures, stream processing, and low-latency compute across a cluster of JVM members. It's the session cache behind high-traffic web applications, the event streaming engine for real-time analytics, and the distributed job scheduler for financial risk systems. Its cluster topology is peer-to-peer — every member knows every other member — but that means a split-brain event, a member failing to rejoin after a GC pause, or a near-cache desync can degrade data availability without killing any individual process. Vigilmon adds an independent external monitoring layer — HTTP health checks that verify real Hazelcast connectivity from your application's network vantage point and alert before data inconsistency reaches your users.

This tutorial shows you how to build meaningful monitoring for Hazelcast-backed applications using Vigilmon.

What You'll Build

  • A health endpoint that probes real Hazelcast map/cache connectivity and checks cluster membership
  • Split-brain detection and member count monitoring
  • A Vigilmon HTTP monitor with appropriate timeout and assertion settings
  • A heartbeat monitor for Hazelcast-backed distributed jobs
  • An alerting strategy tuned for in-memory cluster failure modes

Prerequisites

  • A running Hazelcast cluster (single-member dev or multi-member production)
  • An application connected to Hazelcast via its Java client, REST API, or Python client
  • A free account at vigilmon.online

Step 1: Add a Health Endpoint to Your Hazelcast Service

Hazelcast exposes a built-in REST API on port 5701. The /hazelcast/rest/cluster endpoint returns cluster membership state — use it alongside a map put/get probe to give Vigilmon something meaningful to check.

Node.js (Hazelcast Node.js client)

// routes/health.js
import { Client } from 'hazelcast-client';

let hazelcastClient = null;

async function getClient() {
  if (!hazelcastClient) {
    hazelcastClient = await Client.newHazelcastClient({
      clusterName: process.env.HAZELCAST_CLUSTER_NAME || 'dev',
      network: {
        clusterMembers: (process.env.HAZELCAST_MEMBERS || 'localhost:5701').split(','),
      },
      connectionStrategy: {
        asyncStart: false,
        reconnectMode: 'ON',
        connectionRetry: { clusterConnectTimeoutMillis: 5000 },
      },
    });
  }
  return hazelcastClient;
}

export async function healthHandler(req, res) {
  const checks = {};
  let ok = true;
  const start = Date.now();

  try {
    const client = await getClient();

    // Check 1: Cluster membership
    const clusterService = client.getClusterService();
    const members = clusterService.getMembers();
    checks.pingLatencyMs = Date.now() - start;
    checks.memberCount = members.length;
    checks.connectivity = members.length > 0 ? 'ok' : 'error: no members';

    if (members.length === 0) {
      ok = false;
    }

    // Check 2: Distributed map put/get probe
    try {
      const probeStart = Date.now();
      const map = await client.getMap('vigilmon_probe');
      await map.set('health', { ts: Date.now() });
      const val = await map.get('health');
      checks.mapLatencyMs = Date.now() - probeStart;

      if (!val || !val.ts) {
        checks.mapProbe = 'error: get returned null or invalid value';
        ok = false;
      } else {
        checks.mapProbe = 'ok';
      }
    } catch (err) {
      checks.mapProbe = `error: ${err.message}`;
      ok = false;
    }

    // Check 3: Cluster state via member attribute
    try {
      const lifecycleService = client.getLifecycleService();
      const isRunning = lifecycleService.isRunning();
      checks.clientState = isRunning ? 'running' : 'not running';
      if (!isRunning) ok = false;
    } catch (err) {
      checks.clientState = `error: ${err.message}`;
      ok = false;
    }

  } catch (err) {
    checks.connectivity = `error: ${err.message}`;
    checks.memberCount = 0;
    ok = false;
  }

  res.status(ok ? 200 : 503).json({
    status: ok ? 'ok' : 'degraded',
    service: 'hazelcast-service',
    checks,
  });
}

Python (Hazelcast Python client)

# routers/health.py
import os, time
import hazelcast
from fastapi import APIRouter
from fastapi.responses import JSONResponse

router = APIRouter()

_hz_client = None

def get_client():
    global _hz_client
    if _hz_client is None:
        members = os.environ.get('HAZELCAST_MEMBERS', 'localhost:5701').split(',')
        _hz_client = hazelcast.HazelcastClient(
            cluster_name=os.environ.get('HAZELCAST_CLUSTER_NAME', 'dev'),
            cluster_members=members,
            connection_timeout=5.0,
        )
    return _hz_client

@router.get("/health")
async def health():
    checks = {}
    ok = True
    start = time.monotonic()

    try:
        client = get_client()

        # Check 1: Cluster membership
        members = client.cluster_service.get_members()
        checks['pingLatencyMs'] = int((time.monotonic() - start) * 1000)
        checks['memberCount'] = len(members)
        checks['connectivity'] = 'ok' if members else 'error: no members'
        if not members:
            ok = False

        # Check 2: Distributed map probe
        try:
            probe_start = time.monotonic()
            probe_map = client.get_map('vigilmon_probe').blocking()
            probe_map.set('health', int(time.time()))
            val = probe_map.get('health')
            checks['mapLatencyMs'] = int((time.monotonic() - probe_start) * 1000)

            if val is None:
                checks['mapProbe'] = 'error: get returned None'
                ok = False
            else:
                checks['mapProbe'] = 'ok'
        except Exception as e:
            checks['mapProbe'] = f'error: {e}'
            ok = False

        # Check 3: Client lifecycle
        checks['clientRunning'] = client.lifecycle_service.is_running()
        if not checks['clientRunning']:
            ok = False

    except Exception as e:
        checks['connectivity'] = f'error: {e}'
        checks['memberCount'] = 0
        ok = False

    return JSONResponse(
        status_code=200 if ok else 503,
        content={'status': 'ok' if ok else 'degraded', 'service': 'hazelcast-service', 'checks': checks}
    )

Java (Hazelcast Java client)

// HealthController.java
import com.hazelcast.client.HazelcastClient;
import com.hazelcast.core.*;
import org.springframework.web.bind.annotation.*;
import org.springframework.http.*;
import java.util.*;

@RestController
public class HealthController {

    private final HazelcastInstance hazelcast;

    public HealthController(HazelcastInstance hazelcast) {
        this.hazelcast = hazelcast;
    }

    @GetMapping("/health")
    public ResponseEntity<Map<String, Object>> health() {
        Map<String, Object> checks = new LinkedHashMap<>();
        boolean ok = true;
        long start = System.currentTimeMillis();

        // Check 1: Cluster membership
        try {
            Set<Member> members = hazelcast.getCluster().getMembers();
            checks.put("pingLatencyMs", System.currentTimeMillis() - start);
            checks.put("memberCount", members.size());
            checks.put("connectivity", members.isEmpty() ? "error: no members" : "ok");
            if (members.isEmpty()) ok = false;
        } catch (Exception e) {
            checks.put("connectivity", "error: " + e.getMessage());
            checks.put("memberCount", 0);
            ok = false;
        }

        // Check 2: Distributed map put/get probe
        try {
            long probeStart = System.currentTimeMillis();
            IMap<String, Long> probeMap = hazelcast.getMap("vigilmon_probe");
            probeMap.set("health", System.currentTimeMillis());
            Long val = probeMap.get("health");
            checks.put("mapLatencyMs", System.currentTimeMillis() - probeStart);

            if (val == null) {
                checks.put("mapProbe", "error: get returned null");
                ok = false;
            } else {
                checks.put("mapProbe", "ok");
            }
        } catch (Exception e) {
            checks.put("mapProbe", "error: " + e.getMessage());
            ok = false;
        }

        // Check 3: Cluster state
        try {
            com.hazelcast.cluster.ClusterState clusterState = hazelcast.getCluster().getClusterState();
            checks.put("clusterState", clusterState.toString());
            if (clusterState != com.hazelcast.cluster.ClusterState.ACTIVE) {
                ok = false;
            }
        } catch (Exception e) {
            checks.put("clusterState", "error: " + e.getMessage());
            ok = false;
        }

        Map<String, Object> body = new LinkedHashMap<>();
        body.put("status", ok ? "ok" : "degraded");
        body.put("service", "hazelcast-service");
        body.put("checks", checks);
        return ResponseEntity.status(ok ? 200 : 503).body(body);
    }
}

You can also check the built-in Hazelcast health check endpoint directly if you've enabled REST:

# Hazelcast built-in health check (requires REST enabled in hazelcast.yaml)
curl http://localhost:5701/hazelcast/health
# {"nodeState":"ACTIVE","clusterState":"ACTIVE","clusterSafe":"true","migrationQueueSize":0,"clusterSize":3}

Deploy and verify manually before wiring it into Vigilmon:

curl -i https://your-app.example.com/health
# HTTP/1.1 200 OK
# {"status":"ok","service":"hazelcast-service","checks":{"pingLatencyMs":2,"memberCount":3,"connectivity":"ok","mapLatencyMs":1,"mapProbe":"ok","clusterState":"ACTIVE","clientRunning":true}}

Step 2: Configure the Vigilmon HTTP Monitor

  1. Log in to Vigilmon → Add Monitor → HTTP.
  2. URL: your service's health endpoint (e.g. https://api.example.com/health).
  3. Check interval: 60 seconds.
  4. Response timeout: 10 seconds (Hazelcast in-memory operations are fast; anything over 5 seconds signals cluster stress).
  5. Expected status: 200.
  6. JSON assertion: path status, expected value ok.
  7. Save the monitor.

Vigilmon probes from multiple geographic regions simultaneously, so transient single-region blips won't wake your on-call team. This is important for Hazelcast because member rejoins after GC pauses and partition migrations during scaling cause brief latency spikes that resolve on their own — you want durable alerts on sustained failures, not noise.

Tip: Add a second JSON assertion on checks.clusterState equals ACTIVE. This fires when Hazelcast has entered a non-active cluster state (PASSIVE, FROZEN, IN_TRANSITION) — writes are blocked but individual member health checks may still return OK.

Step 3: Heartbeat Monitoring for Hazelcast-Backed Distributed Jobs

Hazelcast is frequently used to schedule and execute distributed tasks — cache warming jobs, session cleanup, distributed lock-protected batch processes. These jobs can silently fail during split-brain healing, partition migration, or when the Hazelcast Executor Service reaches its thread pool limit. Vigilmon's heartbeat monitors detect this: your job pings Vigilmon after each successful run, and missed pings trigger an alert.

Set Up the Heartbeat Monitor

  1. In Vigilmon, go to Monitors → New Monitor → Heartbeat.
  2. Set the name: hazelcast-job-scheduler.
  3. Set the expected interval: 5 minutes (adjust to your job frequency).
  4. Set the grace period: 10 minutes.
  5. Save — copy the unique heartbeat URL.

Wire It Into Your Distributed Job

Node.js:

import fetch from 'node-fetch';
import { Client } from 'hazelcast-client';

const HEARTBEAT_URL = process.env.VIGILMON_HEARTBEAT_URL;

async function runDistributedJob(client) {
  // Use a distributed lock to ensure single execution
  const lock = await client.getCPSubsystem().getLock('job.sessionCleanup');
  await lock.lock();
  try {
    const map = await client.getMap('user_sessions');
    const now = Date.now();
    const entries = await map.entrySet();
    const expired = entries.filter(([, v]) => v.expiresAt < now);
    for (const [key] of expired) {
      await map.delete(key);
    }
  } finally {
    await lock.unlock();
  }

  // Only ping Vigilmon after successful job completion
  await fetch(HEARTBEAT_URL).catch(() => {});
}

setInterval(async () => {
  try {
    const client = await getClient();
    await runDistributedJob(client);
  } catch (err) {
    console.error(`Distributed job failed — heartbeat NOT sent: ${err.message}`);
  }
}, 5 * 60_000);

Python:

import os, time, requests
import hazelcast

HEARTBEAT_URL = os.environ['VIGILMON_HEARTBEAT_URL']

def run_distributed_job(client):
    # Use a distributed lock to ensure single execution
    lock = client.cp_subsystem.get_lock('job.sessionCleanup').blocking()
    lock.lock()
    try:
        sessions_map = client.get_map('user_sessions').blocking()
        now = int(time.time() * 1000)
        entries = sessions_map.entry_set()
        expired = [k for k, v in entries if v.get('expires_at', 0) < now]
        for key in expired:
            sessions_map.delete(key)
    finally:
        lock.unlock()

    # Only ping after successful job completion
    requests.get(HEARTBEAT_URL, timeout=5)

while True:
    try:
        client = get_client()
        run_distributed_job(client)
    except Exception as e:
        print(f'Distributed job failed — heartbeat NOT sent: {e}')
    time.sleep(300)

Step 4: Management Center and REST API for Direct Cluster Health Checks

For infrastructure-level monitoring alongside Vigilmon's application-layer checks:

# Hazelcast built-in health check endpoint (enable in hazelcast.yaml: rest-api.enabled: true)
curl http://localhost:5701/hazelcast/health | python3 -m json.tool

# Cluster state via REST
curl http://localhost:5701/hazelcast/rest/cluster

# Member list (requires REST enabled)
curl http://localhost:5701/hazelcast/rest/management/cluster/state

# Key fields to monitor:
# nodeState: ACTIVE (healthy) vs PASSIVE/FROZEN/IN_TRANSITION
# clusterState: ACTIVE — whole cluster state
# clusterSafe: true — no pending partition migrations or replica syncs
# migrationQueueSize: 0 — pending migrations (non-zero = rebalancing in progress)
# clusterSize: N — expected member count

# Prometheus metrics via /metrics endpoint (requires Hazelcast metrics enabled):
# hazelcast_map_get_latency_avg — average get latency
# hazelcast_map_put_latency_avg — average put latency
# hazelcast_cluster_size — current cluster member count
# hazelcast_partition_migration_active — 1 if migration in progress

Wrap these checks in a cron job or monitoring probe that pushes metrics to Prometheus or Grafana — and use Vigilmon for the independent external HTTP check that confirms your application can reach the cluster and execute operations end-to-end.

Step 5: Alerting Strategy

| Alert channel | When to use | |---|---| | PagerDuty | Connectivity probe fails (cluster unreachable) | | PagerDuty | clusterState not ACTIVE (cluster passive or frozen) | | PagerDuty | Map put/get probe fails | | Slack webhook | memberCount drops below expected minimum | | Slack webhook | migrationQueueSize > 0 sustained (extended rebalancing) | | Slack + Email | Map operation latency > 5ms (elevated for in-memory data) | | PagerDuty | Distributed job heartbeat missed | | Status page | Any user-facing service backed by Hazelcast |

Set response time thresholds as an early warning:

  • Alert at 3ms for map operation latency (in-memory map operations should be sub-millisecond under normal load)
  • Alert at 5000ms for health endpoint response time (anything over 5s on an in-memory platform signals severe GC or network issues)

What Vigilmon Catches That Internal Monitoring Misses

| Scenario | Management Center / REST metrics | Vigilmon | |---|---|---| | Cluster reachable from ops host but not from app servers | No | HTTP probe from app's perspective | | Cluster entered PASSIVE state (writes silently blocked) | REST health check only | clusterState check fires | | Split-brain merge in progress (data unavailable during reconciliation) | MC alert only | Map probe fails — 503 fires | | Member rejoining after GC pause (brief partition unavailability) | MC member list | Map probe latency spike fires | | Distributed job scheduler silently stopped | No signal — job just stops | Heartbeat grace period expires → alert | | Map operation latency spike (GC pause or off-heap eviction) | Prometheus metric only | Latency threshold fires | | Monitoring pipeline (Prometheus/Grafana) fails | Alerts silently lost | Vigilmon is external — unaffected |

Hazelcast's in-memory speed and distributed consistency depend on a healthy cluster topology. Split-brain events, cluster state transitions, and member failures happen faster than any manual response — and they don't crash individual JVM processes, so process-level monitors give you false comfort. External health checks with Vigilmon give you the independent signal that your distributed caches are actually serving data correctly, before these issues reach your application users.

Start monitoring your Hazelcast applications in under 5 minutes — register free at vigilmon.online.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles