tutorial

How to Monitor Apache Geode Uptime and Health with Vigilmon

Apache Geode split-brain scenarios, region entry eviction under memory pressure, WAN gateway queue overflow, and silent locator disconnections can degrade your distributed cache invisibly. Learn how to monitor Apache Geode externally with Vigilmon HTTP probes and heartbeat monitors for scheduled region backup jobs and WAN replication sync tasks.

Apache Geode is a distributed in-memory data grid used to power high-throughput, low-latency data access for enterprise applications — caching financial positions, session state, real-time analytics, and IoT event streams across clusters of dozens to hundreds of nodes. But "in-memory" and "distributed" compound the failure surface in ways that traditional monitoring misses. A Geode cluster undergoing a network partition may elect a new coordinator while retaining an isolated sub-cluster in split-brain, causing both partitions to accept writes that will conflict on reconnection. A region configured with eviction-action=local-destroy under memory pressure silently removes entries that your application assumes are present. A WAN gateway sender queue that grows beyond its configured limit drops events without an alert unless you have a dedicated statistics monitor configured. And a locator — the cluster membership coordinator — that becomes isolated from member JVMs causes new connections to fail while existing members continue to operate, creating a silent half-available state.

Vigilmon gives you external visibility into Apache Geode health through HTTP probe monitoring and heartbeat monitoring for scheduled backup jobs and WAN replication sync tasks. This tutorial walks through both.

Why Apache Geode Monitoring Needs More Than JMX Dashboards

Apache Geode exposes a rich JMX MBean interface and the Pulse monitoring UI. They cannot tell you:

  • Whether Geode is reachable from your application servers with a valid ClientCache connection through the locator, or whether the locator itself is isolated from the cluster
  • Whether a split-brain partition has formed — both partitions report themselves as healthy to their own internal monitors; only an external probe that detects conflicting writes reveals the split
  • Whether a region is silently evicting entries under memory pressure with eviction-action=local-destroy, causing cache misses that your application interprets as a database fallback
  • Whether the WAN gateway sender queue on a hub-spoke topology has overflowed and is dropping cross-region replication events
  • Whether a scheduled async event queue consumer (AEQ) that writes Geode region entries to a backing relational database has stalled or fallen behind
  • Whether a disk store persistence operation has failed and Geode is no longer persisting region entries, silently losing data on member restart

These failure modes cause silent data loss, cache inconsistency, and split-brain write conflicts that JMX dashboards do not surface as actionable alerts. External monitoring through Vigilmon catches them by probing the actual read/write paths your application depends on.

Step 1: Build an Apache Geode Health Endpoint

Apache Geode's native REST API (enabled with start-dev-rest-api=true on the server) exposes a /geode/v1 endpoint that accepts GET and PUT operations on regions. Alternatively, deploy a sidecar service that connects to Geode via the Java or Node.js client and exposes health results via HTTP.

Option A: Use the Built-in Geode REST API

If the Geode REST API is enabled on your cluster, configure a health region and probe it directly:

Enable the REST API in gemfire.properties or the cluster config:

start-dev-rest-api=true
http-service-port=7070
http-service-bind-address=0.0.0.0

Create a health region (idempotent, run once):

gfsh> create region --name=_VigilmonHealth --type=REPLICATE

Probe the region via REST:

# PUT a sentinel entry
curl -X PUT \
  http://geode-server:7070/geode/v1/_VigilmonHealth/probe \
  -H 'Content-Type: application/json' \
  -d '{"probe":"vigilmon","ts":1234567890}'

# GET it back
curl http://geode-server:7070/geode/v1/_VigilmonHealth/probe

Configure Vigilmon to probe the GET endpoint: http://geode-server:7070/geode/v1/_VigilmonHealth/probe.

Option B: Sidecar Health Service (Java / Spring Boot)

// GeodeHealthController.java — Spring Boot sidecar connecting to Geode via client cache
import org.apache.geode.cache.Region;
import org.apache.geode.cache.client.ClientCache;
import org.apache.geode.cache.client.ClientCacheFactory;
import org.apache.geode.cache.client.ClientRegionShortcut;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.RestController;
import org.springframework.http.ResponseEntity;
import java.time.Instant;
import java.util.Map;

@RestController
public class GeodeHealthController {

    private final ClientCache clientCache;
    private final Region<String, Map<String, Object>> healthRegion;

    public GeodeHealthController() {
        this.clientCache = new ClientCacheFactory()
            .addPoolLocator(
                System.getenv().getOrDefault("GEODE_LOCATOR_HOST", "localhost"),
                Integer.parseInt(System.getenv().getOrDefault("GEODE_LOCATOR_PORT", "10334"))
            )
            .set("log-level", "warning")
            .create();
        this.healthRegion = clientCache
            .<String, Map<String, Object>>createClientRegionFactory(ClientRegionShortcut.PROXY)
            .create("_VigilmonHealth");
    }

    @GetMapping("/health/geode")
    public ResponseEntity<Map<String, Object>> geodeHealth() {
        try {
            long writeStart = System.currentTimeMillis();

            // Put a sentinel entry — verifies locator connectivity and server write path
            healthRegion.put("probe", Map.of(
                "probe", "vigilmon",
                "ts", Instant.now().toString()
            ));
            long writeMs = System.currentTimeMillis() - writeStart;

            // Get it back — verifies read path
            long readStart = System.currentTimeMillis();
            Object result = healthRegion.get("probe");
            long readMs = System.currentTimeMillis() - readStart;

            if (result == null) {
                return ResponseEntity.status(503).body(Map.of(
                    "status", "degraded",
                    "reason", "sentinel_entry_missing_after_put",
                    "write_ms", writeMs,
                    "read_ms", readMs
                ));
            }

            return ResponseEntity.ok(Map.of(
                "status", "ok",
                "write_ms", writeMs,
                "read_ms", readMs
            ));
        } catch (Exception e) {
            return ResponseEntity.status(503).body(Map.of(
                "status", "down",
                "error", e.getMessage(),
                "error_type", e.getClass().getSimpleName()
            ));
        }
    }
}

Python / FastAPI Sidecar Example

# health_geode.py — FastAPI sidecar using the Geode REST API via requests
import os
import time
import requests as http_client
from fastapi import FastAPI
from fastapi.responses import JSONResponse

app = FastAPI()

GEODE_REST_BASE = os.environ.get("GEODE_REST_URL", "http://localhost:7070/geode/v1")
HEALTH_REGION = "_VigilmonHealth"
HEALTH_KEY = "probe"

@app.get("/health/geode")
async def geode_health():
    try:
        # PUT sentinel entry
        write_start = time.monotonic()
        put_response = http_client.put(
            f"{GEODE_REST_BASE}/{HEALTH_REGION}/{HEALTH_KEY}",
            json={"probe": "vigilmon", "ts": time.time()},
            timeout=5,
        )
        write_ms = int((time.monotonic() - write_start) * 1000)

        if put_response.status_code not in (200, 201):
            return JSONResponse(
                status_code=503,
                content={
                    "status": "degraded",
                    "reason": "put_failed",
                    "geode_status": put_response.status_code,
                    "write_ms": write_ms,
                },
            )

        # GET it back
        read_start = time.monotonic()
        get_response = http_client.get(
            f"{GEODE_REST_BASE}/{HEALTH_REGION}/{HEALTH_KEY}",
            timeout=5,
        )
        read_ms = int((time.monotonic() - read_start) * 1000)

        if get_response.status_code != 200:
            return JSONResponse(
                status_code=503,
                content={
                    "status": "degraded",
                    "reason": "get_failed_after_put",
                    "geode_status": get_response.status_code,
                    "write_ms": write_ms,
                    "read_ms": read_ms,
                },
            )

        return {"status": "ok", "write_ms": write_ms, "read_ms": read_ms}

    except http_client.exceptions.Timeout:
        return JSONResponse(
            status_code=503,
            content={"status": "down", "error": "Geode REST API timed out"},
        )
    except Exception as e:
        return JSONResponse(
            status_code=503,
            content={"status": "down", "error": str(e)},
        )

Verify the endpoint before wiring up Vigilmon:

curl -i http://your-geode-health-service.internal/health/geode
# HTTP/1.1 200 OK
# {"status":"ok","write_ms":5,"read_ms":3}

Step 2: Configure Vigilmon HTTP Monitor for Apache Geode

  1. Log in to vigilmon.online and go to Monitors → New Monitor
  2. Choose HTTP / HTTPS
  3. Set the URL to your health endpoint: http://your-geode-health-service.internal/health/geode
  4. Set the check interval to 1 minute
  5. Under Expected response, configure:
    • Status code: 200
    • Response body contains: "status":"ok"
    • Response time threshold: 2000ms
  6. Under Alert channels, assign your Slack or PagerDuty channel
  7. Save the monitor

Vigilmon probes from multiple geographic regions simultaneously, requiring multi-region consensus before opening an incident. This eliminates false alarms from transient single-probe network blips and distinguishes genuine Geode cluster degradation from locator connectivity issues.

Monitoring WAN Gateway Health Separately

In a WAN topology (hub-spoke or active-active multi-site), each site's gateway sender should be monitored independently. Configure a Vigilmon monitor per WAN gateway:

  • [geode-primary] /health/geode — P1 page on failure; local cluster is unavailable
  • [geode-wan-gateway] /health/geode/wan — P2 Slack alert; cross-region replication is stalled but local operations continue

Extend the sidecar health endpoint to check the WAN gateway queue size via JMX or the Geode REST statistics API and return 503 when the queue exceeds a configured threshold.

Step 3: Heartbeat Monitoring for Geode Backup Jobs and AEQ Consumers

Disk Store Backup Jobs

Apache Geode regions configured with disk persistence should be backed up on a schedule using gfsh backup disk-store. A backup job that fails (due to insufficient disk space, a member that disconnects mid-backup, or a Geode upgrade that changed the disk store format) produces a partial backup without raising an alert in the Pulse UI.

Async Event Queue (AEQ) Consumer Monitoring

AEQ listeners that write Geode region entries to a relational database (MySQL, PostgreSQL) or a message broker (Kafka) run asynchronously and can fall behind during traffic spikes. A consumer that exceeds its queue capacity starts dropping events silently when AsyncEventQueue is configured with dispatcher-threads=1 and batch-size too small for the incoming write rate.

Vigilmon heartbeat monitors detect these stalls: your backup job or AEQ consumer pings Vigilmon on each successful completion or processing batch. If pings stop arriving within the expected window, Vigilmon fires an alert.

Set Up the Heartbeat Monitor

  1. In Vigilmon, go to Monitors → New Monitor → Heartbeat
  2. Set the name: geode-disk-backup (or geode-aeq-consumer-orders)
  3. Set the expected interval: 24 hours for disk backups; 5 minutes for AEQ consumer heartbeats
  4. Set the grace period: 2 hours for backups; 15 minutes for AEQ consumers
  5. Save — copy the unique heartbeat URL, e.g. https://vigilmon.online/heartbeat/abc123xyz

Shell Script Heartbeat for Geode Disk Backups

#!/bin/bash
# geode-backup.sh — run via cron or your orchestrator

set -euo pipefail

GFSH_PATH="${GFSH_PATH:-/opt/geode/bin/gfsh}"
LOCATOR_HOST="${GEODE_LOCATOR_HOST:-localhost}"
LOCATOR_PORT="${GEODE_LOCATOR_PORT:-10334}"
BACKUP_DIR="${GEODE_BACKUP_DIR:-/var/backups/geode}"
HEARTBEAT_URL="${VIGILMON_HEARTBEAT_URL}"
TIMESTAMP=$(date +%Y%m%d_%H%M%S)
BACKUP_PATH="${BACKUP_DIR}/${TIMESTAMP}"

echo "Starting Geode disk-store backup to ${BACKUP_PATH}"

"${GFSH_PATH}" -e "
  connect --locator=${LOCATOR_HOST}[${LOCATOR_PORT}]
  backup disk-store --dir=${BACKUP_PATH}
  disconnect
"

echo "Backup complete. Sending Vigilmon heartbeat..."
curl -fsS "${HEARTBEAT_URL}" --max-time 5

echo "Done."

Java Heartbeat Example for AEQ Consumer Listener

// VigilmonHeartbeatAEQListener.java — AsyncEventListener that pings Vigilmon
import org.apache.geode.cache.asyncqueue.AsyncEvent;
import org.apache.geode.cache.asyncqueue.AsyncEventListener;
import java.net.URI;
import java.net.http.HttpClient;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse;
import java.time.Instant;
import java.time.Duration;
import java.util.List;
import java.util.concurrent.atomic.AtomicLong;

public class VigilmonHeartbeatAEQListener implements AsyncEventListener {

    private final HttpClient httpClient = HttpClient.newHttpClient();
    private final String heartbeatUrl = System.getenv("VIGILMON_HEARTBEAT_URL");
    private final long heartbeatIntervalMs = 4 * 60 * 1000L; // 4 minutes
    private final AtomicLong lastHeartbeatAt = new AtomicLong(0);

    @Override
    public boolean processEvents(List<AsyncEvent> events) {
        try {
            for (AsyncEvent event : events) {
                // Process each event — write to backing store, Kafka, etc.
                processRegionEvent(event);
            }

            // Ping Vigilmon periodically to prove the AEQ consumer is alive
            long now = System.currentTimeMillis();
            if (now - lastHeartbeatAt.get() >= heartbeatIntervalMs) {
                pingVigilmon();
                lastHeartbeatAt.set(now);
            }

            return true; // returning true removes events from the queue

        } catch (Exception e) {
            System.err.println("AEQ listener failed: " + e.getMessage());
            return false; // returning false retries the batch
            // No heartbeat — Vigilmon alerts after grace period
        }
    }

    private void processRegionEvent(AsyncEvent event) {
        // Your event processing logic (write to DB, publish to Kafka, etc.)
        String key = event.getKey().toString();
        Object value = event.getDeserializedValue();
        System.out.println("Processing event: " + event.getOperation() + " key=" + key);
    }

    private void pingVigilmon() {
        if (heartbeatUrl == null || heartbeatUrl.isEmpty()) return;
        try {
            HttpRequest request = HttpRequest.newBuilder()
                .uri(URI.create(heartbeatUrl))
                .timeout(Duration.ofSeconds(5))
                .GET()
                .build();
            httpClient.send(request, HttpResponse.BodyHandlers.discarding());
            System.out.println("Vigilmon heartbeat sent at " + Instant.now());
        } catch (Exception e) {
            System.err.println("Failed to send Vigilmon heartbeat: " + e.getMessage());
        }
    }

    @Override
    public void close() {}
}

Register the listener in your Geode cache configuration:

<!-- cache.xml -->
<async-event-queue id="orders-aeq-queue"
                   persistent="true"
                   parallel="false"
                   dispatcher-threads="2"
                   batch-size="500"
                   batch-time-interval="500">
  <async-event-listener>
    <class-name>com.yourcompany.geode.VigilmonHeartbeatAEQListener</class-name>
  </async-event-listener>
</async-event-queue>

Step 4: Alert Routing and Response Time Thresholds

Configure alert routing in Vigilmon to match the severity of each Apache Geode failure mode:

| Monitor | Alert Channel | Priority | |---|---|---| | Geode cluster health /health/geode | Slack + PagerDuty | P1 | | WAN gateway health /health/geode/wan | Slack | P2 | | Heartbeat: disk-store backup | Slack + email | P2 | | Heartbeat: AEQ consumer | Slack | P2 |

Set response time thresholds as early warnings before full availability failures:

  • Alert at 500ms for the health endpoint — an in-memory Geode PUT + GET round-trip should complete in under 10ms; latency above 500ms indicates GC pressure, region entry eviction under memory pressure, or locator connectivity degradation
  • Alert at 2000ms as a second threshold — signals severe GC pause, member departure during write, or split-brain detection (both partitions still accepting writes but with elevated latency)
  • Set a P1 page threshold at 10000ms to alert on-call before Vigilmon marks the monitor fully down — gives the team time to identify whether a locator is isolated before the cluster becomes quorumless

Summary

Apache Geode failures surface in subtle ways — silent entry eviction under memory pressure, split-brain partitions accepting conflicting writes, WAN gateway queue overflow dropping cross-region events, and stalled AEQ consumers delivering stale data to backing stores — long before your application surfaces them as user-visible errors. Vigilmon gives you external visibility across the full failure surface:

| Monitor Type | What It Covers | |---|---| | HTTP monitor on /health/geode | Locator connectivity, authenticated PUT/GET round-trip, member availability | | HTTP monitor on /health/geode/wan | WAN gateway sender queue health and cross-region replication continuity | | Heartbeat monitor: disk-store backup | Silent backup failures due to member disconnection or disk exhaustion | | Heartbeat monitor: AEQ consumer | Stalled async event processing and backing-store write lag |

Get started free at vigilmon.online — your first Apache Geode monitor is running in under two minutes.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles