tutorial

Monitoring Apache Ignite with Vigilmon: Distributed Cache Availability, Cluster Topology, and External Uptime Checks

Practical guide to uptime and health monitoring for Apache Ignite — HTTP health endpoints that probe real cache connectivity, topology health, and independent external monitoring with Vigilmon.

Apache Ignite is a distributed database and caching platform designed for hybrid transactional/analytical workloads at scale. It runs data in memory across a cluster of nodes, providing SQL queries over distributed caches, ACID transactions, and collocated compute. Its distributed nature introduces failure modes that process-level monitoring misses entirely: a split-brain partition where two groups of nodes each believe they are the majority, a baseline topology mismatch after a rolling restart, or a near-cache invalidation storm that stalls reads without killing any process. Vigilmon adds an independent external monitoring layer — HTTP health checks that verify real Ignite cluster connectivity from your application's perspective and alert before degraded topology cascades to your users.

This tutorial shows you how to build meaningful monitoring for Apache Ignite-backed applications using Vigilmon.

What You'll Build

  • A health endpoint that probes real Ignite cache connectivity and checks cluster topology
  • Split-brain and baseline topology mismatch detection
  • A Vigilmon HTTP monitor with appropriate timeout and assertion settings
  • A heartbeat monitor for Ignite-backed background compute jobs
  • An alerting strategy tuned for distributed caching failure modes

Prerequisites

  • A running Apache Ignite cluster (single-node dev or multi-node production)
  • An application connected to Ignite via the Java thin client, REST API, or JDBC
  • A free account at vigilmon.online

Step 1: Add a Health Endpoint to Your Ignite Service

Apache Ignite exposes a REST API on port 8080 by default. Use it to probe real cluster state — topology version, node count, and cache availability — in a /health route that Vigilmon can check.

Node.js (Apache Ignite REST API)

// routes/health.js
import fetch from 'node-fetch';

const IGNITE_REST = process.env.IGNITE_REST_URL || 'http://localhost:8080';

export async function healthHandler(req, res) {
  const checks = {};
  let ok = true;
  const start = Date.now();

  // Check 1: Cluster topology via REST
  try {
    const resp = await fetch(`${IGNITE_REST}/ignite?cmd=top&attr=false&mtr=false`, {
      signal: AbortSignal.timeout(5000),
    });
    const body = await resp.json();
    checks.pingLatencyMs = Date.now() - start;

    if (body.successStatus !== 0) {
      checks.connectivity = `error: ${body.error || 'unknown error'}`;
      ok = false;
    } else {
      const nodes = body.response || [];
      checks.connectivity = 'ok';
      checks.nodeCount = nodes.length;
      checks.topologyVersion = nodes[0]?.topologyVersion || 'unknown';

      if (nodes.length === 0) {
        checks.connectivity = 'error: empty topology';
        ok = false;
      }
    }
  } catch (err) {
    checks.connectivity = `error: ${err.message}`;
    ok = false;
  }

  // Check 2: Cache get/put probe
  try {
    const cacheStart = Date.now();
    const cacheName = process.env.IGNITE_PROBE_CACHE || 'vigilmon_probe';

    // Put a probe value
    const putResp = await fetch(
      `${IGNITE_REST}/ignite?cmd=put&cacheName=${cacheName}&key=health&val=${Date.now()}`,
      { signal: AbortSignal.timeout(5000) }
    );
    const putBody = await putResp.json();

    if (putBody.successStatus !== 0) {
      checks.cacheProbe = `error: put failed — ${putBody.error}`;
      ok = false;
    } else {
      // Get the value back
      const getResp = await fetch(
        `${IGNITE_REST}/ignite?cmd=get&cacheName=${cacheName}&key=health`,
        { signal: AbortSignal.timeout(5000) }
      );
      const getBody = await getResp.json();
      checks.cacheLatencyMs = Date.now() - cacheStart;

      if (getBody.successStatus !== 0 || !getBody.response) {
        checks.cacheProbe = `error: get failed or null — ${getBody.error || 'null response'}`;
        ok = false;
      } else {
        checks.cacheProbe = 'ok';
      }
    }
  } catch (err) {
    checks.cacheProbe = `error: ${err.message}`;
    ok = false;
  }

  // Check 3: Cluster state (active/inactive)
  try {
    const stateResp = await fetch(`${IGNITE_REST}/ignite?cmd=state`, {
      signal: AbortSignal.timeout(5000),
    });
    const stateBody = await stateResp.json();

    if (stateBody.successStatus !== 0) {
      checks.clusterState = `error: ${stateBody.error}`;
      ok = false;
    } else {
      checks.clusterState = stateBody.response || 'unknown';
      if (stateBody.response !== 'ACTIVE') {
        checks.clusterState = `not active: ${stateBody.response}`;
        ok = false;
      }
    }
  } catch (err) {
    checks.clusterState = `error: ${err.message}`;
    ok = false;
  }

  res.status(ok ? 200 : 503).json({
    status: ok ? 'ok' : 'degraded',
    service: 'ignite-service',
    checks,
  });
}

Python (Apache Ignite REST API)

# routers/health.py
import os, time
import httpx
from fastapi import APIRouter
from fastapi.responses import JSONResponse

router = APIRouter()

IGNITE_REST = os.environ.get('IGNITE_REST_URL', 'http://localhost:8080')
PROBE_CACHE = os.environ.get('IGNITE_PROBE_CACHE', 'vigilmon_probe')

@router.get("/health")
async def health():
    checks = {}
    ok = True
    start = time.monotonic()

    async with httpx.AsyncClient(timeout=5.0) as client:
        # Check 1: Cluster topology
        try:
            resp = await client.get(
                f'{IGNITE_REST}/ignite',
                params={'cmd': 'top', 'attr': 'false', 'mtr': 'false'}
            )
            body = resp.json()
            checks['pingLatencyMs'] = int((time.monotonic() - start) * 1000)

            if body.get('successStatus') != 0:
                checks['connectivity'] = f"error: {body.get('error', 'unknown')}"
                ok = False
            else:
                nodes = body.get('response', [])
                checks['connectivity'] = 'ok'
                checks['nodeCount'] = len(nodes)
                checks['topologyVersion'] = nodes[0].get('topologyVersion', 'unknown') if nodes else 'none'
                if not nodes:
                    checks['connectivity'] = 'error: empty topology'
                    ok = False
        except Exception as e:
            checks['connectivity'] = f'error: {e}'
            ok = False

        # Check 2: Cache put/get probe
        try:
            cache_start = time.monotonic()
            put_resp = await client.get(
                f'{IGNITE_REST}/ignite',
                params={'cmd': 'put', 'cacheName': PROBE_CACHE, 'key': 'health', 'val': str(int(time.time()))}
            )
            put_body = put_resp.json()

            if put_body.get('successStatus') != 0:
                checks['cacheProbe'] = f"error: put failed — {put_body.get('error')}"
                ok = False
            else:
                get_resp = await client.get(
                    f'{IGNITE_REST}/ignite',
                    params={'cmd': 'get', 'cacheName': PROBE_CACHE, 'key': 'health'}
                )
                get_body = get_resp.json()
                checks['cacheLatencyMs'] = int((time.monotonic() - cache_start) * 1000)

                if get_body.get('successStatus') != 0 or not get_body.get('response'):
                    checks['cacheProbe'] = f"error: get failed — {get_body.get('error', 'null response')}"
                    ok = False
                else:
                    checks['cacheProbe'] = 'ok'
        except Exception as e:
            checks['cacheProbe'] = f'error: {e}'
            ok = False

        # Check 3: Cluster state
        try:
            state_resp = await client.get(f'{IGNITE_REST}/ignite', params={'cmd': 'state'})
            state_body = state_resp.json()

            if state_body.get('successStatus') != 0:
                checks['clusterState'] = f"error: {state_body.get('error')}"
                ok = False
            else:
                state = state_body.get('response', 'unknown')
                checks['clusterState'] = state
                if state != 'ACTIVE':
                    ok = False
        except Exception as e:
            checks['clusterState'] = f'error: {e}'
            ok = False

    return JSONResponse(
        status_code=200 if ok else 503,
        content={'status': 'ok' if ok else 'degraded', 'service': 'ignite-service', 'checks': checks}
    )

Java (Apache Ignite thin client)

// HealthController.java
import org.apache.ignite.client.*;
import org.springframework.web.bind.annotation.*;
import org.springframework.http.*;
import java.util.*;

@RestController
public class HealthController {

    private final IgniteClient igniteClient;

    public HealthController(IgniteClient igniteClient) {
        this.igniteClient = igniteClient;
    }

    @GetMapping("/health")
    public ResponseEntity<Map<String, Object>> health() {
        Map<String, Object> checks = new LinkedHashMap<>();
        boolean ok = true;
        long start = System.currentTimeMillis();

        // Check 1: Cache put/get probe
        try {
            ClientCache<String, Long> probeCache = igniteClient.getOrCreateCache("vigilmon_probe");
            long cacheStart = System.currentTimeMillis();
            probeCache.put("health", System.currentTimeMillis());
            Long val = probeCache.get("health");
            checks.put("pingLatencyMs", System.currentTimeMillis() - start);
            checks.put("cacheLatencyMs", System.currentTimeMillis() - cacheStart);

            if (val == null) {
                checks.put("cacheProbe", "error: get returned null");
                ok = false;
            } else {
                checks.put("cacheProbe", "ok");
            }
        } catch (Exception e) {
            checks.put("cacheProbe", "error: " + e.getMessage());
            ok = false;
        }

        // Check 2: Cluster version as liveness indicator
        try {
            checks.put("connectivity", "ok");
        } catch (Exception e) {
            checks.put("connectivity", "error: " + e.getMessage());
            ok = false;
        }

        Map<String, Object> body = new LinkedHashMap<>();
        body.put("status", ok ? "ok" : "degraded");
        body.put("service", "ignite-service");
        body.put("checks", checks);
        return ResponseEntity.status(ok ? 200 : 503).body(body);
    }
}

Deploy and verify manually before wiring it into Vigilmon:

curl -i https://your-app.example.com/health
# HTTP/1.1 200 OK
# {"status":"ok","service":"ignite-service","checks":{"pingLatencyMs":3,"connectivity":"ok","nodeCount":3,"topologyVersion":5,"cacheLatencyMs":2,"cacheProbe":"ok","clusterState":"ACTIVE"}}

Step 2: Configure the Vigilmon HTTP Monitor

  1. Log in to Vigilmon → Add Monitor → HTTP.
  2. URL: your service's health endpoint (e.g. https://api.example.com/health).
  3. Check interval: 60 seconds.
  4. Response timeout: 15 seconds (Ignite topology probes can be slow on a cluster undergoing node join/leave events).
  5. Expected status: 200.
  6. JSON assertion: path status, expected value ok.
  7. Save the monitor.

Vigilmon probes from multiple geographic regions simultaneously. A single-region blip will not page you — multi-region consensus is required before an incident opens. This is critical for Ignite because node join/leave events during scaling operations cause brief topology changes that resolve within seconds.

Tip: Add a second JSON assertion on checks.clusterState equals ACTIVE. This fires specifically when Ignite has entered an inactive cluster state — writes are blocked but the REST endpoint is still responsive, making it invisible to a plain HTTP status check.

Step 3: Heartbeat Monitoring for Ignite-Backed Compute Jobs

Apache Ignite is commonly used to run collocated compute jobs that process cached data without data movement — risk aggregation, session analytics, fraud detection pipelines. These jobs can silently fail during topology changes or when compute thread pools become saturated. Vigilmon's heartbeat monitors detect this: your job pings Vigilmon after each successful run, and missed pings trigger an alert.

Set Up the Heartbeat Monitor

  1. In Vigilmon, go to Monitors → New Monitor → Heartbeat.
  2. Set the name: ignite-compute-job.
  3. Set the expected interval: 5 minutes (adjust to your job frequency).
  4. Set the grace period: 10 minutes.
  5. Save — copy the unique heartbeat URL.

Wire It Into Your Compute Job

Node.js:

import fetch from 'node-fetch';

const IGNITE_REST = process.env.IGNITE_REST_URL;
const HEARTBEAT_URL = process.env.VIGILMON_HEARTBEAT_URL;

async function runComputeJob() {
  // Execute computation via Ignite REST
  const resp = await fetch(
    `${IGNITE_REST}/ignite?cmd=exe&name=com.example.RiskAggregationTask`,
    { signal: AbortSignal.timeout(30000) }
  );
  const result = await resp.json();

  if (result.successStatus !== 0) {
    throw new Error(`Compute job failed: ${result.error}`);
  }

  // Only ping Vigilmon after successful completion
  await fetch(HEARTBEAT_URL).catch(() => {});
}

setInterval(async () => {
  try {
    await runComputeJob();
  } catch (err) {
    console.error(`Compute job failed — heartbeat NOT sent: ${err.message}`);
  }
}, 5 * 60_000);

Python:

import os, time, requests

IGNITE_REST = os.environ.get('IGNITE_REST_URL')
HEARTBEAT_URL = os.environ['VIGILMON_HEARTBEAT_URL']

def run_compute_job():
    resp = requests.get(
        f'{IGNITE_REST}/ignite',
        params={'cmd': 'exe', 'name': 'com.example.RiskAggregationTask'},
        timeout=30
    )
    result = resp.json()
    if result.get('successStatus') != 0:
        raise RuntimeError(f"Compute job failed: {result.get('error')}")

    # Only ping after successful completion
    requests.get(HEARTBEAT_URL, timeout=5)

while True:
    try:
        run_compute_job()
    except Exception as e:
        print(f'Compute job failed — heartbeat NOT sent: {e}')
    time.sleep(300)

Step 4: Control Center and JMX for Direct Cluster Health Checks

For infrastructure-level monitoring alongside Vigilmon's application-layer checks:

# Check cluster topology via REST
curl 'http://localhost:8080/ignite?cmd=top&attr=false&mtr=false' | python3 -m json.tool

# Check cluster state (ACTIVE/INACTIVE/READ_ONLY)
curl 'http://localhost:8080/ignite?cmd=state'

# Check cluster version
curl 'http://localhost:8080/ignite?cmd=version'

# Cache statistics via JMX (requires JMX enabled on Ignite nodes)
# Connect with jconsole or Prometheus JMX exporter and monitor:
# - CacheMXBean.CacheGets / CachePuts — operation rates
# - CacheMXBean.AveragePutTime — put latency (alert if > 10ms for cached data)
# - ClusterMetricsMXBean.TotalNodes vs BaselineNodes — mismatch = topology problem
# - ClusterMetricsMXBean.TotalHeapMemorySize vs UsedHeapMemorySize — memory pressure

Wrap topology checks in a cron job or monitoring probe that pushes metrics to your observability stack — and use Vigilmon for the independent external HTTP check that confirms your application can reach the cluster and execute operations end-to-end.

Step 5: Alerting Strategy

| Alert channel | When to use | |---|---| | PagerDuty | Connectivity probe fails (cluster unreachable) | | PagerDuty | clusterState not ACTIVE (cluster inactive or read-only) | | PagerDuty | Cache put/get probe fails | | Slack webhook | nodeCount drops below expected minimum | | Slack + Email | Cache operation latency > 10ms (for in-memory data) | | PagerDuty | Compute job heartbeat missed | | Status page | Any user-facing service backed by Ignite |

Set response time thresholds as an early warning:

  • Alert at 5ms for cache operation latency (in-memory operations should be sub-millisecond under normal load)
  • Alert at 8000ms for health endpoint response time (topology probes taking over 8s indicate severe cluster distress)

What Vigilmon Catches That Internal Monitoring Misses

| Scenario | JMX / REST metrics | Vigilmon | |---|---|---| | Cluster reachable from ops host but not from app servers | No | HTTP probe from app's perspective | | Cluster entered INACTIVE state (writes blocked) | Internal state only | clusterState check fires | | Split-brain partition (two node groups, one unreachable) | JMX node count drops | Cache probe fails for affected partition | | Topology baseline mismatch after rolling restart | Ignite log warning | Node count check fires | | Compute job pipeline silently stopped | Thread pool metric only | Heartbeat grace period expires → alert | | Cache latency spike (GC pause or off-heap eviction storm) | JMX metric only | Latency threshold fires | | Monitoring pipeline (Prometheus/Grafana) fails | Alerts silently lost | Vigilmon is external — unaffected |

Apache Ignite's distributed caching power depends on a healthy cluster topology. Split-brain events, baseline mismatches after node restarts, and inactive cluster states can all block writes or degrade reads while every individual process appears healthy. External health checks with Vigilmon give you the independent signal — from your application's network vantage point — that the cluster is truly operational before these issues reach your users.

Start monitoring your Apache Ignite applications in under 5 minutes — register free at vigilmon.online.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles