How to Monitor Your Apache Accumulo Cluster (Free, Multi-Region)
Apache Accumulo is a sorted, distributed key-value store built on top of Apache Hadoop, ZooKeeper, and Apache Thrift. It powers cell-level security, massive-scale analytics, and geospatial indexing workloads at government agencies and enterprises worldwide. When it works, it handles billions of records with fine-grained access control. When it doesn't — when a tablet server crashes, a compaction queue backs up, or a ZooKeeper session expires — data ingest silently stalls, queries time out, and downstream applications start reading stale data.
By the end of this guide you'll have external uptime monitoring, multi-region health probes, heartbeat monitoring for Accumulo ingest jobs, and instant alerts — all on the free tier.
Why Accumulo failures are hard to catch
Accumulo is a distributed system with multiple moving parts — each can degrade independently and silently:
Tablet server crashes — Accumulo distributes data across tablet servers. When one crashes, the Master reassigns its tablets, but this recovery takes time. During recovery, queries targeting those tablets return errors or block indefinitely. Your monitoring won't catch this unless it actively queries affected key ranges.
Compaction backlog — Accumulo writes data to in-memory structures and flushes them as RFiles. Minor and major compactions merge these files. If compaction falls behind ingest rate, the number of RFiles per tablet grows, query latency increases, and eventually ingest can stall. The cluster looks healthy by CPU and memory metrics while this happens.
ZooKeeper session expiry — Accumulo relies heavily on ZooKeeper for coordination. A network blip between Accumulo and ZooKeeper can cause session expiry, triggering partial leadership re-election. Clients get sporadic AccumuloSecurityException errors that resolve then recur.
Bulk ingest pipeline stalls — Accumulo bulk ingest jobs that load pre-sorted RFiles directly bypass the write-ahead log. If the import directory isn't cleaned up correctly, future bulk imports fail with file system errors — silently producing no output and no alert.
Step 1: Expose a health endpoint that exercises Accumulo
A health endpoint that doesn't actually query Accumulo proves nothing. Exercise the real client:
With Spring Boot:
import org.apache.accumulo.core.client.*;
import org.apache.accumulo.core.client.security.tokens.PasswordToken;
import org.apache.accumulo.core.data.Key;
import org.apache.accumulo.core.data.Mutation;
import org.apache.accumulo.core.data.Range;
import org.apache.accumulo.core.data.Value;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.RestController;
import java.util.Map;
@RestController
public class HealthController {
private final AccumuloClient client;
private static final String HEALTH_TABLE = "_health_probe";
public HealthController() throws Exception {
client = Accumulo.newClient()
.to(System.getenv("ACCUMULO_INSTANCE"), System.getenv("ACCUMULO_ZK_HOSTS"))
.as(System.getenv("ACCUMULO_USER"),
new PasswordToken(System.getenv("ACCUMULO_PASSWORD")))
.build();
if (!client.tableOperations().exists(HEALTH_TABLE)) {
client.tableOperations().create(HEALTH_TABLE);
}
}
@GetMapping("/health")
public Map<String, Object> health() throws Exception {
long ts = System.currentTimeMillis();
// Write a probe mutation
try (BatchWriter writer = client.createBatchWriter(HEALTH_TABLE)) {
Mutation m = new Mutation("probe");
m.put("cf", "ts", new Value(Long.toString(ts)));
writer.addMutation(m);
}
// Read it back to verify round-trip
long readTs = -1;
try (Scanner scanner = client.createScanner(HEALTH_TABLE, Authorizations.EMPTY)) {
scanner.setRange(new Range("probe"));
for (Map.Entry<Key, Value> entry : scanner) {
readTs = Long.parseLong(entry.getValue().toString());
break;
}
}
if (readTs != ts) {
throw new IllegalStateException("Read-back timestamp mismatch: wrote "
+ ts + ", read " + readTs);
}
return Map.of(
"status", "UP",
"accumulo", "ok",
"instance", System.getenv("ACCUMULO_INSTANCE"),
"roundTripMs", System.currentTimeMillis() - ts
);
}
}
With a plain servlet:
import org.apache.accumulo.core.client.*;
import org.apache.accumulo.core.client.security.tokens.PasswordToken;
import org.apache.accumulo.core.data.*;
import javax.servlet.annotation.WebServlet;
import javax.servlet.http.*;
import java.io.*;
@WebServlet("/health")
public class AccumuloHealthServlet extends HttpServlet {
private AccumuloClient client;
@Override
public void init() throws javax.servlet.ServletException {
try {
client = Accumulo.newClient()
.to(System.getenv("ACCUMULO_INSTANCE"), System.getenv("ACCUMULO_ZK_HOSTS"))
.as(System.getenv("ACCUMULO_USER"),
new PasswordToken(System.getenv("ACCUMULO_PASSWORD")))
.build();
if (!client.tableOperations().exists("_health_probe")) {
client.tableOperations().create("_health_probe");
}
} catch (Exception e) {
throw new javax.servlet.ServletException("Accumulo init failed", e);
}
}
@Override
protected void doGet(HttpServletRequest req, HttpServletResponse res) throws IOException {
res.setContentType("application/json");
try {
long ts = System.currentTimeMillis();
try (BatchWriter w = client.createBatchWriter("_health_probe")) {
Mutation m = new Mutation("probe");
m.put("cf", "cq", new Value(Long.toString(ts)));
w.addMutation(m);
}
res.getWriter().write("{\"status\":\"UP\",\"ts\":" + ts + "}");
} catch (Exception e) {
res.setStatus(500);
res.getWriter().write("{\"status\":\"DOWN\",\"error\":\"" + e.getMessage() + "\"}");
}
}
}
The health check performs a real write-then-read against a dedicated probe table. A tablet server failure, ZooKeeper session expiry, or permission issue causes a non-2xx response, triggering your alert.
Step 2: Set up external monitoring with Vigilmon
With /health live, point Vigilmon at it:
- Sign up at vigilmon.online — free tier, no credit card
- Click New Monitor → HTTP
- Enter
https://yourdomain.com/health - Set check interval (5 minutes on free tier)
- Save
Vigilmon checks from multiple geographic regions. A non-2xx response or timeout opens an incident and sends you an alert before users start seeing query timeouts or empty results.
Add monitors for each critical endpoint:
| Endpoint | What it catches |
|---|---|
| /health | Tablet server failures, ZooKeeper session expiry, write failures |
| /api/query | Query path health, scanner timeouts |
| /api/ingest | Ingest API availability |
Step 3: Heartbeat monitoring for bulk ingest jobs
Accumulo bulk ingest jobs that load pre-sorted RFiles directly into the cluster run as scheduled batch processes with no HTTP endpoint. Heartbeat monitoring catches silent ingest failures:
import org.apache.accumulo.core.client.*;
import org.apache.accumulo.core.client.security.tokens.PasswordToken;
import java.net.HttpURLConnection;
import java.net.URL;
import java.nio.file.Path;
public class NightlyBulkIngestJob {
private final String heartbeatUrl = System.getenv("HEARTBEAT_INGEST_URL");
public void run() throws Exception {
AccumuloClient client = Accumulo.newClient()
.to(System.getenv("ACCUMULO_INSTANCE"), System.getenv("ACCUMULO_ZK_HOSTS"))
.as(System.getenv("ACCUMULO_USER"),
new PasswordToken(System.getenv("ACCUMULO_PASSWORD")))
.build();
Path importDir = prepareRFiles(); // generate sorted RFiles from source data
if (importDir == null) {
throw new IllegalStateException("RFile generation produced no output");
}
// Bulk import into Accumulo — faster than mutation-based ingest
client.tableOperations().importDirectory(importDir.toString())
.to("events")
.tableTime(true)
.load();
// Verify the import by checking row count
long rowCount = 0;
try (Scanner scanner = client.createScanner("events", Authorizations.EMPTY)) {
for (Map.Entry<Key, Value> ignored : scanner) {
rowCount++;
if (rowCount > 0) break; // just confirm rows exist
}
}
if (rowCount == 0) {
throw new IllegalStateException("Post-import scan returned 0 rows");
}
// Ping heartbeat only after successful verification
if (heartbeatUrl != null && !heartbeatUrl.isBlank()) {
HttpURLConnection conn = (HttpURLConnection) new URL(heartbeatUrl).openConnection();
conn.setRequestMethod("GET");
conn.setConnectTimeout(5_000);
conn.getResponseCode();
conn.disconnect();
}
client.close();
}
private Path prepareRFiles() {
// ... RFile generation from source data
return null;
}
}
In Vigilmon:
- Click New Monitor → Heartbeat
- Set the expected interval (e.g. 25 hours for a nightly ingest)
- Copy the unique ping URL
- Set the environment variable:
HEARTBEAT_INGEST_URL=https://vigilmon.online/api/heartbeat/your-unique-token
If the RFile generation fails, the bulk import throws, the post-import scan returns empty, or the import directory isn't cleaned up, the heartbeat is never pinged and you get an alert after one missed interval.
Step 4: Validate cluster health at startup
Accumulo cluster misconfigurations are easiest to catch at application startup rather than at 2 AM when the first query arrives:
import org.apache.accumulo.core.client.*;
import org.apache.accumulo.core.client.security.tokens.PasswordToken;
import org.springframework.boot.context.event.ApplicationReadyEvent;
import org.springframework.context.event.EventListener;
import org.springframework.stereotype.Component;
@Component
public class AccumuloStartupValidator {
private final AccumuloClient client;
public AccumuloStartupValidator(AccumuloClient client) {
this.client = client;
}
@EventListener(ApplicationReadyEvent.class)
public void validate() throws Exception {
// Verify we can list tables (proves ZooKeeper connectivity and auth)
var tables = client.tableOperations().list();
if (tables.isEmpty()) {
throw new IllegalStateException(
"Accumulo reports no tables — ZooKeeper connectivity or auth issue");
}
// Verify the required application tables exist
for (String required : new String[]{"events", "users", "audit_log"}) {
if (!client.tableOperations().exists(required)) {
throw new IllegalStateException(
"Required Accumulo table '" + required + "' does not exist");
}
}
// Verify write path works
if (!client.tableOperations().exists("_health_probe")) {
client.tableOperations().create("_health_probe");
}
try (BatchWriter w = client.createBatchWriter("_health_probe")) {
Mutation m = new Mutation("startup-probe");
m.put("cf", "ts",
new org.apache.accumulo.core.data.Value(
Long.toString(System.currentTimeMillis())));
w.addMutation(m);
}
}
}
A missing table, ZooKeeper connectivity failure, or permission misconfiguration now causes a startup crash, caught by your health check immediately after deploy rather than silently failing the first user query.
Step 5: Alerts and badge embed
Slack/Discord alerts:
- In Vigilmon go to Notifications → New Channel
- Choose Slack or Discord, paste your webhook URL
- Enable it on your monitors
You get an instant alert when a monitor trips and a recovery notification when it's back.
Add an uptime badge to your README:
[](https://vigilmon.online?utm_source=devto&utm_medium=article&utm_campaign=accumulo-tutorial)
What you've built
| What | How |
|---|---|
| External health checks | /health exercising real Accumulo write-then-read round-trip |
| Tablet server failure detection | Round-trip failure surfaces as non-2xx immediately |
| ZooKeeper session monitoring | Auth and connectivity probe in both health check and startup |
| Bulk ingest job monitoring | Heartbeat ping after each verified successful import |
| Startup validation | Table existence and write-path checks at deploy time |
| Instant alerts | Slack/Discord notifications |
| README status badge | Vigilmon badge embed |
The whole setup runs on the free tier in under 30 minutes. You'll catch tablet server failures, ZooKeeper session expiry, compaction-related query degradation, and silently stalled bulk ingest jobs before users see empty results or query timeouts.
Get started free at vigilmon.online — monitors running in under a minute, no credit card required.