How to Monitor Apache HBase Health and Region Server Availability with Vigilmon

Apache HBase is the distributed NoSQL database powering real-time read/write access to massive datasets on Hadoop — but when a region server crashes, a master failover stalls, or ZooKeeper loses quorum, your reads and writes start throwing org.apache.hadoop.hbase.NotServingRegionException silently. Unlike HTTP services, HBase failures often don't surface to users immediately; they manifest as timeouts, stale data, and slow scans that creep up over minutes before anyone notices.

Vigilmon gives you external visibility into HBase health through HTTP probe monitoring (via HBase REST Server or a custom health sidecar) and heartbeat monitoring for your HBase client applications. This tutorial covers both.

Why HBase Needs External Monitoring

HBase's built-in tooling (HBase Web UI on port 16010, JMX metrics, Ganglia/Nagios integrations) provides rich internal metrics — but only if someone is watching. External monitoring with Vigilmon adds:

• Proactive alerting when the HBase Master web UI becomes unreachable (early sign of master failure or GC storm)
• Region server availability detection via the REST API or a health sidecar that checks live region counts
• Heartbeat monitoring so you know immediately when a client application stops successfully writing to or reading from HBase
• Multi-region availability checking from outside your Hadoop cluster network

These layers work together: the HBase Master can appear healthy while a region server is unavailable for a specific table, and a region server can be reachable while the regions it serves are in transition due to a failed assignment.

Step 1: Build an HBase Health Endpoint

HBase exposes a web UI and REST API, but you need a simple HTTP health endpoint that Vigilmon can probe.

Using the HBase REST Server

If you have HBase REST Server running (default port 8080), it exposes a /version/cluster endpoint:

# Check HBase cluster version and status
curl -i http://hbase-rest.example.com:8080/version/cluster

# List tables — confirms region assignment is healthy
curl -i http://hbase-rest.example.com:8080/

# Check a specific table's regions
curl -i http://hbase-rest.example.com:8080/mytable/regions

Point a Vigilmon monitor directly at http://hbase-rest.example.com:8080/version/cluster for a quick reachability check.

Custom Java Health Sidecar

For deeper health checks — region server count, ZooKeeper connectivity, master liveness — build a lightweight Spring Boot health endpoint alongside your HBase application:

// HBaseHealthController.java
@RestController
public class HBaseHealthController {

    @Autowired
    private Connection hbaseConnection;

    @GetMapping("/health/hbase")
    public ResponseEntity<Map<String, Object>> checkHealth() {
        Map<String, Object> status = new LinkedHashMap<>();
        try {
            Admin admin = hbaseConnection.getAdmin();

            // Check master is alive
            boolean masterRunning = admin.isMasterRunning();
            if (!masterRunning) {
                status.put("status", "down");
                status.put("reason", "master_not_running");
                return ResponseEntity.status(503).body(status);
            }

            // Count live region servers
            ClusterStatus clusterStatus = admin.getClusterStatus();
            int liveServers = clusterStatus.getServersSize();
            int deadServers = clusterStatus.getDeadServerNames().size();

            if (liveServers == 0) {
                status.put("status", "down");
                status.put("reason", "no_live_region_servers");
                return ResponseEntity.status(503).body(status);
            }

            if (deadServers > 0) {
                status.put("status", "degraded");
                status.put("live_servers", liveServers);
                status.put("dead_servers", deadServers);
                return ResponseEntity.status(503).body(status);
            }

            status.put("status", "ok");
            status.put("live_servers", liveServers);
            status.put("regions_in_transition",
                clusterStatus.getRegionStatesInTransition().size());
            admin.close();
            return ResponseEntity.ok(status);

        } catch (IOException e) {
            status.put("status", "down");
            status.put("error", e.getMessage());
            return ResponseEntity.status(503).body(status);
        }
    }
}

Python Health Sidecar (happybase)

# hbase_health.py
from flask import Flask, jsonify
import happybase
import os

app = Flask(__name__)

@app.route('/health/hbase')
def hbase_health():
    try:
        connection = happybase.Connection(
            host=os.environ.get('HBASE_THRIFT_HOST', 'localhost'),
            port=int(os.environ.get('HBASE_THRIFT_PORT', '9090')),
            timeout=5000
        )
        # Listing tables confirms ZooKeeper and master connectivity
        tables = connection.tables()
        connection.close()
        return jsonify({'status': 'ok', 'table_count': len(tables)}), 200
    except Exception as e:
        return jsonify({'status': 'down', 'error': str(e)}), 503

if __name__ == '__main__':
    app.run(port=8090)

Step 2: Configure Vigilmon HTTP Monitor for HBase

1. Log in to vigilmon.online and go to Monitors → New Monitor
2. Choose HTTP / HTTPS
3. Set the URL to your HBase health endpoint: https://your-app.example.com/health/hbase
4. Set the check interval to 1 minute
5. Under Expected response, configure:
   • Status code: 200
   • Response body contains: "status":"ok"
   • Response time threshold: 10000ms (HBase admin API calls involve ZooKeeper round-trips)
6. Under Alert channels, assign your Slack or PagerDuty channel
7. Save the monitor

Add a second monitor for the HBase REST Server directly:

• URL: http://hbase-rest.example.com:8080/version/cluster
• Expected: 200
• Interval: 2 minutes
• Alert channel: on-call channel

Vigilmon's multi-region probe consensus prevents false positives from transient network blips between probe nodes and your Hadoop cluster.

Step 3: Heartbeat Monitoring for HBase Client Applications

Region server health and ZooKeeper connectivity are necessary — but not sufficient. An HBase client application can be stuck in retry loops on a degraded table, blocked on a slow compaction, or quietly failing row mutations, while the cluster metrics look healthy.

Vigilmon heartbeat monitors detect silent application stalls: your application pings Vigilmon after each successful HBase operation batch. If pings stop, Vigilmon fires an alert.

Set Up the Heartbeat Monitor

1. In Vigilmon, go to Monitors → New Monitor → Heartbeat
2. Set the name: hbase-writer-application
3. Set the expected interval: 5 minutes (adjust to your write throughput)
4. Set the grace period: 10 minutes
5. Save — copy the heartbeat URL, e.g. https://vigilmon.online/heartbeat/abc123xyz

Wire It Into Your Java HBase Client

// HBaseWriter.java
import org.apache.hadoop.hbase.client.*;
import java.net.http.*;
import java.net.URI;

public class HBaseWriter {

    private final Connection connection;
    private final String heartbeatUrl = System.getenv("VIGILMON_HEARTBEAT_URL");
    private final HttpClient httpClient = HttpClient.newHttpClient();
    private long operationsSinceHeartbeat = 0;
    private static final int HEARTBEAT_EVERY = 500;

    public void writeRow(String tableName, Put put) throws IOException {
        try (Table table = connection.getTable(TableName.valueOf(tableName))) {
            table.put(put);
            operationsSinceHeartbeat++;

            if (operationsSinceHeartbeat >= HEARTBEAT_EVERY) {
                pingVigilmon();
                operationsSinceHeartbeat = 0;
            }
        }
    }

    private void pingVigilmon() {
        try {
            HttpRequest req = HttpRequest.newBuilder()
                .uri(URI.create(heartbeatUrl))
                .GET().build();
            httpClient.sendAsync(req, HttpResponse.BodyHandlers.discarding());
        } catch (Exception e) {
            // Non-fatal: don't let heartbeat failure interrupt HBase writes
        }
    }
}

Python HBase Client with Heartbeat (happybase)

# hbase_worker.py
import happybase, requests, os, time

connection = happybase.Connection(
    host=os.environ['HBASE_THRIFT_HOST'],
    port=int(os.environ.get('HBASE_THRIFT_PORT', '9090'))
)
table = connection.table(os.environ['HBASE_TABLE'])

last_heartbeat = time.time()
HEARTBEAT_INTERVAL = 60  # seconds
rows_processed = 0

for row_key, data in get_rows_to_process():
    table.put(row_key, data)
    rows_processed += 1

    if time.time() - last_heartbeat > HEARTBEAT_INTERVAL:
        try:
            requests.get(os.environ['VIGILMON_HEARTBEAT_URL'], timeout=5)
        except Exception:
            pass
        last_heartbeat = time.time()

print(f"Processed {rows_processed} rows")

Step 4: Alert Routing for HBase Failures

HBase failures cascade: a ZooKeeper session expiry causes the master to restart, which causes region servers to lose their leases, which causes regions to enter the OFFLINE state, which causes client reads and writes to fail with RegionServerStoppedException. Alert routing should reflect this cascade order.

Configure alert routing in Vigilmon:

| Monitor | Alert Channel | Priority | |---|---|---| | HBase cluster health /health/hbase | Slack + PagerDuty | P1 | | HBase REST Server /version/cluster | Slack | P2 | | Heartbeat: HBase writer application | Slack + email | P2 | | Heartbeat: HBase reader/analytics job | Email | P3 |

Set response time thresholds for early warning:

• Alert at 5000ms for the cluster health endpoint (slow ZooKeeper round-trips signal quorum pressure)
• Alert at 15000ms for the REST server endpoint (slow REST responses signal GC pressure on region servers)

For write-critical applications, enable two consecutive failures before alerting — HBase's split/compaction operations can briefly cause elevated latency without representing a real outage.

Summary

HBase failures are silent at the application layer. External monitoring catches master failovers, region server losses, and client application stalls before they become data availability outages:

| Monitor Type | What It Covers | |---|---| | HTTP monitor on /health/hbase | Master liveness, region server count, ZooKeeper connectivity | | HTTP monitor on HBase REST Server | REST API availability, table accessibility | | Heartbeat monitor | Client application liveness, write/read processing health |

Get started free at vigilmon.online — your first HBase monitor is running in under two minutes.