How to Monitor RocksDB Uptime and Health with Vigilmon

RocksDB is the storage engine inside CockroachDB, TiKV, MyRocks, and dozens of other production systems at scale. It delivers extraordinary write throughput through log-structured merge trees, but that same design creates failure modes that conventional process monitoring cannot see: write stalls, compaction debt, block cache eviction pressure, and column family inconsistencies that let the process keep running while silently rejecting writes.

Vigilmon gives you external visibility into RocksDB-backed services through HTTP probe monitoring and heartbeat monitoring for background compaction and ingest pipelines. This tutorial walks through setting up both.

Why RocksDB Monitoring Matters

A running RocksDB process is not necessarily a healthy one. Internal metrics that matter but go undetected without instrumentation:

• Write stalls — RocksDB throttles incoming writes when L0 SST file count or pending compaction bytes exceed thresholds. Your application blocks; no crash occurs.
• Compaction debt — When compaction can't keep up with writes, read amplification spikes and latency balloons.
• Block cache pressure — A hot key access pattern can evict the block cache working set, degrading read performance for all keys.
• Column family divergence — In multi-CF setups, one CF can fall behind while others remain healthy.
• WAL corruption — An abrupt power loss can leave the write-ahead log in a state that blocks reopening.

External monitoring through Vigilmon catches these failure modes by testing the actual read/write path your users depend on.

Step 1: Build a RocksDB Health Endpoint

RocksDB has no network interface. You expose health via a thin HTTP wrapper in your application.

C++ with a Sidecar HTTP Server

For native C++ applications, a minimal embedded HTTP server (e.g., cpp-httplib) is the cleanest approach:

// health_server.cpp — RocksDB health endpoint
#include "httplib.h"
#include "rocksdb/db.h"
#include "rocksdb/status.h"
#include <string>
#include <nlohmann/json.hpp>

void StartHealthServer(rocksdb::DB* db) {
    httplib::Server svr;

    svr.Get("/health/rocksdb", [db](const httplib::Request&, httplib::Response& res) {
        const std::string PROBE_KEY = "__vigilmon_health_probe__";
        const std::string PROBE_VAL = "ok";

        rocksdb::WriteOptions wo;
        wo.sync = false;
        rocksdb::Status s = db->Put(wo, PROBE_KEY, PROBE_VAL);
        if (!s.ok()) {
            res.status = 503;
            res.set_content(R"({"status":"down","error":"write failed"})", "application/json");
            return;
        }

        std::string val;
        s = db->Get(rocksdb::ReadOptions(), PROBE_KEY, &val);
        db->Delete(rocksdb::WriteOptions(), PROBE_KEY);

        if (!s.ok() || val != PROBE_VAL) {
            res.status = 503;
            res.set_content(R"({"status":"down","error":"read-back mismatch"})", "application/json");
            return;
        }

        // Check for write stall via DB property
        std::string stall;
        db->GetProperty("rocksdb.is-write-stopped", &stall);
        if (stall == "1") {
            res.status = 503;
            res.set_content(R"({"status":"degraded","reason":"write_stopped"})", "application/json");
            return;
        }

        res.status = 200;
        res.set_content(R"({"status":"ok"})", "application/json");
    });

    svr.listen("0.0.0.0", 9090);
}

Java Example (using RocksDB JNI)

// RocksHealthEndpoint.java — Spring Boot health indicator for RocksDB
import org.rocksdb.*;
import org.springframework.boot.actuate.health.*;
import org.springframework.stereotype.Component;

@Component
public class RocksDbHealthIndicator implements HealthIndicator {
    private final RocksDB db;

    public RocksDbHealthIndicator(RocksDB db) {
        this.db = db;
    }

    @Override
    public Health health() {
        byte[] probeKey = "__vigilmon_probe__".getBytes();
        byte[] probeVal = "ok".getBytes();

        try {
            db.put(probeKey, probeVal);
            byte[] result = db.get(probeKey);
            db.delete(probeKey);

            if (result == null || !new String(result).equals("ok")) {
                return Health.down().withDetail("error", "read-back mismatch").build();
            }

            // Check write stall property
            String isWriteStopped = db.getProperty("rocksdb.is-write-stopped");
            if ("1".equals(isWriteStopped)) {
                return Health.down().withDetail("reason", "write_stopped").build();
            }

            return Health.up().build();
        } catch (RocksDBException e) {
            return Health.down(e).build();
        }
    }
}

Python Example (using python-rocksdb)

# healthcheck.py — RocksDB health endpoint for FastAPI
import rocksdb
import os
from fastapi import FastAPI
from fastapi.responses import JSONResponse

app = FastAPI()
db = rocksdb.DB(os.environ["ROCKSDB_PATH"], rocksdb.Options(), read_only=False)

@app.get("/health/rocksdb")
def rocksdb_health():
    try:
        probe_key = b"__vigilmon_health_probe__"
        probe_val = b"ok"

        db.put(probe_key, probe_val)
        result = db.get(probe_key)
        db.delete(probe_key)

        if result != probe_val:
            raise ValueError("Read-back mismatch")

        # Check write stall
        write_stopped = db.get_property(b"rocksdb.is-write-stopped")
        if write_stopped == b"1":
            return JSONResponse(status_code=503, content={
                "status": "degraded",
                "reason": "write_stopped",
            })

        return {"status": "ok"}
    except Exception as e:
        return JSONResponse(status_code=503, content={
            "status": "down",
            "error": str(e),
        })

Verify the endpoint manually:

curl -i https://your-app.example.com/health/rocksdb
# HTTP/1.1 200 OK
# {"status":"ok"}

Step 2: Configure a Vigilmon HTTP Monitor for RocksDB

1. Log in to vigilmon.online and go to Monitors → New Monitor
2. Choose HTTP / HTTPS
3. Set the URL to your RocksDB health endpoint: https://your-app.example.com/health/rocksdb
4. Set the check interval to 1 minute
5. Under Expected response, configure:
   • Status code: 200
   • Response body contains: "status":"ok"
   • Response time threshold: 3000ms (allow for brief compaction pauses)
6. Under Alert channels, assign your Slack or PagerDuty channel
7. Save the monitor

Vigilmon probes from multiple geographic regions simultaneously. Multi-region consensus prevents false positives from transient network issues.

What This Catches

| Failure | Process monitoring | Vigilmon | |---|---|---| | Write stall (L0 file count limit) | ✗ | ✓ | | WAL corruption on reopen | ✗ | ✓ | | Disk exhaustion blocking writes | ✗ | ✓ | | Compaction backpressure latency spike | ✗ | ✓ | | Application crash wrapping RocksDB | ✓ | ✓ |

Step 3: Heartbeat Monitoring for Compaction and Ingest Pipelines

Applications built on RocksDB often run:

• Bulk SST ingest pipelines (IngestExternalFile) that import data in batch
• Manual compaction jobs (CompactRange) triggered on a schedule
• Background analytics scanning column families

These jobs can stall silently when write stalls propagate upstream. Vigilmon's heartbeat monitors catch silent stalls: your pipeline pings Vigilmon after each successful cycle. If the ping stops arriving, Vigilmon fires an alert.

Set Up the Heartbeat Monitor

1. In Vigilmon, go to Monitors → New Monitor → Heartbeat
2. Set the name: rocksdb-ingest-pipeline
3. Set the expected interval: match your pipeline's cycle time
4. Set the grace period: 2× the interval
5. Save — copy the unique heartbeat URL

Wire It Into Your Pipeline

Python ingest pipeline:

import requests, os

def run_ingest_pipeline():
    while True:
        ingest_batch()  # your RocksDB IngestExternalFile logic
        requests.get(os.environ["VIGILMON_HEARTBEAT_URL"], timeout=5)
        time.sleep(3600)  # hourly ingest

Java batch job:

public void runCompactionJob() throws Exception {
    db.compactRange();  // manual compaction
    HttpClient client = HttpClient.newHttpClient();
    HttpRequest req = HttpRequest.newBuilder()
        .uri(URI.create(System.getenv("VIGILMON_HEARTBEAT_URL")))
        .GET().build();
    client.send(req, HttpResponse.BodyHandlers.discarding());
}

Step 4: Expose RocksDB Internal Metrics in Your Health Endpoint

RocksDB exposes rich internal statistics via GetProperty. Surface the most important ones in your health response to give Vigilmon context during incidents:

std::string pending_compaction;
db->GetProperty("rocksdb.estimate-pending-compaction-bytes", &pending_compaction);

std::string num_l0_files;
db->GetProperty("rocksdb.num-files-at-level0", &num_l0_files);

std::string mem_table_size;
db->GetProperty("rocksdb.size-all-mem-tables", &mem_table_size);

Useful RocksDB properties to expose:

| Property | What It Reveals | |---|---| | rocksdb.is-write-stopped | Active write stall | | rocksdb.is-write-slowdown | Approaching write stall | | rocksdb.num-files-at-level0 | L0 compaction pressure | | rocksdb.estimate-pending-compaction-bytes | Compaction debt | | rocksdb.block-cache-usage | Block cache utilization |

Return HTTP 503 when is-write-stopped=1 or num-files-at-level0 exceeds your configured level0_stop_writes_trigger.

Step 5: Alert Routing for RocksDB Failures

RocksDB write stalls are time-sensitive: if writes queue up in your application layer while RocksDB is stalled, your service's memory can exhaust before the stall clears. Alert fast.

In Vigilmon, configure per monitor:

1. Primary RocksDB health monitor → immediate Slack + PagerDuty page (P1)
2. Write slowdown threshold → Slack (P2 — early warning before full stall)
3. Ingest pipeline heartbeat → Slack + email (P2)

Set response time thresholds aggressively:

• Alert at 500ms for the health endpoint (healthy RocksDB responds in single-digit milliseconds)
• Alert at 2000ms for API endpoints backed by RocksDB reads

Summary

RocksDB's write stalls and compaction behavior are invisible to process-level monitoring but immediately user-visible. Vigilmon gives you:

| Monitor Type | What It Covers | |---|---| | HTTP monitor on /health/rocksdb | Write stall, WAL corruption, disk pressure, crash | | Heartbeat monitor | Ingest pipeline and compaction job liveness | | Response time threshold | Compaction backpressure early warning |

Get started free at vigilmon.online — your first RocksDB monitor is running in under two minutes.