How to Monitor OpenTSDB Availability and HBase Backend Health with Vigilmon

OpenTSDB is a distributed time-series database built on Apache HBase, designed to store billions of data points with millisecond-precision timestamps — but when an HBase region server becomes unavailable, a TSD process exhausts its async I/O thread pool, or a compaction storm drives write latency above the client timeout threshold, data points are silently dropped without triggering any outbound alert. OpenTSDB exposes an HTTP API and version endpoint, but these require active polling and give no push notifications when HBase connectivity or write throughput degrades.

Vigilmon gives you external visibility into OpenTSDB through HTTP monitors on its version and query endpoints, a sidecar that probes write latency and HBase connectivity, and heartbeat monitors that confirm your metrics ingestion pipeline is completing end-to-end. This tutorial covers both.

Why OpenTSDB Needs External Monitoring

OpenTSDB's built-in tooling (the TSD process logs, HBase status UI, and internal Telnet/HTTP metrics) requires a human operator or internal alerting system actively watching those outputs. External monitoring with Vigilmon adds:

• TSD process availability alerting when the OpenTSDB daemon becomes unreachable before your monitoring pipeline stalls
• HBase connectivity health checks detecting region server failures that cause write errors before clients report data gaps
• Write latency monitoring catching compaction storms or GC pauses that drive PUT latency above acceptable thresholds
• Query endpoint liveness verifying that metric reads are available, not just that the HTTP port is open
• End-to-end ingestion heartbeats confirming your pipeline is successfully storing and retrieving time-series data

These layers matter because OpenTSDB failures are often partial: the TSD HTTP API may respond to version requests while all write operations are queued and timing out due to an HBase region server failure in a specific key range.

Step 1: Probe OpenTSDB's Built-in HTTP Endpoints

OpenTSDB exposes HTTP endpoints for version and API liveness:

# Check OpenTSDB version and liveness
curl -s http://opentsdb.example.com:4242/version

# Expected response:
# {"short_revision":"...","version":"2.4.x","timestamp":...}

# Check API liveness
curl -s http://opentsdb.example.com:4242/api/version

# Check configuration (confirms HBase ZooKeeper quorum is configured)
curl -s http://opentsdb.example.com:4242/api/config

Point a Vigilmon monitor directly at http://opentsdb.example.com:4242/api/version for immediate TSD availability alerting.

Test a simple metric query to confirm read availability:

# Query the last 1 hour of a known metric (replace with your metric name)
curl -s "http://opentsdb.example.com:4242/api/query?start=1h-ago&m=sum:sys.cpu.user%7Bhost%3Dwildcard%28*%29%7D"

# If OpenTSDB can read from HBase, it returns an array (possibly empty for new metrics)

Step 2: Build an OpenTSDB Health Sidecar

For comprehensive health checks — write latency, HBase connectivity, and query response time — build a sidecar that exercises OpenTSDB's HTTP API.

Python Health Sidecar

# opentsdb_health.py
from flask import Flask, jsonify
import requests
import time
import os

app = Flask(__name__)

OPENTSDB_URL       = os.environ.get('OPENTSDB_URL',       'http://localhost:4242')
WRITE_LATENCY_MS   = int(os.environ.get('WRITE_LATENCY_MS', '2000'))
PROBE_METRIC       = os.environ.get('PROBE_METRIC',       'vigilmon.probe.write')

@app.route('/health/opentsdb')
def opentsdb_health():
    try:
        r = requests.get(f'{OPENTSDB_URL}/api/version', timeout=5)
        if r.status_code != 200:
            return jsonify({'status': 'down', 'error': f'http_{r.status_code}'}), 503
    except Exception as e:
        return jsonify({'status': 'down', 'error': str(e)}), 503

    return jsonify({'status': 'ok', 'version': r.json().get('version', 'unknown')}), 200

@app.route('/health/opentsdb/write')
def opentsdb_write():
    """Test a write to OpenTSDB and measure latency."""
    ts   = int(time.time())
    data = [{'metric': PROBE_METRIC, 'timestamp': ts, 'value': 1, 'tags': {'source': 'vigilmon'}}]
    try:
        start = time.time()
        r = requests.post(f'{OPENTSDB_URL}/api/put?summary', json=data, timeout=10)
        elapsed_ms = int((time.time() - start) * 1000)
        if r.status_code not in (200, 204):
            return jsonify({'status': 'down', 'error': f'write_http_{r.status_code}', 'body': r.text}), 503
    except Exception as e:
        return jsonify({'status': 'down', 'error': str(e)}), 503

    if elapsed_ms > WRITE_LATENCY_MS:
        return jsonify({'status': 'degraded', 'write_latency_ms': elapsed_ms,
                        'threshold_ms': WRITE_LATENCY_MS}), 200

    return jsonify({'status': 'ok', 'write_latency_ms': elapsed_ms}), 200

@app.route('/health/opentsdb/query')
def opentsdb_query():
    """Test that OpenTSDB can execute a query (confirms HBase read path)."""
    params = {
        'start':  '1h-ago',
        'm':      f'sum:{PROBE_METRIC}{{source=vigilmon}}'
    }
    try:
        start = time.time()
        r = requests.get(f'{OPENTSDB_URL}/api/query', params=params, timeout=15)
        elapsed_ms = int((time.time() - start) * 1000)
        if r.status_code not in (200, 400):
            # 400 = no data found — still means the query path works
            return jsonify({'status': 'down', 'error': f'query_http_{r.status_code}'}), 503
    except Exception as e:
        return jsonify({'status': 'down', 'error': str(e)}), 503

    return jsonify({'status': 'ok', 'query_latency_ms': elapsed_ms}), 200

if __name__ == '__main__':
    app.run(port=8096)

Node.js Health Sidecar

// opentsdb-health.js
const express = require('express');
const axios   = require('axios');

const app = express();

const OPENTSDB_URL     = process.env.OPENTSDB_URL     || 'http://localhost:4242';
const WRITE_LATENCY_MS = parseInt(process.env.WRITE_LATENCY_MS || '2000', 10);
const PROBE_METRIC     = process.env.PROBE_METRIC     || 'vigilmon.probe.write';

app.get('/health/opentsdb', async (req, res) => {
  try {
    const r = await axios.get(`${OPENTSDB_URL}/api/version`, { timeout: 5000 });
    return res.status(200).json({ status: 'ok', version: r.data?.version });
  } catch (e) {
    return res.status(503).json({ status: 'down', error: e.message });
  }
});

app.get('/health/opentsdb/write', async (req, res) => {
  const ts      = Math.floor(Date.now() / 1000);
  const payload = [{ metric: PROBE_METRIC, timestamp: ts, value: 1, tags: { source: 'vigilmon' } }];
  const start   = Date.now();
  try {
    const r = await axios.post(`${OPENTSDB_URL}/api/put?summary`, payload, { timeout: 10000 });
    const elapsed = Date.now() - start;
    if (![200, 204].includes(r.status)) {
      return res.status(503).json({ status: 'down', error: `write_http_${r.status}` });
    }
    if (elapsed > WRITE_LATENCY_MS) {
      return res.status(200).json({ status: 'degraded', write_latency_ms: elapsed });
    }
    return res.status(200).json({ status: 'ok', write_latency_ms: elapsed });
  } catch (e) {
    return res.status(503).json({ status: 'down', error: e.message });
  }
});

app.listen(3012, () => console.log('opentsdb-health listening on 3012'));

Step 3: Configure Vigilmon HTTP Monitor for OpenTSDB

1. Log in to vigilmon.online and go to Monitors → New Monitor
2. Choose HTTP / HTTPS
3. Set the URL to http://opentsdb.example.com:4242/api/version
4. Set the check interval to 1 minute
5. Under Expected response, configure:
   • Status code: 200
   • Response body contains: "version"
   • Response time threshold: 5000ms
6. Under Alert channels, assign your infrastructure on-call channel
7. Save the monitor

Add a write health sidecar monitor:

• URL: https://your-app.example.com/health/opentsdb/write
• Expected: 200, body contains "status":"ok"
• Interval: 2 minutes
• Alert channel: metrics infrastructure Slack channel

Add a query path monitor:

• URL: https://your-app.example.com/health/opentsdb/query
• Expected: 200
• Interval: 5 minutes
• Alert channel: monitoring infrastructure channel

If you run multiple OpenTSDB TSD instances behind a load balancer, add a monitor per TSD node to detect single-node failures that the load balancer might mask from clients.

Step 4: Heartbeat Monitoring for OpenTSDB Ingestion Pipelines

Write latency probes confirm the TSD can accept individual data points — but they don't verify that your full ingestion pipeline (collector → OpenTSDB → query) is functioning end-to-end. A TSD can accept individual test writes while your bulk metrics collector has stalled due to a connection pool exhaustion.

Vigilmon heartbeat monitors detect these failures: your ingestion pipeline pings Vigilmon after successfully completing a full write-and-verify cycle.

Set Up the Heartbeat Monitor

1. In Vigilmon, go to Monitors → New Monitor → Heartbeat
2. Set the name: opentsdb-ingestion-pipeline
3. Set the expected interval: 5 minutes
4. Set the grace period: 15 minutes
5. Save — copy the heartbeat URL, e.g. https://vigilmon.online/heartbeat/abc123xyz

Wire It Into an OpenTSDB Ingestion Probe

# opentsdb_ingestion_probe.py — write a test metric and ping heartbeat on success
import requests
import time
import os

OPENTSDB_URL = os.environ['OPENTSDB_URL']
VIGILMON_HB  = os.environ['VIGILMON_HEARTBEAT_URL']
PROBE_METRIC = 'vigilmon.ingestion.probe'

ts   = int(time.time())
data = [{'metric': PROBE_METRIC, 'timestamp': ts, 'value': 1.0, 'tags': {'source': 'heartbeat'}}]

# Write the test metric
write_r = requests.post(f'{OPENTSDB_URL}/api/put?summary', json=data, timeout=10)
if write_r.status_code not in (200, 204):
    raise RuntimeError(f'OpenTSDB write failed: {write_r.status_code} {write_r.text}')

print(f'Test metric written at ts={ts}.')

# Brief wait for HBase write to propagate (adjust based on your HBase latency)
time.sleep(2)

# Verify the metric is queryable
query_r = requests.get(
    f'{OPENTSDB_URL}/api/query',
    params={'start': f'{ts}', 'end': f'{ts + 60}', 'm': f'sum:{PROBE_METRIC}{{source=heartbeat}}'},
    timeout=15
)
if query_r.status_code not in (200, 400):
    raise RuntimeError(f'OpenTSDB query check failed: {query_r.status_code}')

print('Query path verified.')

# Send heartbeat
requests.get(VIGILMON_HB, timeout=10)
print('Vigilmon heartbeat sent.')

Run this probe every 5 minutes via cron or a scheduler to confirm the full read/write path through OpenTSDB and HBase.

Step 5: Alert Routing for OpenTSDB Failures

Configure alert routing in Vigilmon:

| Monitor | Alert Channel | Priority | |---|---|---| | OpenTSDB /api/version | Slack + PagerDuty | P1 | | Write latency sidecar | Slack + PagerDuty | P1 | | Query path sidecar | Slack | P2 | | Heartbeat: ingestion pipeline | Slack + PagerDuty | P1 | | Per-TSD node availability | Slack | P2 |

Set response time thresholds for early warning:

• Alert at 3000ms for the /api/version endpoint (slow version responses precede full TSD unavailability)
• Alert at 5000ms for write latency probes (elevated write latency indicates HBase region server pressure)

For production monitoring systems where OpenTSDB backs dashboards or automated alerting, configure a tight grace period on heartbeats: an ingestion pipeline that runs every 5 minutes should alert within 20 minutes of silence, not hours later when dashboards show data gaps.

Summary

OpenTSDB failures span TSD process availability, HBase write path health, and end-to-end ingestion. External monitoring catches all three:

| Monitor Type | What It Covers | |---|---| | HTTP monitor on /api/version | TSD process liveness | | HTTP monitor on write latency sidecar | HBase write path and latency | | HTTP monitor on query sidecar | HBase read path availability | | Heartbeat monitor on ingestion probe | End-to-end write-and-verify cycle |

Get started free at vigilmon.online — your first OpenTSDB availability monitor is running in under two minutes.