tutorial

How to Monitor KairosDB Availability and Cassandra Backend Health with Vigilmon

KairosDB Cassandra connection failures, query timeout storms, and datastore write rejections silently degrade metric ingestion. Learn how to monitor KairosDB availability, REST API health, and end-to-end write pipeline status with Vigilmon.

KairosDB is a distributed time-series database built on Apache Cassandra, designed for high-throughput metric ingestion with a REST and Telnet API — but when the Cassandra cluster loses a node and KairosDB's connection pool drains, a write backlog exceeds the internal queue limit and incoming data points are silently dropped, or the query timeout for a wide time range triggers a circuit breaker, there is no built-in outbound alerting. KairosDB exposes a /api/v1/health/status endpoint and a /api/v1/version endpoint, but these require active polling and give no push notifications when the Cassandra backend degrades.

Vigilmon gives you external visibility into KairosDB through HTTP monitors on its health and version endpoints, a sidecar that exercises the write and query API, and heartbeat monitors that confirm your metrics ingestion pipeline is successfully storing and retrieving data. This tutorial covers both.

Why KairosDB Needs External Monitoring

KairosDB's built-in tooling (the web UI, JVM heap metrics, and Cassandra nodetool status) requires a human operator or internal alerting stack actively checking those outputs. External monitoring with Vigilmon adds:

  • KairosDB process availability alerting when the JVM process becomes unreachable before your pipeline stalls
  • Cassandra backend health checks detecting connection pool exhaustion or write rejection before data loss accumulates
  • REST API liveness verification confirming the HTTP API serves requests — not just that the TCP port accepts connections
  • Write latency monitoring catching Cassandra GC pauses or compaction pressure before timeouts propagate to clients
  • End-to-end ingestion heartbeats confirming your metrics pipeline writes to KairosDB and reads back successfully

These layers matter because KairosDB failures are often gradual: the health endpoint may return healthy while the Cassandra connection pool is saturated and write latency climbs from 50ms to 5000ms, causing client timeouts long before the health endpoint reflects the problem.

Step 1: Probe KairosDB's Built-in Health Endpoints

KairosDB exposes REST endpoints for version and health status:

# Check KairosDB version
curl -s http://kairosdb.example.com:8080/api/v1/version

# Expected response:
# {"version":"KairosDB 1.3.0"}

# Check health status (checks Cassandra datastore connectivity)
curl -s http://kairosdb.example.com:8080/api/v1/health/status

# Expected response:
# ["JVM-Thread-Deadlock: OK","Datastore-Query: OK"]

# Check health check details
curl -s http://kairosdb.example.com:8080/api/v1/health/check

Point a Vigilmon monitor directly at http://kairosdb.example.com:8080/api/v1/health/status for immediate availability alerting. Configure it to check for Datastore-Query: OK in the response body.

Test the metric names list endpoint to confirm the data path is working:

# List known metric names (confirms Cassandra read path is operational)
curl -s http://kairosdb.example.com:8080/api/v1/metricnames

# Expected: {"results":["..."]}

Step 2: Build a KairosDB Health Sidecar

For comprehensive health checks — write latency, Cassandra query path, and API response time — build a sidecar that exercises KairosDB's REST API end-to-end.

Python Health Sidecar

# kairosdb_health.py
from flask import Flask, jsonify
import requests
import time
import os

app = Flask(__name__)

KAIROSDB_URL      = os.environ.get('KAIROSDB_URL',      'http://localhost:8080')
WRITE_LATENCY_MS  = int(os.environ.get('WRITE_LATENCY_MS', '2000'))
PROBE_METRIC      = os.environ.get('PROBE_METRIC',      'vigilmon.probe.write')

@app.route('/health/kairosdb')
def kairosdb_health():
    try:
        r = requests.get(f'{KAIROSDB_URL}/api/v1/health/status', timeout=5)
        if r.status_code != 204 and r.status_code != 200:
            return jsonify({'status': 'down', 'error': f'http_{r.status_code}'}), 503
        # KairosDB returns 204 No Content when all checks pass
        if r.status_code == 200:
            checks = r.json() if r.text else []
            failed = [c for c in checks if 'OK' not in c]
            if failed:
                return jsonify({'status': 'down', 'failed_checks': failed}), 503
    except Exception as e:
        return jsonify({'status': 'down', 'error': str(e)}), 503

    return jsonify({'status': 'ok'}), 200

@app.route('/health/kairosdb/write')
def kairosdb_write():
    """Test a datapoint write to KairosDB and measure latency."""
    ts_ms = int(time.time() * 1000)
    payload = {
        'name': PROBE_METRIC,
        'datapoints': [[ts_ms, 1]],
        'tags': {'source': 'vigilmon_probe'}
    }
    try:
        start = time.time()
        r = requests.post(
            f'{KAIROSDB_URL}/api/v1/datapoints',
            json=[payload],
            timeout=10
        )
        elapsed_ms = int((time.time() - start) * 1000)
        if r.status_code not in (200, 204):
            return jsonify({'status': 'down', 'error': f'write_http_{r.status_code}',
                            'body': r.text[:200]}), 503
    except Exception as e:
        return jsonify({'status': 'down', 'error': str(e)}), 503

    if elapsed_ms > WRITE_LATENCY_MS:
        return jsonify({'status': 'degraded', 'write_latency_ms': elapsed_ms,
                        'threshold_ms': WRITE_LATENCY_MS}), 200
    return jsonify({'status': 'ok', 'write_latency_ms': elapsed_ms}), 200

@app.route('/health/kairosdb/query')
def kairosdb_query():
    """Test a metric query to confirm the Cassandra read path is available."""
    ts_end   = int(time.time() * 1000)
    ts_start = ts_end - 300_000  # last 5 minutes
    payload  = {
        'start_absolute': ts_start,
        'end_absolute':   ts_end,
        'metrics': [{'name': PROBE_METRIC, 'tags': {'source': ['vigilmon_probe']}}]
    }
    try:
        start = time.time()
        r = requests.post(
            f'{KAIROSDB_URL}/api/v1/datapoints/query',
            json=payload,
            timeout=15
        )
        elapsed_ms = int((time.time() - start) * 1000)
        if r.status_code != 200:
            return jsonify({'status': 'down', 'error': f'query_http_{r.status_code}'}), 503
    except Exception as e:
        return jsonify({'status': 'down', 'error': str(e)}), 503

    return jsonify({'status': 'ok', 'query_latency_ms': elapsed_ms}), 200

@app.route('/health/kairosdb/metricnames')
def kairosdb_metricnames():
    """Check the metric names endpoint to confirm Cassandra catalog access."""
    try:
        r = requests.get(f'{KAIROSDB_URL}/api/v1/metricnames', timeout=10)
        if r.status_code != 200:
            return jsonify({'status': 'down', 'error': f'http_{r.status_code}'}), 503
        results = r.json().get('results', [])
    except Exception as e:
        return jsonify({'status': 'down', 'error': str(e)}), 503

    return jsonify({'status': 'ok', 'metric_count': len(results)}), 200

if __name__ == '__main__':
    app.run(port=8098)

Node.js Health Sidecar

// kairosdb-health.js
const express = require('express');
const axios   = require('axios');

const app = express();

const KAIROSDB_URL     = process.env.KAIROSDB_URL     || 'http://localhost:8080';
const WRITE_LATENCY_MS = parseInt(process.env.WRITE_LATENCY_MS || '2000', 10);
const PROBE_METRIC     = process.env.PROBE_METRIC     || 'vigilmon.probe.write';

app.get('/health/kairosdb', async (req, res) => {
  try {
    const r = await axios.get(`${KAIROSDB_URL}/api/v1/health/status`, {
      timeout: 5000,
      validateStatus: s => s === 204 || s === 200
    });
    if (r.status === 200 && Array.isArray(r.data)) {
      const failed = r.data.filter(c => !c.includes('OK'));
      if (failed.length) return res.status(503).json({ status: 'down', failed_checks: failed });
    }
    return res.status(200).json({ status: 'ok' });
  } catch (e) {
    return res.status(503).json({ status: 'down', error: e.message });
  }
});

app.get('/health/kairosdb/write', async (req, res) => {
  const ts      = Date.now();
  const payload = [{ name: PROBE_METRIC, datapoints: [[ts, 1]], tags: { source: 'vigilmon_probe' } }];
  const start   = Date.now();
  try {
    const r = await axios.post(`${KAIROSDB_URL}/api/v1/datapoints`, payload, {
      timeout: 10000,
      validateStatus: s => s === 200 || s === 204
    });
    const elapsed = Date.now() - start;
    if (elapsed > WRITE_LATENCY_MS) {
      return res.status(200).json({ status: 'degraded', write_latency_ms: elapsed });
    }
    return res.status(200).json({ status: 'ok', write_latency_ms: elapsed });
  } catch (e) {
    return res.status(503).json({ status: 'down', error: e.message });
  }
});

app.listen(3015, () => console.log('kairosdb-health listening on 3015'));

Step 3: Configure Vigilmon HTTP Monitor for KairosDB

  1. Log in to vigilmon.online and go to Monitors → New Monitor
  2. Choose HTTP / HTTPS
  3. Set the URL to http://kairosdb.example.com:8080/api/v1/health/status
  4. Set the check interval to 1 minute
  5. Under Expected response, configure:
    • Status code: 204 (KairosDB returns 204 when all checks pass; 500 when any fail)
    • Response time threshold: 3000ms
  6. Under Alert channels, assign your metrics infrastructure on-call channel
  7. Save the monitor

Add a write latency sidecar monitor:

  • URL: https://your-app.example.com/health/kairosdb/write
  • Expected: 200, body contains "status":"ok"
  • Interval: 2 minutes
  • Alert channel: metrics pipeline Slack channel

Add a query path monitor:

  • URL: https://your-app.example.com/health/kairosdb/query
  • Expected: 200
  • Interval: 5 minutes
  • Alert channel: monitoring infrastructure channel

Add a metric catalog monitor:

  • URL: https://your-app.example.com/health/kairosdb/metricnames
  • Expected: 200
  • Interval: 5 minutes
  • Alert channel: database operations channel

If you run multiple KairosDB instances behind a load balancer, add per-instance health monitors so a single-node JVM crash is detected independently of the load balancer health check.

Step 4: Heartbeat Monitoring for KairosDB Ingestion Pipelines

Write latency probes confirm the REST API can accept individual data points — but they don't verify that your full metrics collection pipeline (agent → KairosDB → dashboard query) is functioning end-to-end. A KairosDB node can accept individual probe writes while your bulk metrics agent has stalled due to a Cassandra token ring inconsistency or a misconfigured retention TTL silently expiring data.

Vigilmon heartbeat monitors detect these failures: your ingestion pipeline pings Vigilmon after successfully completing a full write-and-query cycle.

Set Up the Heartbeat Monitor

  1. In Vigilmon, go to Monitors → New Monitor → Heartbeat
  2. Set the name: kairosdb-ingestion-pipeline
  3. Set the expected interval: 5 minutes
  4. Set the grace period: 15 minutes
  5. Save — copy the heartbeat URL, e.g. https://vigilmon.online/heartbeat/abc123xyz

Wire It Into a KairosDB Ingestion Probe

# kairosdb_ingestion_probe.py — write a test metric and ping heartbeat on success
import requests
import time
import os

KAIROSDB_URL = os.environ['KAIROSDB_URL']
VIGILMON_HB  = os.environ['VIGILMON_HEARTBEAT_URL']
PROBE_METRIC = 'vigilmon.ingestion.probe'

ts_ms = int(time.time() * 1000)
payload = [{'name': PROBE_METRIC, 'datapoints': [[ts_ms, 1.0]],
            'tags': {'source': 'heartbeat'}}]

# Write the test metric
write_r = requests.post(f'{KAIROSDB_URL}/api/v1/datapoints', json=payload, timeout=10)
if write_r.status_code not in (200, 204):
    raise RuntimeError(f'KairosDB write failed: {write_r.status_code} {write_r.text}')

print(f'Test metric written at ts={ts_ms}ms.')

# Brief wait for Cassandra write to be readable
time.sleep(3)

# Query back the data point to verify the read path
query_payload = {
    'start_absolute': ts_ms - 1000,
    'end_absolute':   ts_ms + 60_000,
    'metrics': [{'name': PROBE_METRIC, 'tags': {'source': ['heartbeat']}}]
}
query_r = requests.post(
    f'{KAIROSDB_URL}/api/v1/datapoints/query',
    json=query_payload,
    timeout=15
)
if query_r.status_code != 200:
    raise RuntimeError(f'KairosDB query failed: {query_r.status_code} {query_r.text}')

queries = query_r.json().get('queries', [])
data_points = queries[0]['results'][0]['values'] if queries else []
if not data_points:
    raise RuntimeError('Probe datapoint not found in KairosDB query result.')

print(f'Probe datapoint verified ({len(data_points)} point(s) returned).')

# Send heartbeat to Vigilmon
requests.get(VIGILMON_HB, timeout=10)
print('Vigilmon heartbeat sent.')

Step 5: Alert Routing for KairosDB Failures

Configure alert routing in Vigilmon:

| Monitor | Alert Channel | Priority | |---|---|---| | KairosDB /api/v1/health/status | Slack + PagerDuty | P1 | | Write latency sidecar | Slack + PagerDuty | P1 | | Query path sidecar | Slack | P2 | | Metric catalog sidecar | Slack | P3 | | Heartbeat: ingestion pipeline | Slack + PagerDuty | P1 |

Set response time thresholds for early warning:

  • Alert at 2000ms for the health status endpoint (slow health checks precede Cassandra connectivity failures)
  • Alert at 3000ms for write latency probes (elevated write latency signals Cassandra compaction pressure)
  • Alert at 10000ms for query path probes (slow queries indicate Cassandra read repair overhead or wide partition scans)

For production monitoring systems where KairosDB backs alerting pipelines or capacity dashboards, configure a tight grace period on heartbeats: a pipeline that runs every 5 minutes should alert within 20 minutes of silence, not on the next morning's on-call handover.

Summary

KairosDB failures span JVM process availability, Cassandra backend connectivity, write path latency, and end-to-end ingestion. External monitoring catches all four:

| Monitor Type | What It Covers | |---|---| | HTTP monitor on /api/v1/health/status | KairosDB and Cassandra datastore health | | HTTP monitor on write sidecar | Cassandra write path latency | | HTTP monitor on query sidecar | Cassandra read path availability | | HTTP monitor on metricnames sidecar | Cassandra catalog (keyspace) access | | Heartbeat monitor on ingestion probe | End-to-end write-and-query cycle |

Get started free at vigilmon.online — your first KairosDB availability monitor is running in under two minutes.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles