tutorial

How to Monitor M3DB Cluster Availability and Replication Health with Vigilmon

M3DB replication failures, coordinator timeouts, and placement imbalances silently degrade time-series write durability. Learn how to monitor M3DB cluster availability, replication factor health, and query coordinator uptime with Vigilmon.

M3DB is Uber's distributed time-series database built for petabyte-scale metrics storage with configurable replication and namespace isolation — but when a placement node goes out of rotation, the coordinator process loses quorum on a shard, or a bootstrap phase stalls after a rolling restart, writes silently degrade without triggering any built-in outbound alert. M3DB exposes a rich set of metrics and health endpoints, but they require active polling and give no push notifications when replication drops below the configured factor.

Vigilmon gives you external visibility into M3DB through HTTP monitors on its coordinator health API and a sidecar that interprets node placement and replication state, plus heartbeat monitors that confirm your metrics ingestion pipeline is delivering data end-to-end. This tutorial covers both.

Why M3DB Needs External Monitoring

M3DB's built-in tooling (the m3coordinator Prometheus endpoint, placement API, and namespace API) requires a human operator or internal alerting stack actively polling those endpoints. External monitoring with Vigilmon adds:

  • Coordinator availability alerting when the m3coordinator process becomes unreachable before your metrics pipeline stalls
  • Replication health checks detecting under-replicated shards or placement nodes that failed to bootstrap
  • Namespace availability monitoring confirming that your time-series namespaces are accepting writes
  • Response time monitoring catching coordinator degradation from shard bootstrap overhead or GC pressure
  • End-to-end ingestion heartbeats confirming your metrics pipeline writes to M3DB and reads back successfully

These layers matter because M3DB failures are often partial: the coordinator HTTP port may respond while individual shards are in BOOTSTRAPPING state, causing write timeouts only for specific series without any obvious top-level error.

Step 1: Probe M3DB's Built-in Health Endpoints

M3DB's coordinator exposes health endpoints you can probe directly:

# Check m3coordinator health
curl -s http://m3db.example.com:7201/health

# Expected response:
# {"ok":true,"status":"ok","uptime":"..."}

# Check placement state
curl -s http://m3db.example.com:7201/api/v1/placement

# Check namespace configuration
curl -s http://m3db.example.com:7201/api/v1/namespace

Point a Vigilmon monitor directly at http://m3db.example.com:7201/health for immediate coordinator availability alerting.

Check individual dbnode health (port 9000 by default):

# dbnode health endpoint
curl -s http://m3db-node1.example.com:9000/health

# Expected response includes bootstrapped state:
# {"ok":true,"status":"ok","bootstrapped":true}

Step 2: Build an M3DB Health Sidecar

For comprehensive health checks — replication factor compliance, shard bootstrap state, and namespace availability — build a sidecar that interprets M3DB's placement API.

Python Health Sidecar

# m3db_health.py
from flask import Flask, jsonify
import requests
import os

app = Flask(__name__)

M3DB_COORDINATOR = os.environ.get('M3DB_COORDINATOR', 'http://localhost:7201')
MIN_REPLICAS     = int(os.environ.get('MIN_REPLICAS', '3'))

@app.route('/health/m3db')
def m3db_health():
    try:
        r = requests.get(f'{M3DB_COORDINATOR}/health', timeout=5)
        if r.status_code != 200:
            return jsonify({'status': 'down', 'error': f'http_{r.status_code}'}), 503
        body = r.json()
        if not body.get('ok'):
            return jsonify({'status': 'down', 'error': 'coordinator_not_ok', 'detail': body}), 503
    except Exception as e:
        return jsonify({'status': 'down', 'error': str(e)}), 503

    return jsonify({'status': 'ok'}), 200

@app.route('/health/m3db/placement')
def m3db_placement():
    errors   = []
    warnings = []
    try:
        r = requests.get(f'{M3DB_COORDINATOR}/api/v1/placement', timeout=10)
        r.raise_for_status()
        placement = r.json().get('placement', {})
    except Exception as e:
        return jsonify({'status': 'down', 'error': str(e)}), 503

    instances = placement.get('instances', {})
    total     = len(instances)
    available = sum(
        1 for inst in instances.values()
        if inst.get('isLeaving') is False and inst.get('isInitializing') is False
    )

    if available < MIN_REPLICAS:
        errors.append(f'only_{available}_of_{total}_instances_available')

    bootstrapping = [
        iid for iid, inst in instances.items()
        if inst.get('isInitializing')
    ]
    if bootstrapping:
        warnings.append(f'bootstrapping_instances: {bootstrapping}')

    if errors:
        return jsonify({'status': 'down', 'errors': errors, 'warnings': warnings}), 503
    if warnings:
        return jsonify({'status': 'degraded', 'warnings': warnings, 'available': available}), 200
    return jsonify({'status': 'ok', 'instances': total, 'available': available}), 200

@app.route('/health/m3db/namespaces')
def m3db_namespaces():
    try:
        r = requests.get(f'{M3DB_COORDINATOR}/api/v1/namespace', timeout=10)
        r.raise_for_status()
        ns = r.json().get('registry', {}).get('namespaces', {})
    except Exception as e:
        return jsonify({'status': 'down', 'error': str(e)}), 503

    if not ns:
        return jsonify({'status': 'down', 'error': 'no_namespaces_configured'}), 503
    return jsonify({'status': 'ok', 'namespace_count': len(ns)}), 200

if __name__ == '__main__':
    app.run(port=8095)

Node.js Health Sidecar

// m3db-health.js
const express = require('express');
const axios   = require('axios');

const app = express();

const M3DB_COORDINATOR = process.env.M3DB_COORDINATOR || 'http://localhost:7201';
const MIN_REPLICAS     = parseInt(process.env.MIN_REPLICAS || '3', 10);

app.get('/health/m3db', async (req, res) => {
  try {
    const r = await axios.get(`${M3DB_COORDINATOR}/health`, { timeout: 5000 });
    if (!r.data?.ok) return res.status(503).json({ status: 'down', error: 'coordinator_not_ok' });
    return res.status(200).json({ status: 'ok' });
  } catch (e) {
    return res.status(503).json({ status: 'down', error: e.message });
  }
});

app.get('/health/m3db/placement', async (req, res) => {
  let instances;
  try {
    const r = await axios.get(`${M3DB_COORDINATOR}/api/v1/placement`, { timeout: 10000 });
    instances = r.data?.placement?.instances || {};
  } catch (e) {
    return res.status(503).json({ status: 'down', error: e.message });
  }

  const total     = Object.keys(instances).length;
  const available = Object.values(instances).filter(
    i => i.isLeaving === false && i.isInitializing === false
  ).length;
  const bootstrapping = Object.keys(instances).filter(id => instances[id].isInitializing);

  if (available < MIN_REPLICAS) {
    return res.status(503).json({ status: 'down', error: `only_${available}_of_${total}_available` });
  }
  if (bootstrapping.length) {
    return res.status(200).json({ status: 'degraded', bootstrapping, available });
  }
  return res.status(200).json({ status: 'ok', instances: total, available });
});

app.listen(3011, () => console.log('m3db-health listening on 3011'));

Step 3: Configure Vigilmon HTTP Monitor for M3DB

  1. Log in to vigilmon.online and go to Monitors → New Monitor
  2. Choose HTTP / HTTPS
  3. Set the URL to http://m3db.example.com:7201/health
  4. Set the check interval to 1 minute
  5. Under Expected response, configure:
    • Status code: 200
    • Response body contains: "ok":true
    • Response time threshold: 3000ms
  6. Under Alert channels, assign your infrastructure on-call channel
  7. Save the monitor

Add a second monitor for the placement health sidecar:

  • URL: https://your-app.example.com/health/m3db/placement
  • Expected: 200, body contains "status":"ok"
  • Interval: 2 minutes
  • Alert channel: database operations channel

Add a namespace availability monitor:

  • URL: https://your-app.example.com/health/m3db/namespaces
  • Expected: 200
  • Interval: 5 minutes
  • Alert channel: metrics infrastructure Slack channel

If you run M3DB in multiple clusters (short-term retention cluster, long-term retention cluster), create a separate monitor set for each with descriptive labels.

Step 4: Heartbeat Monitoring for M3DB Ingestion Pipelines

Placement health checks confirm the cluster is configured correctly — but they don't verify that your metrics pipeline is successfully writing time-series data and receiving reads. A coordinator can be healthy while a misconfigured namespace retention policy silently drops all incoming writes.

Vigilmon heartbeat monitors detect these failures: your ingestion pipeline pings Vigilmon after successfully writing and reading back a test metric.

Set Up the Heartbeat Monitor

  1. In Vigilmon, go to Monitors → New Monitor → Heartbeat
  2. Set the name: m3db-ingestion-pipeline
  3. Set the expected interval: 5 minutes (or your ingestion schedule)
  4. Set the grace period: 10 minutes
  5. Save — copy the heartbeat URL, e.g. https://vigilmon.online/heartbeat/abc123xyz

Wire It Into an M3DB Ingestion Probe

# m3db_ingestion_probe.py — write a test metric to M3DB and ping heartbeat on success
import requests
import time
import os

M3DB_COORDINATOR    = os.environ['M3DB_COORDINATOR']
VIGILMON_HB         = os.environ['VIGILMON_HEARTBEAT_URL']
TEST_METRIC_NAME    = 'vigilmon.ingestion.probe'
TEST_NAMESPACE      = os.environ.get('M3DB_NAMESPACE', 'default_unaggregated')

# Write a test metric via the M3DB Prometheus remote_write compatible endpoint
# or via the native M3DB write endpoint
ts = int(time.time() * 1e9)  # nanosecond timestamp
payload = {
    'namespace': TEST_NAMESPACE,
    'id': TEST_METRIC_NAME,
    'datapoint': {
        'timestamp': ts,
        'value': 1.0
    }
}

write_r = requests.post(
    f'{M3DB_COORDINATOR}/api/v1/writeuttagged',
    json={'namespace': TEST_NAMESPACE, 'id': TEST_METRIC_NAME,
          'tags': [{'name': 'source', 'value': 'vigilmon_probe'}],
          'datapoint': {'timestamp': ts, 'value': 1.0}},
    timeout=10
)

if write_r.status_code not in (200, 204):
    raise RuntimeError(f'M3DB write failed: {write_r.status_code} {write_r.text}')

print('Test metric written to M3DB.')

# Ping Vigilmon heartbeat
requests.get(VIGILMON_HB, timeout=10)
print('Vigilmon heartbeat sent.')

Run this probe on a schedule (cron every 5 minutes) to confirm end-to-end write availability.

Step 5: Alert Routing for M3DB Failures

Configure alert routing in Vigilmon:

| Monitor | Alert Channel | Priority | |---|---|---| | M3DB coordinator /health | Slack + PagerDuty | P1 | | Placement health sidecar | Slack + PagerDuty | P1 | | Namespace availability | Slack | P2 | | Heartbeat: ingestion pipeline | Slack + PagerDuty | P1 | | Per-dbnode health checks | Slack | P2 |

Set response time thresholds for early warning:

  • Alert at 2000ms for the coordinator health endpoint (slow health checks precede shard assignment failures)
  • Alert at 5000ms for the placement sidecar (slow placement API responses signal coordinator overload)

For production metrics pipelines where M3DB backs dashboards or SLO calculations, configure a tight grace period on heartbeats: an ingestion probe that runs every 5 minutes should alert within 15 minutes of silence, not hours later when dashboards show gaps.

Summary

M3DB failures span coordinator availability, replication placement health, and end-to-end ingestion. External monitoring catches all three:

| Monitor Type | What It Covers | |---|---| | HTTP monitor on /health | M3DB coordinator process liveness | | HTTP monitor on placement sidecar | Replication factor and bootstrap state | | HTTP monitor on namespace sidecar | Namespace configuration availability | | Heartbeat monitor on ingestion probe | End-to-end write success |

Get started free at vigilmon.online — your first M3DB availability monitor is running in under two minutes.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles