tutorial

Monitoring YugabyteDB: Tablet Health, Query Latency, and External Checks with Vigilmon

A practical guide to monitoring YugabyteDB distributed SQL — tablet server health, YSQL/YCQL query latency, replication lag, master leader availability, and Vigilmon external health checks.

YugabyteDB's distributed architecture means failures can be subtle: a tablet peer falls behind on replication while YSQL queries still succeed on the surviving majority, a master leader election stalls, or a YCQL connection pool silently drains. Internal observability catches most of this — but external HTTP monitoring catches the failures that internal metrics miss: a node that's up but not accepting connections, a load balancer routing to a dead peer, or a health endpoint returning 200 while the underlying tablet is in a crash loop.

Vigilmon provides external HTTP and heartbeat monitoring that complements YugabyteDB's built-in metrics. This tutorial covers tablet health, query latency exposure, replication lag, and how to wire Vigilmon into a YugabyteDB deployment.

YugabyteDB Architecture Recap

Before setting up monitors, it's useful to know what can fail and where:

  • YB-Master — manages cluster metadata, tablet placement, and leader elections. Each cluster runs 3 or 5 masters; one is the leader.
  • YB-TServer — handles all data reads and writes. Each tablet has 3 peers across TServers; one peer is the Raft leader.
  • YSQL — PostgreSQL-compatible SQL layer, running on port 5433 by default.
  • YCQL — Cassandra-compatible CQL layer, running on port 9042.

Failures propagate bottom-up: a dead TServer causes tablet under-replication; enough under-replicated tablets trigger write failures.

Step 1: Expose a Health Endpoint

YugabyteDB's TServer has a built-in HTTP UI on port 9000. The /healthz endpoint returns true if the server is up. Use it directly or wrap it in an API health check.

Direct TServer Health Check

# Returns "true" if the TServer is healthy
curl http://yb-tserver-1:9000/healthz

For Vigilmon, you need an endpoint reachable from outside your cluster. If your TServer UI is not internet-accessible (it shouldn't be), proxy it through your application:

# FastAPI — proxy YugabyteDB health into your app
import httpx
from fastapi import FastAPI
from fastapi.responses import JSONResponse

app = FastAPI()
YB_TSERVER_HOSTS = [
    "yb-tserver-1:9000",
    "yb-tserver-2:9000",
    "yb-tserver-3:9000",
]

@app.get("/health/yugabyte")
async def yugabyte_health():
    results = {}
    degraded = []

    async with httpx.AsyncClient(timeout=3.0) as client:
        for host in YB_TSERVER_HOSTS:
            try:
                resp = await client.get(f"http://{host}/healthz")
                healthy = resp.status_code == 200 and resp.text.strip() == "true"
                results[host] = "ok" if healthy else "degraded"
                if not healthy:
                    degraded.append(host)
            except Exception as e:
                results[host] = f"unreachable: {e}"
                degraded.append(host)

    status = "ok" if not degraded else "degraded"
    code = 200 if not degraded else 503

    return JSONResponse(status_code=code, content={
        "status": status,
        "tservers": results,
        "degraded": degraded,
    })

Master Leader Check

Master leader availability is critical — without it, schema changes and tablet rebalancing stall. YB-Master exposes its leader status on port 7000:

YB_MASTER_HOSTS = [
    "yb-master-1:7000",
    "yb-master-2:7000",
    "yb-master-3:7000",
]

@app.get("/health/yugabyte/master")
async def master_health():
    leader = None
    async with httpx.AsyncClient(timeout=3.0) as client:
        for host in YB_MASTER_HOSTS:
            try:
                resp = await client.get(f"http://{host}/api/v1/is-leader")
                if resp.status_code == 200:
                    leader = host
                    break
            except Exception:
                continue

    if leader:
        return JSONResponse(status_code=200, content={"status": "ok", "leader": leader})
    return JSONResponse(status_code=503, content={"status": "no_leader", "checked": YB_MASTER_HOSTS})

Step 2: YSQL Query Latency Probe

An availability check confirms the process is running; a latency probe confirms queries are being served within acceptable time. Add a YSQL latency endpoint:

import asyncpg
import time
import os

@app.get("/health/yugabyte/ysql")
async def ysql_latency():
    start = time.monotonic()
    try:
        # YugabyteDB YSQL uses port 5433
        conn = await asyncpg.connect(
            host=os.environ["YB_HOST"],
            port=5433,
            user=os.environ["YB_USER"],
            password=os.environ["YB_PASSWORD"],
            database=os.environ["YB_DATABASE"],
        )
        await conn.fetchval("SELECT 1")
        await conn.close()
        latency_ms = (time.monotonic() - start) * 1000

        status = "ok" if latency_ms < 200 else "slow"
        code = 200 if latency_ms < 500 else 503

        return JSONResponse(status_code=code, content={
            "status": status,
            "latency_ms": round(latency_ms, 2),
        })
    except Exception as e:
        return JSONResponse(status_code=503, content={"status": "down", "error": str(e)})

For YCQL, use the cassandra-driver Python package and probe port 9042 similarly.

Step 3: Replication Lag via YugabyteDB Metrics

YugabyteDB exposes Prometheus metrics on port 9000 (TServer) and 7000 (Master). One of the most important replication metrics is follower_lag_ms — how far behind a tablet follower is from the leader.

# Scrape follower lag from a TServer
curl -s http://yb-tserver-1:9000/prometheus-metrics \
  | grep follower_lag_ms \
  | head -20

Wrap this in a health endpoint that fails when any tablet is lagging more than your SLA threshold:

import re

LAG_THRESHOLD_MS = 5000  # 5 seconds

@app.get("/health/yugabyte/replication")
async def replication_lag():
    async with httpx.AsyncClient(timeout=5.0) as client:
        resp = await client.get("http://yb-tserver-1:9000/prometheus-metrics")

    lagging_tablets = []
    for line in resp.text.splitlines():
        m = re.match(r'follower_lag_ms\{.*tablet_id="([^"]+)".*\}\s+(\d+\.?\d*)', line)
        if m:
            tablet_id, lag_ms = m.group(1), float(m.group(2))
            if lag_ms > LAG_THRESHOLD_MS:
                lagging_tablets.append({"tablet_id": tablet_id, "lag_ms": lag_ms})

    status = "ok" if not lagging_tablets else "degraded"
    code = 200 if not lagging_tablets else 503

    return JSONResponse(status_code=code, content={
        "status": status,
        "lagging_tablets": lagging_tablets,
        "threshold_ms": LAG_THRESHOLD_MS,
    })

Step 4: Configure Vigilmon Monitors

TServer Cluster Health

  1. Log in to vigilmon.online → Monitors → New Monitor
  2. Type: HTTP / HTTPS
  3. URL: https://your-api.example.com/health/yugabyte
  4. Check interval: 1 minute
  5. Expected status: 200; response body contains: "status":"ok"
  6. Response time threshold: 5000ms (distributed query overhead is higher than single-node)
  7. Alert channel: Slack + PagerDuty

Master Leader Monitor

  1. Monitors → New Monitor → HTTP / HTTPS
  2. URL: https://your-api.example.com/health/yugabyte/master
  3. Check interval: 1 minute
  4. Expected status: 200
  5. Alert channel: Slack + PagerDuty (no leader = cluster degraded)

YSQL Latency Monitor

  1. Monitors → New Monitor → HTTP / HTTPS
  2. URL: https://your-api.example.com/health/yugabyte/ysql
  3. Check interval: 2 minutes
  4. Expected status: 200; response body contains: "latency_ms"
  5. Response time threshold: 3000ms
  6. Alert channel: Slack

Replication Lag Monitor

  1. Monitors → New Monitor → HTTP / HTTPS
  2. URL: https://your-api.example.com/health/yugabyte/replication
  3. Check interval: 2 minutes
  4. Expected status: 200
  5. Alert channel: Slack (P2 alert — investigate before it becomes P1)

Step 5: Heartbeat Monitoring for Scheduled YugabyteDB Jobs

YugabyteDB YSQL supports pg_cron (via extension) for scheduled maintenance: statistics refreshes, index rebuilds, TTL sweeps. Monitor these with Vigilmon heartbeats:

-- Install pg_cron in YugabyteDB (YSQL)
CREATE EXTENSION IF NOT EXISTS pg_cron;

-- Nightly ANALYZE with Vigilmon heartbeat ping
SELECT cron.schedule(
  'nightly-analyze',
  '0 3 * * *',
  $$
    DO $$
    BEGIN
      ANALYZE;
      PERFORM http_get('https://vigilmon.online/heartbeat/your-heartbeat-id');
    END;
    $$ LANGUAGE plpgsql;
  $$
);

If pg_cron is not available, use a shell script:

#!/bin/bash
# yb-maintenance.sh
set -euo pipefail

PGPASSWORD="$YB_PASSWORD" psql \
  --host="$YB_HOST" --port=5433 \
  --username="$YB_USER" --dbname="$YB_DATABASE" \
  --command="ANALYZE VERBOSE;"

curl -fsS "https://vigilmon.online/heartbeat/your-heartbeat-id" > /dev/null

Set the Vigilmon heartbeat interval to 1 day with a 2-hour grace period.

Alert Routing Summary

| Monitor | Severity | Channel | |---|---|---| | TServer cluster health | P1 | Slack + PagerDuty | | Master leader availability | P1 | Slack + PagerDuty | | YSQL query latency | P2 | Slack | | Replication lag | P2 | Slack | | Maintenance job heartbeat | P2 | Email + Slack |

Summary

YugabyteDB's distributed design introduces failure modes that single-node monitoring doesn't cover. Vigilmon's external HTTP probes catch the gap between "process is running" and "cluster is healthy":

| Layer | Vigilmon Monitor | |---|---| | TServer availability | HTTP monitor on /health/yugabyte | | Master leader election | HTTP monitor on /health/yugabyte/master | | YSQL query performance | HTTP monitor on /health/yugabyte/ysql | | Tablet replication lag | HTTP monitor on /health/yugabyte/replication | | Scheduled maintenance jobs | Heartbeat monitor |

Start monitoring your YugabyteDB cluster for free at vigilmon.online.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles