Monitoring YugabyteDB with Vigilmon

YugabyteDB promises five-nines availability through its distributed architecture — but that promise only holds if someone is watching the individual components. A master node failure degrades catalog operations; a tablet server failure drops data shards; an expired SSL certificate breaks client connections silently. Vigilmon gives you continuous monitoring across the entire YugabyteDB stack: master UI health, tablet server health, YSQL and YCQL TCP ports, SSL certificates, and tablet server heartbeats.

What You'll Set Up

• HTTP monitor for the YugabyteDB master UI health endpoint
• HTTP monitor for the tablet server (tserver) health check
• TCP port monitors for YSQL (5433) and YCQL (9042)
• SSL certificate expiry alerts for encrypted client connections
• Cron heartbeat for YugabyteDB tablet server availability

Prerequisites

• YugabyteDB 2.18+ cluster with at least 3 nodes
• Master UI reachable over the network (default port 7000)
• Tablet server health endpoint reachable (default port 9000)
• A free Vigilmon account

Step 1: Monitor the YugabyteDB Master Health Endpoint

The YugabyteDB master UI exposes a JSON health endpoint at /api/v1/health on port 7000. This endpoint returns the health status of the master and its reachable peers.

1. Log in to vigilmon.online and click Add Monitor.
2. Set Type to HTTP / HTTPS.
3. Enter the URL: http://yb-master1.example.com:7000/api/v1/health
4. Set Check interval to 1 minute.
5. Set Expected HTTP status to 200.
6. Under Response body must contain, enter "status":"OK" to verify the master is healthy.
7. Click Save.

A healthy response looks like:

{
  "status": "OK",
  "master_uuid": "xxxxxxxxxxxxxxxx",
  "node_instance": {
    "permanent_uuid": "xxxxxxxxxxxxxxxx",
    "instance_seqno": 1
  }
}

Add monitors for each master node in your cluster. YugabyteDB typically runs 3 or 5 master nodes, and you want to know if any individual master becomes unavailable — even if the cluster itself remains healthy with a quorum.

Step 2: Monitor the Tablet Server Health Endpoint

Tablet servers store the actual data shards. The tserver health endpoint is available at /api/v1/health on port 9000.

1. Click Add Monitor.
2. Set Type to HTTP / HTTPS.
3. Enter the URL: http://yb-tserver1.example.com:9000/api/v1/health
4. Set Check interval to 1 minute.
5. Set Expected HTTP status to 200.
6. Under Response body must contain, enter "status":"OK".
7. Click Save.

Repeat for each tablet server in the cluster. In a 3-node deployment, each node runs both a master and a tablet server — add separate monitors for each process on each node.

To verify the health endpoint manually:

curl -s http://yb-tserver1.example.com:9000/api/v1/health | python3 -m json.tool

Step 3: TCP Port Monitors for YSQL and YCQL

YugabyteDB exposes two SQL APIs:

• YSQL (PostgreSQL-compatible) on port 5433
• YCQL (Cassandra-compatible) on port 9042

Add TCP monitors for both:

YSQL monitor:

1. Click Add Monitor → TCP Port.
2. Set Host to yb-tserver1.example.com (or your load balancer VIP).
3. Set Port to 5433.
4. Set Check interval to 1 minute.
5. Click Save.

YCQL monitor:

1. Click Add Monitor → TCP Port.
2. Set Host to yb-tserver1.example.com.
3. Set Port to 9042.
4. Set Check interval to 1 minute.
5. Click Save.

TCP monitors detect port-level failures — process crashes, firewall rule changes, or port conflicts — faster than HTTP monitors because they don't need to wait for a full HTTP response. If you use a load balancer or smart driver (e.g. the YugabyteDB JDBC smart driver), monitor the VIP as well as at least one backend node directly.

Verify connectivity:

# Test YSQL
psql -h yb-tserver1.example.com -p 5433 -U yugabyte -c "SELECT version();"

# Test YCQL
ycqlsh yb-tserver1.example.com 9042 -e "DESCRIBE CLUSTER;"

Step 4: SSL Certificate Alerts for Encrypted Client Connections

YugabyteDB supports TLS for YSQL, YCQL, and the master/tserver UI endpoints. Certificate expiry on any of these breaks client connections silently from the server's perspective.

Enable SSL certificate monitoring on your YSQL and master UI monitors:

For the master UI (if using HTTPS on port 7000):

1. Open the HTTP monitor for https://yb-master1.example.com:7000/api/v1/health.
2. Enable Monitor SSL certificate.
3. Set Alert when certificate expires in less than 30 days.
4. Click Save.

For YSQL with TLS enabled:

Add an HTTP monitor pointing to the YSQL port with SSL enabled, or use a dedicated SSL check:

1. Click Add Monitor → HTTP / HTTPS.
2. Enter https://yb-tserver1.example.com:5433 (if your load balancer terminates TLS).
3. Enable Monitor SSL certificate with a 30 days alert threshold.
4. Click Save.

Check current certificate expiry on the YugabyteDB node:

# Check the node's TLS certificate
openssl x509 -in /home/yugabyte/yugabyte-tls-config/node.crt -noout -dates

# Check the CA certificate
openssl x509 -in /home/yugabyte/yugabyte-tls-config/ca.crt -noout -dates

To rotate certificates:

# Re-run the YugabyteDB TLS configuration script
yugabyted cert generate_server_certs --base_dir=/home/yugabyte/yugabyte-tls-config

Step 5: Heartbeat Monitoring for Tablet Server Availability

Tablet servers must remain active and heartbeat to the master to keep their data shards alive. A tablet server that goes missing for too long triggers tablet re-replication, which puts load on the cluster. Use a Vigilmon cron heartbeat to track tablet server availability from the master's perspective.

Create a monitoring script that runs on a schedule and checks the tserver count:

#!/bin/bash
# /usr/local/bin/yugabyte-heartbeat.sh

MASTER_HOST="yb-master1.example.com"
EXPECTED_TSERVERS=3
HEARTBEAT_URL="https://vigilmon.online/heartbeat/abc123"

# Query the master API for live tserver count
LIVE=$(curl -s "http://${MASTER_HOST}:7000/api/v1/tablet-servers" | \
  python3 -c "
import sys, json
data = json.load(sys.stdin)
live = sum(1 for ts in data.get('tablet_servers', []) if ts.get('alive'))
print(live)
")

if [ "$LIVE" -ge "$EXPECTED_TSERVERS" ]; then
  curl -fsS "$HEARTBEAT_URL"
fi

Schedule it with cron:

# Run every 5 minutes
*/5 * * * * /usr/local/bin/yugabyte-heartbeat.sh

In Vigilmon:

1. Click Add Monitor → Cron Heartbeat.
2. Set Expected ping interval to 7 minutes.
3. Name it yugabyte-tserver-quorum.
4. Paste the heartbeat URL into the script above.
5. Click Save.

If any tserver drops below the expected count, the heartbeat stops — Vigilmon alerts after 7 minutes of silence.

Step 6: Configure Alert Channels

1. Go to Alert Channels in Vigilmon and add Slack, PagerDuty, or email.
2. Set Consecutive failures before alert to 2 on master UI HTTP monitors — brief master leader elections can cause short 503 windows.
3. Set Consecutive failures before alert to 1 on TCP port monitors — a closed YSQL or YCQL port is an immediate application impact.
4. Set Consecutive failures before alert to 1 on the heartbeat monitor — a missing tserver warrants immediate investigation.

Summary

| Monitor | Target | What It Catches | |---|---|---| | HTTP /api/v1/health | Master UI port 7000 | Master crash, failed leader election | | HTTP /api/v1/health | Tserver port 9000 | Tablet server crash, health degradation | | TCP port 5433 | YSQL port | PostgreSQL API unavailability | | TCP port 9042 | YCQL port | Cassandra API unavailability | | SSL certificate | Master + tserver TLS certs | Certificate expiry breaking client connections | | Cron heartbeat | Tserver quorum check | Under-replicated cluster, tserver loss |

YugabyteDB's distributed architecture gives you resilience — but resilience without visibility is just a delayed outage. With Vigilmon watching master health, tserver availability, SQL API ports, SSL certificates, and tablet quorum heartbeats, you'll detect degradation before it becomes downtime.