TiDB Monitoring with Vigilmon

TiDB is an open-source, MySQL-compatible distributed SQL database. It separates compute (TiDB server nodes) from storage (TiKV), with a Placement Driver (PD) cluster managing metadata and scheduling. TiDB can scale horizontally and survive node failures transparently, making it popular for high-throughput OLTP and HTAP workloads that have outgrown single-node MySQL.

However, TiDB's distributed architecture introduces more failure points than a single-node database. A TiDB server can be healthy while PD is degraded, or TiKV regions can be under-replicated after a node failure. This guide covers how to monitor every tier of a TiDB cluster with Vigilmon using built-in HTTP health endpoints.

TiDB Cluster Architecture

| Component | Default Port | Role | |-----------|-------------|------| | TiDB server | 4000 (SQL), 10080 (HTTP) | MySQL-compatible SQL interface | | PD (Placement Driver) | 2379 | Metadata, scheduling, cluster brain | | TiKV | 20160 (gRPC), 20180 (HTTP) | Distributed key-value storage | | TiDB Dashboard | 2379/dashboard | Web UI (served by PD) | | TiFlash (optional) | 8123 (HTTP), 9000 | Columnar storage for analytics |

Each component exposes HTTP health endpoints. Unlike MySQL's binary protocol, TiDB's status endpoints are HTTP-native and purpose-built for monitoring.

Health Endpoints to Monitor

TiDB Server — /status

GET http://tidb-host:10080/status

Response when healthy:

{
  "connections": 12,
  "git_hash": "abc123",
  "lease": "45s",
  "status": 0,
  "version": "5.7.25-TiDB-v7.5.0"
}

The "status": 0 field indicates the server is healthy. A non-zero status or a non-200 HTTP response means the TiDB server is unhealthy.

TiDB Server — /health

GET http://tidb-host:10080/health

A simpler alternative that returns just the health status:

HTTP 200 OK

Or HTTP 503 when unhealthy. Use this for a lightweight ping check.

PD — /pd/api/v1/health

GET http://pd-host:2379/pd/api/v1/health

Response (one entry per PD member):

[
  {
    "name": "pd1",
    "member_id": 12345678901234567,
    "client_urls": ["http://pd1:2379"],
    "health": true
  },
  {
    "name": "pd2",
    "member_id": 12345678901234568,
    "client_urls": ["http://pd2:2379"],
    "health": false
  }
]

Each PD member reports its own health. If any member has "health": false, that PD node is down. PD manages all cluster metadata — a PD outage means TiDB cannot process DDL changes or region scheduling, and writes may stall.

PD — /pd/api/v1/cluster

GET http://pd-host:2379/pd/api/v1/cluster

Returns cluster ID and basic metadata. Use as a connectivity check to confirm PD is responding to API requests.

TiKV — /metrics

TiKV exposes Prometheus metrics on its HTTP port:

GET http://tikv-host:20180/metrics

This endpoint returns a large Prometheus text payload. For Vigilmon, use it as a liveness check (HTTP 200 = TiKV process is running). You can combine it with a keyword check for a specific metric name to confirm the metrics scraper is healthy.

TiKV — Healthz Check

GET http://tikv-host:20180/healthz

Returns:

HTTP 200 OK

This is the preferred TiKV liveness probe.

TiDB Dashboard

The TiDB Dashboard is served by PD:

GET http://pd-host:2379/dashboard

A 200 response with TiDB Dashboard in the HTML confirms the dashboard is accessible. Dashboard availability is lower-priority than cluster health but important for operator access during incidents.

Setting Up Vigilmon Monitors

Monitor 1: TiDB Server Health

1. Log in to Vigilmon → Monitors → New Monitor
2. Type: HTTP
3. Method: GET
4. URL: http://tidb-host:10080/health
5. Interval: 1 minute
6. Expected status: 200

Create one monitor per TiDB server node. In a production cluster with multiple TiDB nodes, monitor each one individually. A degraded TiDB node can still accept connections from the load balancer — you want to know which node is unhealthy.

Monitor 2: TiDB Status (with Keyword)

For more detailed health validation:

1. Type: HTTP
2. Method: GET
3. URL: http://tidb-host:10080/status
4. Interval: 2 minutes
5. Expected status: 200
6. Keyword check: "status":0

The "status":0 keyword catches TiDB's internal unhealthy state even when the HTTP response is 200.

Monitor 3: PD Health API

1. Type: HTTP
2. Method: GET
3. URL: http://pd-host:2379/pd/api/v1/health
4. Interval: 1 minute
5. Expected status: 200
6. Keyword check: "health":true

This confirms at least one PD member is healthy. For strict monitoring, create separate monitors per PD node and alert if any member's health drops to false.

Monitor 4: PD Cluster Connectivity

1. Type: HTTP
2. Method: GET
3. URL: http://pd-host:2379/pd/api/v1/cluster
4. Interval: 2 minutes
5. Expected status: 200

Monitor 5: TiKV Node Liveness

1. Type: HTTP
2. Method: GET
3. URL: http://tikv-host:20180/healthz
4. Interval: 1 minute
5. Expected status: 200

Create one monitor per TiKV node. TiKV stores your actual data — a TiKV node failure reduces the replication factor of all regions on that node, increasing risk of data unavailability if a second node fails.

Monitor 6: TiDB Dashboard

1. Type: HTTP
2. Method: GET
3. URL: http://pd-host:2379/dashboard
4. Interval: 5 minutes
5. Expected status: 200
6. Keyword check: TiDB Dashboard

Kubernetes Deployment Health Checks

TiDB is typically deployed on Kubernetes using the TiDB Operator. The operator manages liveness and readiness probes automatically, but Vigilmon provides the external perspective:

# TiDB server liveness probe (managed by TiDB Operator)
livenessProbe:
  httpGet:
    path: /status
    port: 10080
  initialDelaySeconds: 30
  periodSeconds: 10
  failureThreshold: 3

readinessProbe:
  httpGet:
    path: /status
    port: 10080
  initialDelaySeconds: 5
  periodSeconds: 10
  failureThreshold: 3

# PD liveness probe
livenessProbe:
  httpGet:
    path: /pd/api/v1/health
    port: 2379
  initialDelaySeconds: 30
  periodSeconds: 10

The TiDB Operator's probes check internal cluster health; Vigilmon's external monitors catch ingress failures, load balancer misconfigurations, and TLS certificate expiry that internal probes cannot see.

Alert Configuration

Slack Alerts

In Vigilmon → Alert Channels → Slack, configure alerts:

• TiDB server monitors: Alert after 1 failed check — a TiDB node failure means the load balancer is sending some percentage of client traffic to a down node
• PD health monitor: Alert after 1 failed check — PD is the cluster's brain; degraded PD blocks DDL and may stall writes
• TiKV monitors: Alert after 2 failed checks — a single TiKV node failure reduces replication factor but does not immediately cause data loss

Webhook for Incident Routing

app.post('/webhooks/vigilmon', express.json(), (req, res) => {
  const { event, monitor } = req.body;

  if (event === 'down') {
    const component = monitor.url.includes(':10080') ? 'TiDB server'
      : monitor.url.includes(':2379') ? 'PD'
      : monitor.url.includes(':20180') ? 'TiKV'
      : 'Unknown';

    const severity = component === 'PD' ? 'critical' : 'high';

    createIncident({
      title: `TiDB cluster component DOWN: ${component}`,
      description: `Monitor: ${monitor.url}`,
      severity,
    });
  } else if (event === 'up') {
    resolveIncident({ url: monitor.url });
  }

  res.json({ received: true });
});

Register this URL under Alert Channels → Webhook in Vigilmon.

Alerting Strategy

| Component | Endpoint | Alert Threshold | |-----------|----------|-----------------| | TiDB server | :10080/health | 1 failed check | | TiDB status | :10080/status keyword "status":0 | 1 failed check | | PD health | /pd/api/v1/health | 1 failed check | | TiKV liveness | :20180/healthz | 2 failed checks | | Dashboard | /dashboard | 3 failed checks | | Response latency | Any endpoint > 3000ms | Warning alert |

Summary

| Component | Endpoint | Vigilmon Check | |-----------|----------|----------------| | TiDB server liveness | :10080/health | Status 200 | | TiDB server status | :10080/status | Status 200 + keyword "status":0 | | PD cluster health | :2379/pd/api/v1/health | Status 200 + keyword "health":true | | PD connectivity | :2379/pd/api/v1/cluster | Status 200 | | TiKV node health | :20180/healthz | Status 200 | | Dashboard availability | :2379/dashboard | Status 200 |

TiDB's distributed architecture means a healthy MySQL connection does not imply a healthy cluster. PD could be degraded, TiKV nodes could be under-replicated, or individual TiDB nodes could be crashing. With Vigilmon monitoring each tier independently at 1-minute intervals, you have full visibility into cluster health — not just connection availability.

Further Reading

• Monitoring ClickHouse with Vigilmon
• Monitoring Cassandra with Vigilmon
• Monitoring Elasticsearch with Vigilmon
• Kubernetes Monitoring with Vigilmon