tutorial

Monitoring TiKV with Vigilmon

TiKV is the distributed key-value engine under TiDB — when it degrades, your entire database layer feels it. Here's how to monitor TiKV health endpoints, PD cluster health, client ports, store heartbeats, and SSL certificates with Vigilmon.

TiKV is a CNCF graduated project and the distributed transactional storage layer that powers TiDB. Its split architecture — TiKV stores + Placement Driver (PD) cluster — means there are multiple components that can silently degrade while your application continues to function at reduced capacity. Vigilmon gives you end-to-end visibility: HTTP health checks on the status API and PD, TCP port monitoring for the client endpoint, heartbeat verification for store availability, and SSL certificate alerts for inter-node encryption.

What You'll Set Up

  • HTTP monitor for the TiKV status address health endpoint
  • HTTP monitor for the PD (Placement Driver) health API
  • TCP port monitor for the TiKV client port (20160)
  • Cron heartbeat for TiKV store availability
  • SSL certificate alerts for encrypted inter-node communication

Prerequisites

  • TiKV cluster deployed (standalone or as part of TiDB)
  • PD cluster running and reachable
  • TiKV status address port (default 20180) reachable for HTTP checks
  • A free Vigilmon account

Step 1: Monitor the TiKV Status Address Health Endpoint

TiKV exposes an HTTP status server on port 20180 (configurable via server.status-addr in tikv.toml). This port serves both the Prometheus metrics scrape endpoint (/metrics) and a lightweight health check (/status).

Add a health monitor:

  1. Log in to vigilmon.online and click Add Monitor.
  2. Set Type to HTTP / HTTPS.
  3. Enter the URL: http://tikv1.example.com:20180/status
  4. Set Check interval to 1 minute.
  5. Set Expected HTTP status to 200.
  6. Under Response body must contain, enter TiKV to verify the response is from the correct service.
  7. Click Save.

A healthy /status response returns basic version and build information:

TiKV - Distributed transactional key value database
Version: 7.5.0
...

Add monitors for each TiKV store node in your cluster. In a production setup with 3+ stores, you want per-node visibility — a single-store failure degrades a percentage of reads and writes without taking the cluster down.

For Prometheus metrics, you can also add a separate monitor on the /metrics endpoint:

http://tikv1.example.com:20180/metrics

Set Expected HTTP status to 200 and Response body must contain to tikv_server_start_ts to confirm TiKV is reporting metrics.

Step 2: Monitor the PD Health Endpoint

The Placement Driver (PD) manages cluster topology, region scheduling, and leader election for TiKV. PD exposes a REST API on port 2379 (or 2380 for peer communication). The /pd/api/v1/health endpoint reports the health of all PD members.

  1. Click Add Monitor.
  2. Set Type to HTTP / HTTPS.
  3. Enter the URL: http://pd1.example.com:2379/pd/api/v1/health
  4. Set Check interval to 1 minute.
  5. Set Expected HTTP status to 200.
  6. Under Response body must contain, enter "health":true to verify PD is reporting healthy members.
  7. Click Save.

A healthy PD response:

[
  {
    "name": "pd1",
    "member_id": 12345678901234567,
    "client_urls": ["http://pd1.example.com:2379"],
    "health": true
  },
  {
    "name": "pd2",
    "member_id": 12345678901234568,
    "client_urls": ["http://pd2.example.com:2379"],
    "health": true
  }
]

Add monitors for each PD member endpoint. You can also monitor the /pd/api/v1/leader endpoint to confirm a PD leader is elected:

http://pd1.example.com:2379/pd/api/v1/leader

If this returns an empty response or a non-200, PD has lost its leader — a critical condition that halts region scheduling.

Step 3: TCP Port Monitor for the TiKV Client Port

TiKV clients (the TiDB compute layer, or your application using a TiKV client library) connect on port 20160. This is the gRPC endpoint that handles all transactional KV requests.

Add a TCP monitor:

  1. Click Add Monitor → TCP Port.
  2. Set Host to tikv1.example.com.
  3. Set Port to 20160.
  4. Set Check interval to 1 minute.
  5. Click Save.

Add TCP monitors for all TiKV stores. A closed port 20160 means clients can't reach that store for reads or writes — even if the status address is still responding.

Verify connectivity:

# Test TCP connectivity to TiKV gRPC port
nc -zv tikv1.example.com 20160
# Connection to tikv1.example.com 20160 port [tcp/*] succeeded!

Also add a TCP monitor for the PD client port:

  1. Click Add Monitor → TCP Port.
  2. Set Host to pd1.example.com.
  3. Set Port to 2379.
  4. Set Check interval to 1 minute.
  5. Click Save.

Step 4: SSL Certificate Alerts for Inter-Node TLS

TiKV supports TLS for client-to-server and server-to-server communication. When using TLS, configure it in tikv.toml:

[security]
ca-path = "/path/to/ca.pem"
cert-path = "/path/to/tikv.pem"
key-path = "/path/to/tikv-key.pem"

Add SSL certificate monitoring on your HTTPS monitors:

For TiKV status address (HTTPS):

  1. Update the TiKV status monitor URL to use https://.
  2. Open the monitor and enable Monitor SSL certificate.
  3. Set Alert when certificate expires in less than 30 days.
  4. Click Save.

For PD API (HTTPS):

  1. Update the PD health monitor to https://pd1.example.com:2379/pd/api/v1/health.
  2. Enable Monitor SSL certificate with a 30 days threshold.
  3. Click Save.

Check certificate expiry manually:

# Check TiKV node certificate
openssl x509 -in /path/to/tikv.pem -noout -dates

# Check PD certificate
openssl x509 -in /path/to/pd.pem -noout -dates

# Check CA certificate
openssl x509 -in /path/to/ca.pem -noout -dates

To rotate TiKV certificates with TiUP:

# Re-deploy TLS certificates using TiUP
tiup cluster cert renew <cluster-name>

Step 5: Heartbeat Monitoring for TiKV Store Availability

TiKV stores report their health to PD via heartbeats. When a store misses heartbeats, PD marks it as Disconnected and eventually Down. Use Vigilmon's cron heartbeat to track store availability from the PD API perspective.

Create a script that queries PD for store health:

#!/bin/bash
# /usr/local/bin/tikv-heartbeat.sh

PD_HOST="pd1.example.com"
EXPECTED_STORES=3
HEARTBEAT_URL="https://vigilmon.online/heartbeat/abc123"

# Query PD stores API and count stores with state "Up"
UP_STORES=$(curl -s "http://${PD_HOST}:2379/pd/api/v1/stores" | \
  python3 -c "
import sys, json
data = json.load(sys.stdin)
up = sum(1 for s in data.get('stores', [])
         if s.get('store', {}).get('state_name') == 'Up')
print(up)
")

if [ "$UP_STORES" -ge "$EXPECTED_STORES" ]; then
  curl -fsS "$HEARTBEAT_URL"
fi

Schedule with cron:

# Run every 5 minutes
*/5 * * * * /usr/local/bin/tikv-heartbeat.sh

In Vigilmon:

  1. Click Add Monitor → Cron Heartbeat.
  2. Set Expected ping interval to 7 minutes.
  3. Name it tikv-store-availability.
  4. Paste the heartbeat URL into the script above.
  5. Click Save.

Query PD stores manually to see current state:

# List all stores and their state
curl -s http://pd1.example.com:2379/pd/api/v1/stores | \
  python3 -c "
import sys, json
data = json.load(sys.stdin)
for s in data['stores']:
    store = s['store']
    print(f'{store[\"address\"]} - {store[\"state_name\"]}')
"

Step 6: Configure Alert Channels

  1. Go to Alert Channels in Vigilmon and add Slack, PagerDuty, or email.
  2. Set Consecutive failures before alert to 2 on HTTP health monitors — TiKV and PD can briefly return non-200 during leader re-election.
  3. Set Consecutive failures before alert to 1 on TCP port monitors — a closed client port causes immediate application errors.
  4. Set Consecutive failures before alert to 1 on the heartbeat monitor — a missing store changes cluster capacity immediately.
  5. Add a Maintenance window in Vigilmon during TiKV rolling upgrades via TiUP to suppress expected per-store restarts.

Summary

| Monitor | Target | What It Catches | |---|---|---| | HTTP /status | TiKV status port 20180 | Store process crash, health failure | | HTTP /pd/api/v1/health | PD cluster port 2379 | PD member failure, leader election loss | | TCP port 20160 | TiKV gRPC client port | Client connection failure, firewall block | | TCP port 2379 | PD client port | PD API unavailability | | SSL certificate | TiKV + PD TLS certs | Certificate expiry breaking encrypted connections | | Cron heartbeat | PD stores API — store Up count | Under-capacity cluster, store marked Down |

TiKV's strength is its resilience through replication — but that resilience is silent without monitoring. With Vigilmon watching the TiKV status API, PD health, client ports, SSL certificates, and store availability via heartbeat, you'll know when any part of the distributed storage layer degrades before your application does.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles