Monitoring TiDB Operator with Vigilmon

TiDB Operator is the Kubernetes operator that automates deployment, scaling, backup, and failover of TiDB clusters — the distributed, MySQL-compatible SQL database. When the operator pod fails, nothing catastrophic happens immediately, but automated failovers stop working, scaling requests queue forever, and upgrades stall. More critically, the TiDB SQL endpoint and PD (Placement Driver) quorum are user-facing services that need continuous external monitoring. Vigilmon covers all three layers: operator health, SQL endpoint availability, and PD cluster reachability.

What You'll Set Up

• HTTP monitor on TiDB Operator's metrics/health endpoint
• TCP monitor on the TiDB SQL port (4000) for connection availability
• HTTP monitor on PD's health API for quorum status
• SSL certificate monitoring for TiDB Dashboard
• Alerting tuned for database workload SLAs

Prerequisites

• Kubernetes cluster with TiDB Operator installed (v1.5+)
• At least one TidbCluster resource deployed
• kubectl access to the tidb-admin namespace
• A free Vigilmon account

Step 1: Monitor TiDB Operator Health

TiDB Operator's controller-manager exposes a readiness endpoint. Expose it:

kubectl port-forward -n tidb-admin deploy/tidb-controller-manager 6060:6060 &
curl http://localhost:6060/readyz

For external monitoring, create a NodePort:

kubectl expose deploy tidb-controller-manager \
  -n tidb-admin \
  --name tidb-operator-health \
  --port 6060 \
  --target-port 6060 \
  --type NodePort

Then add a Vigilmon monitor:

1. Log in to vigilmon.online and click Add Monitor.
2. Set Type to HTTP / HTTPS.
3. Enter http://<node-ip>:<node-port>/readyz.
4. Set Check interval to 1 minute.
5. Set Expected HTTP status to 200.
6. Click Save.

This catches operator pod crashes or CrashLoopBackOff states that would otherwise only surface when the next failover or scale event failed.

Step 2: Monitor the TiDB SQL Endpoint (Port 4000)

The TiDB SQL gateway listens on port 4000. This is the port your applications connect to — if it's unreachable, your entire database tier is down.

Expose it externally (it's often already a LoadBalancer service in cloud deployments):

kubectl get svc -n <tidb-namespace> | grep tidb
# Look for: <cluster-name>-tidb  LoadBalancer  ...  4000/TCP

Add a Vigilmon TCP monitor:

1. Click Add Monitor → TCP Port.
2. Enter the LoadBalancer IP or hostname and port 4000.
3. Set Check interval to 30 seconds.
4. Set Alert after to 2 consecutive failures.
5. Click Save.

TCP monitoring catches connection-layer failures (TiDB pods down, load balancer misconfigured) without needing SQL credentials.

For deeper checking, add a companion HTTP monitor against the TiDB status endpoint on port 10080:

curl http://<tidb-lb-ip>:10080/status
# Returns: {"connections":2,"version":"...","git_hash":"..."}

Add http://<tidb-lb-ip>:10080/status as an HTTP monitor expecting HTTP 200 with keyword check "version".

Step 3: Monitor PD (Placement Driver) Health

PD is TiDB's metadata cluster — it manages region assignment, scheduling, and TSO (timestamp oracle). PD runs a REST API on port 2379. Monitor the cluster health endpoint:

curl http://<pd-service>:2379/pd/api/v1/health

A healthy PD returns a JSON array of members, each with "health": true. Expose and monitor it:

kubectl get svc -n <tidb-namespace> | grep pd
# <cluster-name>-pd  ClusterIP  ...  2379/TCP

Expose via NodePort or Ingress, then add a Vigilmon HTTP monitor:

1. URL: http://<pd-host>:2379/pd/api/v1/health
2. Keyword check: "health":true
3. Check interval: 1 minute

If PD loses quorum (majority of PD pods fail), new timestamps stop being issued and the cluster becomes read-only. This monitor catches the earliest detectable signal of that failure mode.

Step 4: Monitor TiDB Dashboard SSL Certificate

TiDB Dashboard (the built-in UI) is often exposed on port 2379 under /dashboard path or on a separate Ingress with TLS. Add an SSL certificate expiry monitor:

1. In Vigilmon, click Add Monitor → SSL Certificate.
2. Enter the dashboard domain: https://tidb-dashboard.example.com.
3. Set Alert when 30 days before expiry.
4. Click Save.

Database teams often forget to renew TLS certificates for internal tools — this alert ensures the dashboard stays accessible for operational tasks.

Step 5: Heartbeat from a Scheduled DB Check

For end-to-end validation that TiDB is accepting SQL, use a CronJob that connects, runs a trivial query, and pings Vigilmon:

apiVersion: batch/v1
kind: CronJob
metadata:
  name: vigilmon-tidb-heartbeat
  namespace: <tidb-namespace>
spec:
  schedule: "*/5 * * * *"
  jobTemplate:
    spec:
      template:
        spec:
          restartPolicy: Never
          containers:
          - name: heartbeat
            image: mysql:8.0
            env:
            - name: TIDB_HOST
              value: "<cluster-name>-tidb.<tidb-namespace>.svc"
            - name: TIDB_PORT
              value: "4000"
            - name: TIDB_USER
              value: "root"
            command:
            - sh
            - -c
            - |
              mysql -h $TIDB_HOST -P $TIDB_PORT -u $TIDB_USER \
                --connect-timeout=5 -e "SELECT 1" && \
              wget -qO- https://vigilmon.online/api/heartbeat/<YOUR_HEARTBEAT_ID>

In Vigilmon:

1. Click Add Monitor → Heartbeat.
2. Set Expected interval to 5 minutes, Grace period to 2 minutes.
3. Paste the heartbeat URL into the manifest and apply it.

This catches failures that slip past TCP monitoring — for example, TiDB accepting connections but rejecting queries due to a PD quorum issue.

Step 6: Alert Configuration

Recommended notification settings for TiDB:

| Monitor | Alert delay | Escalation | |---|---|---| | TiDB Operator readyz | 2 min | On-call channel | | TiDB SQL port 4000 | 1 min | P1 pager | | PD health API | 1 min | P1 pager | | Dashboard SSL cert | Immediate at 30d | Email only | | SQL heartbeat | 7 min | On-call channel |

Set tighter thresholds for PD and the SQL port — database unavailability is always a P1 incident.

What You've Built

| Monitor | Type | Catches | |---|---|---| | Operator /readyz | HTTP | Operator pod failure | | TiDB port 4000 | TCP | SQL gateway unreachable | | TiDB /status | HTTP | TiDB process health | | PD /health | HTTP + keyword | PD quorum loss | | Dashboard TLS | SSL cert | Expired certificate | | SQL heartbeat | Heartbeat | End-to-end query path |

TiDB Operator orchestrates a complex distributed database — a monitoring gap at any layer can mean missed failovers or silent data access failures. This Vigilmon setup gives you external, credential-light coverage of every tier, from the operator control plane down to the SQL query path.