tutorial

How to Monitor Vitess (MySQL Clustering) with Vigilmon

Vitess cluster failures and VSchema routing errors are invisible to standard MySQL monitoring. Learn how to monitor Vitess VTGate, VTTablet health, and query routing with Vigilmon HTTP probes and heartbeat monitors.

Vitess turns MySQL into a horizontally scalable database cluster — but with that power comes a new layer of failure modes that standard MySQL monitoring completely misses. A VTGate pod crash, a VTTablet falling out of the serving pool, or a VSchema routing misconfiguration can silently drop queries or route them to the wrong shard without returning a single MySQL error. By the time users notice, thousands of requests may have failed or hit stale data.

Vigilmon gives you external visibility into Vitess cluster health through HTTP probe monitoring of VTGate and VTorc health endpoints, plus heartbeat monitoring for your application-side query health. This tutorial shows you exactly how to set it up.

Why Vitess Needs External Monitoring

Vitess introduces components between your application and MySQL that each carry their own failure modes:

  • VTGate — the query routing proxy. If it crashes or becomes unhealthy, all application queries fail regardless of whether the underlying MySQL tablets are healthy.
  • VTTablet — the per-shard MySQL proxy. If a tablet falls out of the serving pool, the shard becomes unavailable or read-only.
  • VTorc — the orchestrator integration for automated failover. A dead VTorc means no automatic primary promotion when a shard primary crashes.
  • VTAdmin — the administrative API. Useful for probing cluster-level topology health.

Standard MySQL monitoring (ping checks, slow query monitoring) cannot see these components. External monitoring with Vigilmon adds:

  • Proactive alerting when VTGate or VTTablet health endpoints return non-200
  • Shard availability monitoring via VTGate's per-keyspace health API
  • Heartbeat monitoring so you know when your application has stopped successfully routing queries — even if all Vitess components report healthy
  • Multi-region probe consensus to eliminate false alerts from transient network partitions

Step 1: Expose Vitess Health Endpoints

Vitess ships with built-in health endpoints on each component. No custom code is required to get started.

VTGate Health Endpoint

VTGate exposes a /healthz endpoint on its web port (default 15001):

# Check VTGate health — returns 200 OK when ready to serve queries
curl -i http://vtgate-host:15001/healthz

# Check VTGate debug vars for deeper diagnostics
curl -s http://vtgate-host:15001/debug/vars | jq '.VtgateQueries'

If VTGate is not ready to route queries (e.g., during startup or if it loses contact with all tablets for a keyspace), /healthz returns a non-200 response. This is the primary monitor to configure.

VTTablet Health Endpoint

Each VTTablet exposes its own /healthz on its web port (default 15002):

# Check a specific VTTablet — returns 200 when in the serving pool
curl -i http://vttablet-host:15002/healthz

# Check tablet type (PRIMARY, REPLICA, RDONLY)
curl -s http://vttablet-host:15002/debug/vars | jq '.TabletType'

For a Kubernetes deployment, you typically monitor the VTGate service rather than individual VTTablet pods (the service handles routing to healthy pods). Add VTTablet monitoring for critical shards where you want per-shard visibility.

Custom Application-Level Health Endpoint

For query routing validation — not just connectivity — add an application health endpoint that runs a simple Vitess query and verifies the result:

# health/vitess.py  (FastAPI example)
from fastapi import FastAPI
from sqlalchemy import create_engine, text
import os

app = FastAPI()

# Vitess connection via standard MySQL protocol through VTGate
engine = create_engine(
    f"mysql+pymysql://{os.environ['DB_USER']}:{os.environ['DB_PASSWORD']}"
    f"@{os.environ['VTGATE_HOST']}:3306/{os.environ['DB_KEYSPACE']}"
)

@app.get("/health/vitess")
async def vitess_health():
    try:
        with engine.connect() as conn:
            # Use a scatter query to verify routing across shards
            result = conn.execute(text("SELECT 1 FROM dual"))
            result.fetchone()
        return {"status": "ok", "routing": "healthy"}
    except Exception as e:
        return {"status": "down", "error": str(e)}, 503

// health/vitess.go
package main

import (
    "database/sql"
    "encoding/json"
    "fmt"
    "net/http"
    "os"

    _ "github.com/go-sql-driver/mysql"
)

var db *sql.DB

func vitessHealthHandler(w http.ResponseWriter, r *http.Request) {
    _, err := db.Exec("SELECT 1")
    if err != nil {
        w.WriteHeader(http.StatusServiceUnavailable)
        json.NewEncoder(w).Encode(map[string]string{"status": "down", "error": err.Error()})
        return
    }
    json.NewEncoder(w).Encode(map[string]string{"status": "ok"})
}

func main() {
    dsn := fmt.Sprintf("%s:%s@tcp(%s:3306)/%s",
        os.Getenv("DB_USER"), os.Getenv("DB_PASSWORD"),
        os.Getenv("VTGATE_HOST"), os.Getenv("DB_KEYSPACE"),
    )
    var err error
    db, err = sql.Open("mysql", dsn)
    if err != nil {
        panic(err)
    }

    http.HandleFunc("/health/vitess", vitessHealthHandler)
    http.ListenAndServe(":8080", nil)
}

Step 2: Configure Vigilmon HTTP Monitors for Vitess

Monitor VTGate

  1. Log in to vigilmon.online and go to Monitors → New Monitor
  2. Choose HTTP / HTTPS
  3. Set the URL to your VTGate health endpoint:
    • Direct: http://vtgate-host:15001/healthz
    • Via Kubernetes ingress: https://vtgate.internal.example.com/healthz
  4. Set the check interval to 1 minute
  5. Under Expected response:
    • Status code: 200
    • Response time threshold: 3000ms
  6. Under Alert channels, assign your primary on-call channel (Slack + PagerDuty)
  7. Save the monitor

Monitor VTTablet Shards

Add one monitor per critical shard VTTablet (or per service in Kubernetes):

  • URL: http://vttablet-shard-0-primary:15002/healthz
  • Interval: 1 minute
  • Expected: 200
  • Alert: Slack (P2, secondary to VTGate alert)

Monitor Application Query Routing

Add a monitor for your application-level health endpoint to verify end-to-end query routing:

  • URL: https://your-app.example.com/health/vitess
  • Expected: 200, body contains "status":"ok"
  • Interval: 2 minutes
  • Response time threshold: 5000ms (allows for scatter-gather query time)

Vigilmon's multi-region probes eliminate false positives from regional network issues.

Step 3: Heartbeat Monitoring for Vitess-Backed Jobs

Vitess is frequently used as the database layer for batch jobs, data pipeline workers, and background processors. A worker can connect to VTGate successfully but silently stall mid-job if it hits a tablet that's in the middle of a failover or if VSchema invalidation causes routing errors.

Vigilmon heartbeat monitors detect silent worker stalls: your job sends a ping after completing each work unit. If pings stop, Vigilmon fires an alert.

Set Up the Heartbeat Monitor

  1. In Vigilmon, go to Monitors → New Monitor → Heartbeat
  2. Name it: vitess-batch-processor
  3. Set expected interval: 10 minutes (adjust to your job cadence)
  4. Set grace period: 15 minutes
  5. Save — copy the heartbeat URL, e.g. https://vigilmon.online/heartbeat/abc123xyz

Wire Into Your Batch Processor (Python)

# batch_processor.py
import pymysql
import requests
import os
import time

VIGILMON_HEARTBEAT = os.environ["VIGILMON_HEARTBEAT_URL"]

def process_batch(batch_id: int) -> None:
    conn = pymysql.connect(
        host=os.environ["VTGATE_HOST"],
        port=3306,
        user=os.environ["DB_USER"],
        password=os.environ["DB_PASSWORD"],
        database=os.environ["DB_KEYSPACE"],
    )
    try:
        with conn.cursor() as cur:
            cur.execute(
                "SELECT id, payload FROM jobs WHERE batch_id = %s AND status = 'pending'",
                (batch_id,)
            )
            rows = cur.fetchall()
            for row in rows:
                process_job(row)
                cur.execute(
                    "UPDATE jobs SET status = 'done' WHERE id = %s", (row[0],)
                )
            conn.commit()
        # Ping Vigilmon after each successful batch
        requests.get(VIGILMON_HEARTBEAT, timeout=5)
    finally:
        conn.close()

Wire Into Your Go Worker

// worker.go
package main

import (
    "database/sql"
    "fmt"
    "net/http"
    "os"
    "time"

    _ "github.com/go-sql-driver/mysql"
)

func processJobs(db *sql.DB) error {
    rows, err := db.Query("SELECT id, payload FROM jobs WHERE status = 'pending' LIMIT 100")
    if err != nil {
        return fmt.Errorf("query failed: %w", err)
    }
    defer rows.Close()

    for rows.Next() {
        var id int
        var payload string
        if err := rows.Scan(&id, &payload); err != nil {
            return err
        }
        if err := handleJob(id, payload); err != nil {
            return err
        }
    }

    // Ping Vigilmon to confirm healthy processing
    http.Get(os.Getenv("VIGILMON_HEARTBEAT_URL"))
    return nil
}

func main() {
    db, _ := sql.Open("mysql", fmt.Sprintf(
        "%s:%s@tcp(%s:3306)/%s",
        os.Getenv("DB_USER"), os.Getenv("DB_PASSWORD"),
        os.Getenv("VTGATE_HOST"), os.Getenv("DB_KEYSPACE"),
    ))
    defer db.Close()

    ticker := time.NewTicker(5 * time.Minute)
    for range ticker.C {
        if err := processJobs(db); err != nil {
            fmt.Printf("batch error: %v\n", err)
        }
    }
}

Step 4: Alert Routing for Vitess Failures

Vitess failures cascade in a specific pattern: a tablet failover causes brief unavailability, which causes VTGate to mark that shard degraded, which causes application query errors. Configure alert routing to match this cascade:

| Monitor | Alert Channel | Priority | |---|---|---| | VTGate /healthz | Slack + PagerDuty | P1 | | VTTablet /healthz (primary shard) | Slack + PagerDuty | P1 | | VTTablet /healthz (replica shards) | Slack | P2 | | App /health/vitess query routing | Slack | P2 | | Heartbeat: batch processor | Email + Slack | P2 |

Set response time thresholds for early warning:

  • Alert at 2000ms for VTGate health (slow health check signals VTGate under load)
  • Alert at 3000ms for VTTablet health (slow response precedes falling out of serving pool)
  • Alert at 8000ms for application query routing (scatter-gather query latency climbing)

For production Vitess clusters, configure escalation: if a VTTablet P2 alert fires and is unacknowledged for 5 minutes, escalate to P1 so the on-call engineer investigates before the tablet falls completely out of the serving pool.

Summary

Vitess adds query routing, sharding, and automatic failover to MySQL — but each of those features introduces failure modes that MySQL monitoring cannot see. External monitoring with Vigilmon closes that gap:

| Monitor Type | What It Covers | |---|---| | HTTP monitor on VTGate /healthz | VTGate liveness, query routing availability | | HTTP monitor on VTTablet /healthz | Per-shard tablet serving pool status | | HTTP monitor on app /health/vitess | End-to-end query routing validation | | Heartbeat monitor | Batch job and worker liveness |

Get started free at vigilmon.online — your first Vitess monitor is running in under two minutes.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles