tutorial

How to Monitor GreptimeDB Health and Time-Series Ingestion with Vigilmon

GreptimeDB ingestion failures or compaction backlogs silently corrupt your metrics pipeline. Learn how to monitor GreptimeDB frontend health, gRPC and SQL protocol availability, Prometheus remote-write ingestion rate, and object storage backend status with Vigilmon.

GreptimeDB is a unified time-series and analytics database — a single store for Prometheus metrics, OpenTelemetry traces, and SQL analytics. But when the frontend goes down or object storage starts returning errors, the failure is silent: Prometheus remote-write silently drops samples, Grafana dashboards stop updating, and you're debugging stale metrics with no live data.

Vigilmon gives you external visibility into GreptimeDB through HTTP health probes, TCP port monitors, and heartbeat monitors for your ingestion pipelines. This tutorial covers frontend health, gRPC and SQL protocol availability, Prometheus remote-write ingestion, object storage health, and compaction lag.

Why GreptimeDB Needs External Monitoring

GreptimeDB's built-in metrics (via its Prometheus /metrics endpoint) provide rich internal observability — but internal metrics require GreptimeDB itself to be functioning. External monitoring with Vigilmon adds:

  • Frontend health checking via the /health endpoint — the first signal that the frontend component is down
  • Protocol availability monitoring — gRPC (4001), MySQL (4002), and PostgreSQL (4003) each serve different clients; any can fail independently
  • Prometheus remote-write proxy monitoring — if remote-write silently fails, Prometheus loses its long-term storage backend
  • Object storage health checks — GreptimeDB writes to S3/MinIO; storage errors cause write failures before they appear in any dashboard
  • Multi-region external probe consensus to distinguish GreptimeDB failures from local network issues

Step 1: GreptimeDB's Built-In Health Endpoint

GreptimeDB exposes a /health endpoint on its HTTP port (default 4000):

# HTTP health check
curl http://localhost:4000/health
# Returns: {"status":"ok"}

# Prometheus metrics endpoint (extensive internal metrics)
curl http://localhost:4000/metrics

# GreptimeDB admin dashboard API
curl http://localhost:4000/v1/sql?sql=SELECT+1

You can point Vigilmon directly at http://your-greptimedb-host:4000/health without a sidecar for basic health monitoring. For richer signal (ingestion rates, object storage health, compaction status), build an aggregator.

Step 2: Build a GreptimeDB Health Aggregator

// greptimedb-health.js
const express = require('express');
const axios = require('axios');

const app = express();

const GREPTIMEDB_HTTP = process.env.GREPTIMEDB_HTTP_URL || 'http://localhost:4000';

app.get('/health/greptimedb', async (req, res) => {
  try {
    const response = await axios.get(`${GREPTIMEDB_HTTP}/health`, { timeout: 5000 });

    if (response.data?.status !== 'ok') {
      return res.status(503).json({ status: 'degraded', upstream: response.data });
    }

    return res.status(200).json({ status: 'ok' });
  } catch (err) {
    return res.status(503).json({ status: 'down', error: err.message });
  }
});

// Ingestion health — execute a test write via HTTP API and measure write latency
app.get('/health/greptimedb/ingestion', async (req, res) => {
  const start = Date.now();
  try {
    // InfluxDB line protocol write — simplest ingestion path
    const lineProtocolData = `health_check,source=vigilmon value=1 ${Date.now() * 1000000}`;

    await axios.post(
      `${GREPTIMEDB_HTTP}/v1/influxdb/write?db=public`,
      lineProtocolData,
      { headers: { 'Content-Type': 'text/plain' }, timeout: 10000 }
    );

    const latencyMs = Date.now() - start;
    const threshold = parseInt(process.env.WRITE_LATENCY_THRESHOLD_MS || '500');

    if (latencyMs > threshold) {
      return res.status(503).json({
        status: 'degraded',
        reason: 'write_latency_high',
        latency_ms: latencyMs,
        threshold_ms: threshold,
      });
    }

    return res.status(200).json({ status: 'ok', write_latency_ms: latencyMs });
  } catch (err) {
    return res.status(503).json({ status: 'down', error: err.message });
  }
});

// SQL query health — test MySQL-compatible query endpoint
app.get('/health/greptimedb/query', async (req, res) => {
  const start = Date.now();
  try {
    const response = await axios.get(
      `${GREPTIMEDB_HTTP}/v1/sql?sql=SELECT+1+AS+health`,
      { timeout: 10000 }
    );

    const latencyMs = Date.now() - start;
    const rows = response.data?.output?.[0]?.records?.rows;

    if (!rows || rows.length === 0) {
      return res.status(503).json({ status: 'degraded', reason: 'no_query_result' });
    }

    const threshold = parseInt(process.env.QUERY_LATENCY_THRESHOLD_MS || '10000');
    if (latencyMs > threshold) {
      return res.status(503).json({
        status: 'degraded',
        reason: 'query_latency_high',
        latency_ms: latencyMs,
        threshold_ms: threshold,
      });
    }

    return res.status(200).json({ status: 'ok', query_latency_ms: latencyMs });
  } catch (err) {
    return res.status(503).json({ status: 'down', error: err.message });
  }
});

// Object storage health — check GreptimeDB internal metrics for storage errors
app.get('/health/greptimedb/storage', async (req, res) => {
  try {
    const response = await axios.get(`${GREPTIMEDB_HTTP}/metrics`, { timeout: 10000 });
    const metricsText = response.data;

    // Parse relevant Prometheus metrics from the text exposition format
    const parseMetric = (name) => {
      const regex = new RegExp(`^${name}\\s+(\\S+)`, 'm');
      const match = metricsText.match(regex);
      return match ? parseFloat(match[1]) : null;
    };

    const storageErrors = parseMetric('greptime_object_store_request_errors_total');
    const storageRequests = parseMetric('greptime_object_store_request_total');

    let errorRate = 0;
    if (storageRequests && storageRequests > 0 && storageErrors !== null) {
      errorRate = (storageErrors / storageRequests) * 100;
    }

    const threshold = parseFloat(process.env.STORAGE_ERROR_RATE_THRESHOLD || '1.0');
    if (errorRate > threshold) {
      return res.status(503).json({
        status: 'degraded',
        reason: 'storage_error_rate_high',
        error_rate_pct: errorRate.toFixed(2),
        threshold_pct: threshold,
      });
    }

    return res.status(200).json({
      status: 'ok',
      storage_error_rate_pct: errorRate.toFixed(2),
    });
  } catch (err) {
    return res.status(503).json({ status: 'down', error: err.message });
  }
});

// Compaction health — detect compaction falling behind
app.get('/health/greptimedb/compaction', async (req, res) => {
  try {
    const response = await axios.get(`${GREPTIMEDB_HTTP}/metrics`, { timeout: 10000 });
    const metricsText = response.data;

    const parseMetric = (name) => {
      const regex = new RegExp(`^${name}\\s+(\\S+)`, 'm');
      const match = metricsText.match(regex);
      return match ? parseFloat(match[1]) : null;
    };

    const pendingCompactions = parseMetric('greptime_mito_compaction_pending_tasks');
    const threshold = parseInt(process.env.COMPACTION_PENDING_THRESHOLD || '50');

    if (pendingCompactions !== null && pendingCompactions > threshold) {
      return res.status(503).json({
        status: 'degraded',
        reason: 'compaction_backlog_high',
        pending_compactions: pendingCompactions,
        threshold,
      });
    }

    return res.status(200).json({
      status: 'ok',
      pending_compactions: pendingCompactions ?? 'unknown',
    });
  } catch (err) {
    return res.status(503).json({ status: 'down', error: err.message });
  }
});

app.listen(3012, () => console.log('GreptimeDB health aggregator listening on :3012'));

Python Alternative

# greptimedb_health.py
from flask import Flask, jsonify
import requests, os, time

app = Flask(__name__)

GREPTIMEDB_HTTP = os.environ.get('GREPTIMEDB_HTTP_URL', 'http://localhost:4000')

@app.get('/health/greptimedb')
def health():
    try:
        r = requests.get(f'{GREPTIMEDB_HTTP}/health', timeout=5)
        r.raise_for_status()
        data = r.json()
        if data.get('status') != 'ok':
            return jsonify(status='degraded', upstream=data), 503
        return jsonify(status='ok'), 200
    except Exception as e:
        return jsonify(status='down', error=str(e)), 503

@app.get('/health/greptimedb/query')
def query_health():
    start = time.time()
    try:
        r = requests.get(f'{GREPTIMEDB_HTTP}/v1/sql?sql=SELECT+1+AS+health', timeout=10)
        r.raise_for_status()
        latency_ms = int((time.time() - start) * 1000)
        return jsonify(status='ok', query_latency_ms=latency_ms), 200
    except Exception as e:
        return jsonify(status='down', error=str(e)), 503

if __name__ == '__main__':
    app.run(port=3012)

Step 3: Configure Vigilmon Monitors for GreptimeDB

HTTP Monitor — Frontend Health

  1. Log in to vigilmon.online and go to Monitors → New Monitor
  2. Choose HTTP / HTTPS
  3. URL: http://your-greptimedb-host.example.com:4000/health (or: https://your-greptimedb-host.example.com/health/greptimedb via sidecar)
  4. Interval: 1 minute
  5. Expected response:
    • Status code: 200
    • Response body contains: "status":"ok"
    • Response time threshold: 5000ms
  6. Alert channel: P1 (frontend down = zero ingestion and queries)
  7. Save

TCP Port Monitor — gRPC (Port 4001)

High-throughput OTLP and Prometheus remote-write ingestion flows through gRPC port 4001. A TCP probe catches gRPC failures that the HTTP health check won't detect:

  1. Create a new monitor, type TCP Port
  2. Host: your-greptimedb-host.example.com, Port: 4001
  3. Interval: 1 minute
  4. Alert channel: P1 (gRPC down = OTLP ingestion stopped)

TCP Port Monitor — MySQL Protocol (Port 4002)

Grafana's GreptimeDB data source plugin and SQL analytics clients connect on MySQL port 4002:

  1. Create a new monitor, type TCP Port
  2. Host: your-greptimedb-host.example.com, Port: 4002
  3. Interval: 1 minute
  4. Alert channel: P2 (MySQL port down = SQL analytics clients blocked)

TCP Port Monitor — PostgreSQL Protocol (Port 4003)

psql, pgAdmin, and JDBC-based clients use port 4003:

  1. Create a new monitor, type TCP Port
  2. Host: your-greptimedb-host.example.com, Port: 4003
  3. Interval: 1 minute
  4. Alert channel: P2

HTTP Monitor — Ingestion Latency

  • URL: https://your-greptimedb-host.example.com/health/greptimedb/ingestion
  • Expected: 200, body contains "status":"ok"
  • Response time threshold: 10000ms
  • Interval: 2 minutes
  • Alert channel: P1 (write failures mean data loss)

HTTP Monitor — Query Latency

  • URL: https://your-greptimedb-host.example.com/health/greptimedb/query
  • Expected: 200, body contains "status":"ok"
  • Response time threshold: 15000ms
  • Interval: 5 minutes
  • Alert channel: P2

HTTP Monitor — Object Storage Health

  • URL: https://your-greptimedb-host.example.com/health/greptimedb/storage
  • Expected: 200, body contains "status":"ok"
  • Interval: 2 minutes
  • Alert channel: P1 (storage errors cause write failures)

HTTP Monitor — Compaction Backlog

  • URL: https://your-greptimedb-host.example.com/health/greptimedb/compaction
  • Expected: 200, body contains "status":"ok"
  • Interval: 10 minutes
  • Alert channel: P2

Step 4: Heartbeat Monitoring for Remote-Write Senders

Prometheus scrapes and remote-writes data to GreptimeDB on a schedule. If the remote-write sender stops (Prometheus crash, network partition), GreptimeDB can look healthy while your metrics pipeline silently gaps. Wire a Vigilmon heartbeat into the remote-write process:

Set Up the Heartbeat Monitor

  1. In Vigilmon, go to Monitors → New Monitor → Heartbeat
  2. Name: prometheus-remote-write-to-greptimedb
  3. Expected interval: 5 minutes (Prometheus typically scrapes every 15s–1m; 5m gives headroom)
  4. Grace period: 10 minutes
  5. Save — copy the heartbeat URL

Add Remote-Write Health Check to Prometheus Alertmanager

Create a Prometheus alert rule that fires a webhook to Vigilmon when remote-write is functioning:

# prometheus/rules/greptimedb-remotewrite.yml
groups:
  - name: greptimedb_remote_write
    interval: 2m
    rules:
      - record: greptimedb_remote_write_health
        expr: rate(prometheus_remote_storage_samples_total{remote_name="greptimedb"}[5m]) > 0

Or use a Prometheus lifecycle webhook script:

#!/bin/bash
# /opt/scripts/check-remote-write.sh
# Run via cron every 3 minutes

PROM_URL="${PROMETHEUS_URL:-http://localhost:9090}"
HEARTBEAT_URL="${VIGILMON_HEARTBEAT_URL}"

# Check that samples are flowing to GreptimeDB remote-write
RATE=$(curl -s "${PROM_URL}/api/v1/query?query=rate(prometheus_remote_storage_samples_total{remote_name=\"greptimedb\"}[5m])" \
  | python3 -c "import sys,json; data=json.load(sys.stdin); print(data['data']['result'][0]['value'][1] if data['data']['result'] else '0')" 2>/dev/null)

if [ "$(echo "$RATE > 0" | bc -l)" = "1" ]; then
  curl -s "$HEARTBEAT_URL" > /dev/null
fi

Step 5: Alert Routing for GreptimeDB

| Monitor | Alert Channel | Priority | |---|---|---| | Frontend health /health | Slack + PagerDuty | P1 | | gRPC port 4001 (TCP) | Slack + PagerDuty | P1 | | Object storage health | Slack + PagerDuty | P1 | | Ingestion latency | Slack + PagerDuty | P1 | | MySQL port 4002 (TCP) | Slack | P2 | | PostgreSQL port 4003 (TCP) | Slack | P2 | | Query latency | Slack | P2 | | Compaction backlog | Slack | P2 | | Heartbeat: remote-write sender | Slack + email | P2 |

Configure response time thresholds for early warning:

  • Alert at 3000ms on the /health endpoint (slow health checks precede full frontend overload)
  • Alert at 8000ms on write latency (P95 rising signals object storage pressure before write failures)

Summary

GreptimeDB's unified architecture simplifies your observability stack — but a single component failure (frontend down, gRPC port closed, object storage errors) can silently stop all metric ingestion. External monitoring with Vigilmon catches these failures before Grafana dashboards go stale:

| Monitor Type | What It Covers | |---|---| | HTTP monitor on /health | Frontend process health | | TCP port monitor on 4001 | gRPC ingestion availability | | TCP port monitor on 4002 | MySQL protocol for SQL analytics clients | | TCP port monitor on 4003 | PostgreSQL protocol for psql/pgAdmin | | HTTP monitor on ingestion endpoint | Write latency and error rate | | HTTP monitor on storage endpoint | Object storage (S3/MinIO) health | | Heartbeat monitor | Prometheus remote-write pipeline liveness |

Get started free at vigilmon.online — your first GreptimeDB monitor is running in under two minutes.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles