tutorial

How to Monitor StarRocks with Vigilmon

StarRocks (formerly DorisDB) delivers MySQL-compatible analytical queries at petabyte scale — but FE node failures, BE storage pressure, and replica imbalance are invisible without external monitoring. Learn how to monitor StarRocks end-to-end with Vigilmon.

StarRocks (formerly known as DorisDB) is a high-performance MPP analytical database built for real-time and ad hoc queries over massive datasets. Its MySQL protocol compatibility makes it easy to adopt — but its distributed architecture of Frontend (FE) and Backend (BE) nodes means there are failure modes that look silent from the inside. A BE node losing disk space silently degrades replica health. An FE follower falling behind the leader log causes query routing errors that surface only under load. Load job failures quietly stop data from arriving.

Vigilmon gives you external visibility into StarRocks health through HTTP probe monitoring and heartbeat monitors for your load pipelines. This tutorial covers FE node health, BE storage health, query latency percentiles, load job success rate, replica balance metrics, and Vigilmon integration.

Why StarRocks Needs External Monitoring

StarRocks provides built-in system tables (in information_schema) and an HTTP web UI — but both require an active connection and functioning FE nodes to query. External monitoring with Vigilmon adds:

  • FE node liveness probes that detect follower failures and leader failover events
  • BE storage pressure alerting before disk fullness causes tablet errors and replica loss
  • Query latency percentile monitoring that catches degradation earlier than average-latency metrics
  • Load job failure detection for Routine Load, Broker Load, and Stream Load pipelines
  • Replica health checks that detect under-replicated tablets before they cause query errors

The critical blind spot: StarRocks query routing silently degrades when BE node count drops. Queries continue to execute but with fewer replicas available for parallel execution — latency climbs and you won't see an explicit error until a node threshold is crossed.

Step 1: Build StarRocks Health Endpoints

StarRocks FE and BE nodes expose HTTP APIs on dedicated ports. These are your first probe targets.

Native StarRocks Health Endpoints

# FE node health (default port 8030)
curl -i http://starrocks-fe:8030/api/health

# FE node leader check
curl -i http://starrocks-fe:8030/api/show_proc?path=/

# BE node health (default port 8040)
curl -i http://starrocks-be:8040/api/health

The FE /api/health returns {"status": "OK"} with HTTP 200 when the FE is healthy and the leader is reachable.

FE Cluster Health Sidecar

For multi-FE deployments, check that a leader is elected and followers are in sync:

# starrocks_health.py
from flask import Flask, jsonify
import requests
import pymysql
import os

app = Flask(__name__)

FE_HTTP_URL = os.environ.get('SR_FE_HTTP_URL', 'http://localhost:8030')
SR_HOST = os.environ.get('SR_HOST', 'localhost')
SR_PORT = int(os.environ.get('SR_PORT', '9030'))
SR_USER = os.environ.get('SR_USER', 'root')
SR_PASSWORD = os.environ.get('SR_PASSWORD', '')

def get_mysql_connection():
    return pymysql.connect(
        host=SR_HOST,
        port=SR_PORT,
        user=SR_USER,
        password=SR_PASSWORD,
        database='information_schema',
        connect_timeout=5
    )

@app.route('/health/starrocks/fe')
def fe_health():
    try:
        resp = requests.get(f'{FE_HTTP_URL}/api/health', timeout=5)
        if resp.status_code != 200:
            return jsonify({'status': 'down', 'error': f'HTTP {resp.status_code}'}), 503

        data = resp.json()
        if data.get('status') != 'OK':
            return jsonify({'status': 'degraded', 'reason': data.get('status')}), 503

        # Check FE follower health via SQL
        conn = get_mysql_connection()
        with conn.cursor() as cur:
            cur.execute("SHOW PROC '/frontends'")
            rows = cur.fetchall()

        fe_nodes = [dict(zip([d[0] for d in cur.description], row)) for row in rows]
        conn.close()

        unhealthy_fe = [
            fe for fe in fe_nodes
            if fe.get('Alive', '').lower() != 'true'
        ]

        if unhealthy_fe:
            return jsonify({
                'status': 'degraded',
                'reason': 'fe_nodes_down',
                'unhealthy': [fe.get('Name') for fe in unhealthy_fe]
            }), 503

        return jsonify({'status': 'ok', 'fe_count': len(fe_nodes)}), 200

    except Exception as e:
        return jsonify({'status': 'down', 'error': str(e)}), 503

BE Storage Health

Check BE node disk usage and tablet health before storage pressure causes replica errors:

@app.route('/health/starrocks/be')
def be_health():
    disk_threshold = float(os.environ.get('SR_DISK_THRESHOLD_PCT', '85'))
    try:
        conn = get_mysql_connection()
        with conn.cursor() as cur:
            # Check BE node status
            cur.execute("SHOW PROC '/backends'")
            rows = cur.fetchall()
            cols = [d[0] for d in cur.description]

        backends = [dict(zip(cols, row)) for row in rows]
        conn.close()

        issues = []
        for be in backends:
            if be.get('Alive', '').lower() != 'true':
                issues.append({'backend': be.get('BackendId'), 'reason': 'not_alive'})
                continue

            # Parse disk usage
            disk_used = be.get('DataUsedCapacity', '0.000 GB')
            disk_total = be.get('AvailCapacity', '0.000 GB')
            # Note: in production parse these properly from the response

        if issues:
            return jsonify({'status': 'degraded', 'issues': issues}), 503

        return jsonify({'status': 'ok', 'backends': len(backends)}), 200

    except Exception as e:
        return jsonify({'status': 'down', 'error': str(e)}), 503

Query Latency Percentile Health

@app.route('/health/starrocks/query')
def query_latency_health():
    latency_p99_threshold_ms = int(os.environ.get('SR_QUERY_P99_THRESHOLD_MS', '3000'))
    try:
        conn = get_mysql_connection()
        with conn.cursor() as cur:
            # Use information_schema.query_profile or a test query
            import time
            start = time.time()
            cur.execute("SELECT COUNT(*) FROM information_schema.tables LIMIT 1")
            cur.fetchone()
            latency_ms = int((time.time() - start) * 1000)

        conn.close()

        if latency_ms > latency_p99_threshold_ms:
            return jsonify({
                'status': 'degraded',
                'reason': 'high_query_latency',
                'latency_ms': latency_ms
            }), 503

        return jsonify({'status': 'ok', 'latency_ms': latency_ms}), 200

    except Exception as e:
        return jsonify({'status': 'down', 'error': str(e)}), 503

Replica Balance Health

Under-replicated tablets indicate BE node pressure or recent node loss:

@app.route('/health/starrocks/replicas')
def replica_health():
    try:
        conn = get_mysql_connection()
        with conn.cursor() as cur:
            cur.execute("""
                SELECT
                    ReplicaState,
                    COUNT(*) as count
                FROM information_schema.be_tablets
                GROUP BY ReplicaState
            """)
            rows = cur.fetchall()
        conn.close()

        state_counts = {row[0]: row[1] for row in rows}
        bad_states = {
            state: count for state, count in state_counts.items()
            if state not in ('NORMAL',)
        }

        if bad_states:
            return jsonify({
                'status': 'degraded',
                'reason': 'unhealthy_replicas',
                'states': bad_states
            }), 503

        return jsonify({'status': 'ok', 'replica_states': state_counts}), 200

    except Exception as e:
        return jsonify({'status': 'down', 'error': str(e)}), 503

Load Job Success Rate

@app.route('/health/starrocks/load')
def load_health():
    try:
        conn = get_mysql_connection()
        with conn.cursor() as cur:
            # Check recent Routine Load job status
            cur.execute("""
                SELECT Name, State, ErrorLogUrls
                FROM information_schema.routine_load_jobs
                WHERE State NOT IN ('CANCELLED', 'STOPPED')
            """)
            rows = cur.fetchall()
        conn.close()

        paused_jobs = [row[0] for row in rows if row[1] == 'PAUSED']

        if paused_jobs:
            return jsonify({
                'status': 'degraded',
                'reason': 'routine_load_paused',
                'jobs': paused_jobs
            }), 503

        return jsonify({'status': 'ok', 'active_jobs': len(rows)}), 200

    except Exception as e:
        return jsonify({'status': 'down', 'error': str(e)}), 503

if __name__ == '__main__':
    app.run(port=9091)

Step 2: Configure Vigilmon HTTP Monitors for StarRocks

  1. Log in to vigilmon.online and go to Monitors → New Monitor
  2. Choose HTTP / HTTPS
  3. Add monitors for each health surface:

| Monitor Name | URL | Expected | Interval | |---|---|---|---| | StarRocks FE liveness | http://sr-fe:8030/api/health | 200, body "OK" | 1 min | | StarRocks FE cluster | https://your-sidecar/health/starrocks/fe | 200, body "ok" | 2 min | | StarRocks BE health | http://sr-be:8040/api/health | 200 | 1 min | | StarRocks BE storage | https://your-sidecar/health/starrocks/be | 200, body "ok" | 5 min | | StarRocks query latency | https://your-sidecar/health/starrocks/query | 200, body "ok" | 1 min | | StarRocks replica health | https://your-sidecar/health/starrocks/replicas | 200, body "ok" | 5 min | | StarRocks load jobs | https://your-sidecar/health/starrocks/load | 200, body "ok" | 2 min |

Set the response time alert on the query latency monitor to 2000ms — earlier than the P99 threshold so you get a warning before the threshold fires.

Step 3: Heartbeat Monitoring for Load Pipelines

StarRocks Routine Load runs continuously inside the cluster — but the data producers that feed Kafka (or write directly via Stream Load) can stall without touching StarRocks at all. Vigilmon heartbeats detect upstream stalls.

Set Up the Heartbeat Monitor

  1. In Vigilmon, go to Monitors → New Monitor → Heartbeat
  2. Set the name: starrocks-stream-load-pipeline
  3. Set the expected interval: 10 minutes
  4. Set the grace period: 20 minutes
  5. Save — copy the heartbeat URL

Stream Load Pipeline with Heartbeat (Python)

# starrocks_stream_load.py
import os
import requests
import json
import time

SR_HOST = os.environ['SR_HOST']
SR_HTTP_PORT = os.environ.get('SR_HTTP_PORT', '8030')
SR_USER = os.environ['SR_USER']
SR_PASSWORD = os.environ['SR_PASSWORD']
SR_DATABASE = os.environ['SR_DATABASE']
SR_TABLE = os.environ['SR_TABLE']
VIGILMON_HEARTBEAT_URL = os.environ['VIGILMON_HEARTBEAT_URL']

def stream_load_batch(data: list) -> bool:
    """Load a batch of records via StarRocks Stream Load."""
    json_data = '\n'.join(json.dumps(row) for row in data)

    response = requests.put(
        f'http://{SR_HOST}:{SR_HTTP_PORT}/api/{SR_DATABASE}/{SR_TABLE}/_stream_load',
        auth=(SR_USER, SR_PASSWORD),
        headers={
            'Expect': '100-continue',
            'format': 'json',
            'strip_outer_array': 'false',
            'Content-Type': 'application/json',
        },
        data=json_data,
        timeout=60
    )

    result = response.json()
    return result.get('Status') == 'Success'

def run_pipeline():
    last_heartbeat = time.time()
    HEARTBEAT_INTERVAL = 10 * 60  # 10 minutes

    while True:
        try:
            batch = fetch_next_batch()
            if batch:
                success = stream_load_batch(batch)
                if success:
                    now = time.time()
                    if now - last_heartbeat >= HEARTBEAT_INTERVAL:
                        requests.get(VIGILMON_HEARTBEAT_URL, timeout=5)
                        last_heartbeat = now
                else:
                    print("Stream load failed — skipping heartbeat")
        except Exception as e:
            print(f"Pipeline error: {e}")

        time.sleep(30)

Broker Load Job Wrapper

For scheduled Broker Load jobs (S3, HDFS), wrap the job submission:

#!/bin/bash
# broker_load_with_heartbeat.sh

LABEL="load_$(date +%Y%m%d_%H%M%S)"
SR_HOST=${SR_HOST:-localhost}
SR_PORT=${SR_PORT:-9030}
VIGILMON_URL=${VIGILMON_HEARTBEAT_URL}

mysql -h "$SR_HOST" -P "$SR_PORT" -u root << EOF
LOAD LABEL ${SR_DATABASE}.${LABEL}
(
    DATA INFILE("s3://${S3_BUCKET}/data/*.parquet")
    INTO TABLE ${SR_TABLE}
    FORMAT AS "parquet"
)
WITH BROKER "s3" (
    "aws.s3.access_key" = "${AWS_ACCESS_KEY}",
    "aws.s3.secret_key" = "${AWS_SECRET_KEY}",
    "aws.s3.region" = "${AWS_REGION}"
);
EOF

# Poll for job completion
while true; do
    STATE=$(mysql -h "$SR_HOST" -P "$SR_PORT" -u root -se \
        "SHOW LOAD WHERE Label='${LABEL}'\G" | grep State | awk '{print $2}')

    case "$STATE" in
        FINISHED)
            curl -s "$VIGILMON_URL" > /dev/null
            echo "Load job finished, heartbeat sent"
            exit 0
            ;;
        CANCELLED)
            echo "Load job cancelled — no heartbeat"
            exit 1
            ;;
    esac
    sleep 10
done

Step 4: Alert Routing for StarRocks Failures

StarRocks failure severity depends on which tier is affected:

  • FE leader failure: all DDL and DML stop; queries continue briefly then fail as routing breaks
  • BE failure: query capacity shrinks; replica rebalancing begins but takes time
  • Replica degradation: queries may return incomplete results with degraded table availability
  • Load job failure: data freshness silently degrades

Configure alert routing in Vigilmon:

| Monitor | Alert Channel | Priority | |---|---|---| | StarRocks FE liveness | Slack + PagerDuty | P1 | | StarRocks query latency | Slack + PagerDuty | P1 | | StarRocks BE health | Slack | P2 | | StarRocks replica health | Slack | P2 | | StarRocks load jobs | Slack + email | P2 | | StarRocks BE storage | Email | P3 | | StarRocks stream load heartbeat | Slack | P2 |

Add a response time escalation on the FE liveness monitor: alert at 1000ms for early warning of FE pressure before follower lag causes visible issues.

Summary

StarRocks reliability depends on the health of both FE coordination and BE storage tiers — and load pipeline health is entirely outside what internal monitoring can see. Vigilmon covers all three:

| Monitor Type | What It Covers | |---|---| | HTTP probe on FE/BE /api/health | Service process liveness | | HTTP probe on query sidecar | End-to-end query latency | | HTTP probe on replica sidecar | Tablet replica balance | | HTTP probe on load sidecar | Routine Load and Stream Load health | | Heartbeat monitor | Data pipeline and Broker Load liveness |

Get started free at vigilmon.online — your first StarRocks monitor is live in under two minutes.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles