tutorial

How to Monitor RisingWave with Vigilmon

"A step-by-step guide to monitoring your RisingWave cloud-native streaming SQL database with health checks, materialized view lag tracking, and Vigilmon integration."

How to Monitor RisingWave with Vigilmon

RisingWave is a cloud-native streaming SQL database designed for real-time analytics. It lets you write SQL queries over live data streams and materialized views that update continuously. That power comes with operational complexity: nodes fail, materialized views fall behind, connectors stall, and checkpoints miss their windows. Without the right monitoring in place, you find out about these problems from your users rather than your dashboards.

This tutorial walks through building a comprehensive monitoring setup for RisingWave—covering node health, materialized view refresh lag, connector health, checkpoint success rates, and compute metrics—and integrating everything with Vigilmon for alerting.

Prerequisites

  • RisingWave running (local Docker, Kubernetes, or RisingWave Cloud)
  • Vigilmon account at vigilmon.online
  • psql or any PostgreSQL-compatible client
  • curl for HTTP checks

Understanding RisingWave's Architecture

RisingWave runs as a cluster of specialized nodes:

  • Frontend nodes accept SQL connections and route queries
  • Compute nodes execute streaming operators and maintain state
  • Meta nodes coordinate cluster state, manage checkpoints, and orchestrate DDL
  • Compactor nodes compact storage in the background

Each node exposes metrics over HTTP at port 1260 (Prometheus format) and accepts SQL connections on port 4566. Your monitoring strategy needs to cover both layers.

Step 1: Verify Node Health via HTTP

Every RisingWave node exposes a health endpoint. The simplest check confirms a node is alive and accepting requests.

# Check frontend node health
curl -sf http://risingwave-frontend:1260/health
# Returns 200 OK when healthy

# Check meta node health
curl -sf http://risingwave-meta:1260/health

# Check compute node health
curl -sf http://risingwave-compute:1260/health

For a multi-node cluster, script this across all node addresses:

#!/bin/bash
NODES=(
  "risingwave-frontend:1260"
  "risingwave-meta:1260"
  "risingwave-compute-0:1260"
  "risingwave-compute-1:1260"
  "risingwave-compactor:1260"
)

for node in "${NODES[@]}"; do
  status=$(curl -sf -o /dev/null -w "%{http_code}" "http://$node/health")
  if [ "$status" != "200" ]; then
    echo "UNHEALTHY: $node returned $status"
    exit 1
  fi
done
echo "All nodes healthy"

Step 2: Monitor Materialized View Refresh Lag

Materialized views are the core of RisingWave's value proposition. When they fall behind their source streams, downstream queries return stale data. Query the rw_fragments and rw_materialized_views system tables to track freshness:

-- Connect to RisingWave
psql -h risingwave-frontend -p 4566 -U root dev

-- Check materialized view definitions and status
SELECT
  id,
  name,
  schema_id,
  definition
FROM rw_materialized_views
ORDER BY name;

RisingWave exposes watermark and lag metrics via Prometheus. Scrape compute node metrics to track barrier lag, which indicates how far behind the streaming computation is:

# Pull streaming barrier latency from a compute node
curl -s http://risingwave-compute:1260/metrics | grep barrier_latency_seconds

A healthy cluster shows barrier latency under 1 second. Sustained latency above 5 seconds means backpressure is building or a compute node is struggling.

Write a lag check script:

#!/bin/bash
MAX_LAG_SECONDS=10

lag=$(curl -s http://risingwave-compute:1260/metrics \
  | grep 'stream_barrier_latency_seconds_bucket' \
  | awk -F' ' '{print $NF}' \
  | sort -n | tail -1)

if (( $(echo "$lag > $MAX_LAG_SECONDS" | bc -l) )); then
  echo "CRITICAL: barrier lag ${lag}s exceeds threshold ${MAX_LAG_SECONDS}s"
  exit 2
fi
echo "OK: barrier lag ${lag}s"

Step 3: Monitor Connector Source and Sink Health

RisingWave connects to external systems via sources (Kafka, Kinesis, S3) and sinks (Kafka, JDBC, Iceberg). A stalled connector silently drops data. Check connector status through the system catalog:

-- List all sources and their connector types
SELECT
  id,
  name,
  connector,
  schema_id
FROM rw_sources
ORDER BY name;

-- List all sinks
SELECT
  id,
  name,
  connector,
  schema_id
FROM rw_sinks
ORDER BY name;

For Kafka-backed sources, the key metric is consumer group lag. RisingWave creates a consumer group per source. Check it with Kafka tools:

kafka-consumer-groups.sh \
  --bootstrap-server kafka:9092 \
  --describe \
  --group risingwave-consumer-group-<source_id>

A lag of zero means the source is fully caught up. A growing lag means RisingWave cannot keep up with the Kafka topic's throughput—scale compute nodes or optimize your materialized view queries.

Step 4: Track Checkpoint Success Rate

RisingWave checkpoints its streaming state periodically. A failed checkpoint means the cluster cannot recover cleanly if a node crashes. Monitor checkpoint metrics from the meta node:

# Pull checkpoint metrics
curl -s http://risingwave-meta:1260/metrics | grep -E 'stream_barrier_manager|checkpoint'

Key metrics to watch:

  • stream_barrier_manager_progress: number of checkpoints completed
  • recovery_failure_count: how many times the cluster has had to recover
  • stream_barrier_latency_seconds: how long each checkpoint barrier takes to propagate

Build a checkpoint health check:

#!/bin/bash
# Fetch metrics and check for recent checkpoint failures
FAILURES=$(curl -s http://risingwave-meta:1260/metrics \
  | grep 'recovery_failure_count' \
  | grep -v '#' \
  | awk '{print $2}')

if [ -z "$FAILURES" ]; then
  echo "WARNING: Could not read checkpoint metrics"
  exit 1
fi

if [ "$FAILURES" -gt "0" ]; then
  echo "CRITICAL: $FAILURES recovery failures detected"
  exit 2
fi

echo "OK: No checkpoint failures"

Step 5: Monitor Compute Node CPU and Memory

Compute nodes do the heavy lifting. When they run hot, streaming latency climbs and you risk out-of-memory kills. Expose compute resource metrics:

# CPU usage across compute nodes (from Prometheus metrics)
curl -s http://risingwave-compute:1260/metrics \
  | grep 'process_cpu_seconds_total'

# Memory usage
curl -s http://risingwave-compute:1260/metrics \
  | grep 'process_resident_memory_bytes'

For alerting thresholds, set memory warnings at 75% of the container limit and critical at 90%. CPU warnings at 80% sustained for 5 minutes.

If you run on Kubernetes, pair RisingWave's own metrics with kube-state-metrics to correlate pod restarts with metric spikes:

kubectl top pods -l app=risingwave --containers -n risingwave

Step 6: Set Up Vigilmon Checks

With your health checks validated locally, configure Vigilmon to run them on schedule and alert your team when something breaks.

Log in to vigilmon.online and navigate to Monitors → New Monitor.

Node Health Check

Create an HTTP check for each node type:

| Field | Value | |-------|-------| | Name | RisingWave Frontend Health | | Type | HTTP | | URL | http://risingwave-frontend:1260/health | | Method | GET | | Expected Status | 200 | | Interval | 60 seconds | | Timeout | 10 seconds |

Repeat for meta, compute, and compactor nodes.

Checkpoint Health Check

RisingWave doesn't expose a single pass/fail endpoint for checkpoints, so use Vigilmon's Script monitor type to run your checkpoint check:

#!/bin/bash
FAILURES=$(curl -s http://risingwave-meta:1260/health | python3 -c "
import sys, json
data = json.load(sys.stdin)
print(data.get('failed_checkpoints', 0))
" 2>/dev/null || echo "0")

[ "$FAILURES" = "0" ] && exit 0 || exit 1

Streaming Lag Check

Use Vigilmon's HTTP monitor to poll a custom lag-reporting endpoint you expose from your application:

# Simple Flask endpoint to expose lag status
from flask import Flask, jsonify
import subprocess, re

app = Flask(__name__)

@app.route('/risingwave/lag')
def check_lag():
    metrics = subprocess.check_output(
        ['curl', '-s', 'http://risingwave-compute:1260/metrics'],
        text=True
    )
    # Extract barrier latency p99
    match = re.search(r'stream_barrier_latency_seconds\{.*\} (\d+\.?\d*)', metrics)
    lag = float(match.group(1)) if match else 0
    status = 'ok' if lag < 5 else 'degraded'
    return jsonify({'lag_seconds': lag, 'status': status}), 200 if status == 'ok' else 503

if __name__ == '__main__':
    app.run(port=8080)

Then add a Vigilmon HTTP monitor pointing to http://your-app:8080/risingwave/lag expecting status 200.

Alert Configuration

Set up notification channels in Vigilmon under Settings → Notifications:

  • Email: ops team distribution list
  • Slack: #platform-alerts channel
  • PagerDuty: for checkpoint failures and compute OOM (P1)

Configure escalation: warn after 1 failure, page after 3 consecutive failures in 5 minutes.

Step 7: Build a Runbook

Attach a runbook to each Vigilmon alert so on-call engineers know what to do:

Node down:

  1. Check pod logs: kubectl logs -l component=frontend -n risingwave --tail=100
  2. Check node resource pressure: kubectl describe node <node>
  3. Restart the pod if OOM: kubectl delete pod <pod> -n risingwave

High barrier lag:

  1. Check for hot partitions in Kafka sources
  2. Review compute node CPU: kubectl top pods -n risingwave
  3. Scale compute nodes: increase replicas in the RisingWave Helm chart

Checkpoint failures:

  1. Check meta node logs for barrier timeout messages
  2. Verify object storage (S3/MinIO) is accessible from compute nodes
  3. Consider reducing checkpoint interval to reduce state size per checkpoint

What to Monitor at a Glance

| Check | Tool | Threshold | |-------|------|-----------| | Node HTTP health | Vigilmon HTTP | 200 OK | | Barrier/streaming lag | Custom endpoint | < 5 seconds | | Checkpoint failures | Script monitor | 0 failures | | Compute memory | Prometheus | < 90% of limit | | Kafka source lag | Script monitor | < 10k messages | | Connector sink errors | Log scraping | 0 errors/min |

Conclusion

RisingWave makes real-time SQL over streams accessible, but running it reliably in production requires monitoring at every layer. By combining RisingWave's built-in Prometheus metrics with Vigilmon's HTTP and script monitors, you get continuous visibility into node health, streaming lag, checkpoint integrity, and resource utilization. When something breaks, you find out immediately—before your users do.

Start with the node health checks in Vigilmon, add the barrier lag endpoint, and build out checkpoint monitoring as your cluster matures. Each monitor you add narrows the gap between "something went wrong" and "we know what went wrong and how to fix it."

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles