Monitoring Databend with Vigilmon

Databend is an open-source cloud-native data warehouse built on Apache Arrow and designed to be a Snowflake-compatible alternative. It supports large-scale analytics workloads with columnar storage, elastic compute, and a familiar SQL interface. When a Databend cluster node fails or the meta service loses quorum, queries fail silently for end users. Vigilmon gives you health endpoint monitoring, cluster node checks, TCP port verification, and SSL certificate alerts across your Databend deployment.

What You'll Set Up

HTTP monitor for the /v1/health endpoint
Query handler port availability check (default port 8000)
Cluster node health monitoring via /v1/cluster
TCP port checks for the meta service (9191) and Flight SQL (8900)
SSL certificate expiry alerts for production deployments

Prerequisites

Databend running in standalone or cluster mode (v1.0+)
Databend HTTP handler accessible on port 8000
A free Vigilmon account

Step 1: Monitor the /v1/health Endpoint

Databend exposes a /v1/health HTTP endpoint on its query handler port (default 8000). This is the primary health signal for the Databend service:

Log in to vigilmon.online and click Add Monitor.
Set Type to HTTP / HTTPS.
Enter the health endpoint URL: http://databend-host:8000/v1/health
Set Check interval to 1 minute.
Set Expected HTTP status to 200.
Click Save.

A healthy Databend node returns HTTP 200 from /v1/health. If the node is starting up, shutting down, or has lost connectivity to the meta service, this endpoint will return a non-200 status or fail to respond entirely.

Step 2: Monitor the Query Handler HTTP Port

The query handler on port 8000 is the primary interface for SQL queries over HTTP. Beyond the health check, verify the port itself is accepting connections:

Click Add Monitor → HTTP / HTTPS.
Set the URL to http://databend-host:8000/.
Set Expected HTTP status to 200 or 404 — the root path may return 404 but the service is still running.
Set Check interval to 1 minute.
Click Save.

Alternatively, use a TCP monitor to check raw port availability without depending on the HTTP response body:

Click Add Monitor → TCP Port.
Enter Host: databend-host and Port: 8000.
Set Check interval to 1 minute.
Click Save.

Step 3: Monitor Cluster Node Health via /v1/cluster

In a Databend cluster deployment, multiple query nodes work together. The /v1/cluster endpoint returns the list of registered cluster nodes and their status. Monitor this endpoint to detect node dropouts:

Click Add Monitor → HTTP / HTTPS.
Set the URL to http://databend-host:8000/v1/cluster.
Set Expected HTTP status to 200.
Optionally, set Expected response body to contain "id" to confirm the response includes node data.
Set Check interval to 2 minutes.
Click Save.

If a node leaves the cluster unexpectedly, this endpoint may return an empty node list or a degraded-state response, depending on your Databend version. Combine this with alerts routed to your on-call channel for immediate cluster health visibility.

Step 4: TCP Port Checks for Meta Service and Flight SQL

Databend runs two additional services that are critical for cluster operation:

Meta service (default port 9191): Manages cluster metadata, catalog, and schema. If the meta service is unreachable, query nodes can't coordinate.
Flight SQL (default port 8900): Apache Arrow Flight SQL interface for high-performance query clients.

Add a TCP monitor for each:

Meta Service (Port 9191)

Click Add Monitor → TCP Port.
Enter Host: databend-meta-host and Port: 9191.
Set Check interval to 1 minute.
Click Save.

Flight SQL (Port 8900)

Click Add Monitor → TCP Port.
Enter Host: databend-host and Port: 8900.
Set Check interval to 1 minute.
Click Save.

These TCP checks detect port-level failures even when the HTTP handler on 8000 is still responsive — a common split-brain scenario in distributed data systems.

Step 5: SSL Certificate Monitoring

For Databend deployments behind a TLS-terminating reverse proxy (nginx, Caddy, or a load balancer), add SSL certificate expiry monitoring:

Open the HTTP / HTTPS monitor for your Databend health endpoint.
Enable Monitor SSL certificate under the SSL section.
Set Alert when certificate expires in less than 21 days.
Click Save.

A typical nginx TLS proxy configuration for Databend:

server {
    listen 443 ssl;
    server_name databend.yourdomain.com;

    ssl_certificate /etc/letsencrypt/live/databend.yourdomain.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/databend.yourdomain.com/privkey.pem;

    location /v1/ {
        proxy_pass http://127.0.0.1:8000;
    }
}

Vigilmon's SSL monitor will alert you if the Let's Encrypt certificate renewal fails — before your analytics clients start seeing TLS errors.

Step 6: Configure Alert Channels

Go to Alert Channels in Vigilmon and add Slack, email, or PagerDuty via webhook.
Set Consecutive failures before alert to 2 for HTTP monitors to avoid false positives from transient network issues.
Set TCP port monitors to alert on the first failure — a closed port is rarely transient and indicates a service crash.
Route cluster health alerts (/v1/cluster) to your data engineering on-call channel separately from general uptime alerts.

Summary

| Monitor | Target | What It Catches | |---|---|---| | Health endpoint | http://databend-host:8000/v1/health | Service startup failures, meta connection loss | | Query handler | http://databend-host:8000/ | HTTP handler crash | | Cluster nodes | http://databend-host:8000/v1/cluster | Node dropout, cluster degradation | | Meta service TCP | databend-meta-host:9191 | Meta service crash | | Flight SQL TCP | databend-host:8900 | Flight SQL interface unavailable | | SSL certificate | Production HTTPS endpoint | Certificate renewal failure |

Databend's distributed architecture means failures can be partial — a single node or service going down degrades queries without an obvious error. Vigilmon's combination of health endpoint checks, TCP port monitors, and cluster API verification gives you full-stack visibility into your Databend cluster's health.