rqlite Monitoring with Vigilmon

rqlite is a distributed relational database built on SQLite. It uses the Raft consensus algorithm to replicate SQLite across a cluster of nodes, making SQLite durable and highly available for the first time. Because rqlite is built for lightweight, embedded-style deployments, teams use it in edge infrastructure, IoT backends, and microservices where a full Postgres cluster would be over-engineered.

Despite its simplicity, rqlite has real failure modes: leader elections can stall, nodes can fall behind in replication, and a misconfigured cluster can silently accept writes that are never committed. This guide covers how to monitor rqlite with Vigilmon using its built-in HTTP API.

rqlite's HTTP API

rqlite exposes an HTTP API on port 4001 by default. All monitoring endpoints are part of this API — no special monitoring port required.

/readyz — Readiness Probe

GET http://your-rqlite-host:4001/readyz

Response when the node is ready:

HTTP 200 OK

Response when the node is not ready (e.g., still joining the cluster):

HTTP 503 Service Unavailable

This is the canonical health check for rqlite. Use it as your primary liveness monitor.

/status — Node and Cluster Status

GET http://your-rqlite-host:4001/status

Response (abbreviated):

{
  "node": {
    "start_time": "2024-01-01T00:00:00Z",
    "current_time": "2024-01-01T01:00:00Z",
    "uptime": "1h0m0s"
  },
  "store": {
    "leader": {
      "addr": "your-rqlite-host:4002",
      "node_id": "node1"
    },
    "raft": {
      "state": "Leader",
      "num_peers": 2,
      "last_log_index": 42,
      "applied_index": 42,
      "commit_index": 42
    },
    "sqlite3": {
      "path": "/data/rqlite.db",
      "mem_stats": {}
    }
  },
  "http": {
    "addr": "0.0.0.0:4001"
  }
}

Key fields to watch:

• store.raft.state — should be Leader or Follower; Candidate indicates an ongoing election
• store.raft.applied_index vs store.raft.commit_index — if applied_index falls significantly behind commit_index, the node is lagging
• store.leader.addr — if empty, the cluster has no leader

/nodes — Cluster Membership

GET http://your-rqlite-host:4001/nodes

Response:

{
  "node1": {
    "api_addr": "http://node1:4001",
    "addr": "node1:4002",
    "voter": true,
    "reachable": true,
    "leader": true
  },
  "node2": {
    "api_addr": "http://node2:4001",
    "addr": "node2:4002",
    "voter": true,
    "reachable": true,
    "leader": false
  }
}

If reachable is false for any node, a cluster member is offline. Monitor this endpoint to catch node failures before they accumulate and cause a quorum loss.

Setting Up Vigilmon Monitors

Monitor 1: Node Readiness

1. Log in to Vigilmon → Monitors → New Monitor
2. Type: HTTP
3. Method: GET
4. URL: http://your-rqlite-host:4001/readyz
5. Interval: 1 minute
6. Expected status: 200

Create one monitor per rqlite node. In a 3-node cluster, that is 3 separate Vigilmon monitors — one per node. If any single node goes down, you want to know immediately.

Monitor 2: Leader Availability

The leader handles all write operations in an rqlite cluster. If the cluster has no leader, writes fail until a new one is elected.

1. Type: HTTP
2. Method: GET
3. URL: http://your-rqlite-host:4001/status
4. Interval: 1 minute
5. Keyword check: "state":"Leader" or "state":"Follower"

Since only one node will have "state":"Leader" at a time, monitor the /status endpoint on each node and alert if the overall cluster lacks a leader by checking for the absence of any "state":"Leader" across your monitors.

A simpler alternative is to point a load balancer at your rqlite nodes and monitor the balancer's health endpoint — but the per-node approach gives you more granularity.

Monitor 3: Cluster Node Reachability

1. Type: HTTP
2. Method: GET
3. URL: http://your-rqlite-host:4001/nodes
4. Interval: 2 minutes
5. Keyword check: "reachable":true

If "reachable":false appears in the response, a node has gone offline. Configure an alert to fire after 2 failed checks to avoid alerting on transient network blips during node restarts.

Monitor 4: Liveness Check (Alternative Port)

If you expose rqlite behind a reverse proxy or load balancer:

1. Type: HTTP
2. Method: GET
3. URL: https://rqlite.your-domain.com/readyz
4. Interval: 1 minute
5. Expected status: 200

Docker and Kubernetes Deployment Health Checks

Docker Compose

services:
  rqlite:
    image: rqlite/rqlite:latest
    ports:
      - "4001:4001"
      - "4002:4002"
    command: >
      rqlited
      -node-id node1
      -http-addr 0.0.0.0:4001
      -raft-addr 0.0.0.0:4002
      /var/lib/rqlite
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:4001/readyz"]
      interval: 30s
      timeout: 10s
      retries: 3
      start_period: 10s

Kubernetes Probes

livenessProbe:
  httpGet:
    path: /readyz
    port: 4001
  initialDelaySeconds: 15
  periodSeconds: 30
  failureThreshold: 3

readinessProbe:
  httpGet:
    path: /readyz
    port: 4001
  initialDelaySeconds: 5
  periodSeconds: 10
  failureThreshold: 2

Kubernetes readiness probes prevent traffic from being sent to a node that is still joining the Raft cluster after a restart. Vigilmon provides the external perspective — it catches failures at the ingress or load balancer level that internal Kubernetes probes cannot observe.

Alert Configuration

Slack Alerts

In Vigilmon → Alert Channels → Slack, add your Slack webhook. For the /readyz monitors on each node, set alerts to trigger after 1 failed check — a node that fails the readiness probe may need to be replaced before the next scheduled Raft heartbeat window expires.

Webhook for Automated Recovery

app.post('/webhooks/vigilmon', express.json(), (req, res) => {
  const { event, monitor } = req.body;

  if (event === 'down' && monitor.url.includes('/readyz')) {
    const nodeHost = new URL(monitor.url).hostname;

    // Log for ops runbook
    console.error(`[rqlite] Node DOWN: ${nodeHost}`);

    // Optionally trigger auto-recovery: restart container, alert on-call
    notifyOncall({
      message: `rqlite node ${nodeHost} is unreachable`,
      severity: 'high',
    });
  }

  res.json({ received: true });
});

Alerting Strategy

| Scenario | Monitor | Alert Threshold | |----------|---------|-----------------| | Node unreachable | /readyz per node | 1 failed check | | No cluster leader | /status keyword Leader missing | 2 failed checks | | Node reachability drop | /nodes keyword "reachable":false | 2 failed checks | | High response latency | /readyz > 2000ms | Warning alert |

Summary

| Component | Endpoint | Vigilmon Check | |-----------|----------|----------------| | Node liveness | /readyz | Status 200 | | Raft cluster state | /status | Keyword "state":"Leader" or "Follower" | | Cluster membership | /nodes | Keyword "reachable":true | | External availability | Load balancer /readyz | Status 200 |

rqlite's lightweight footprint makes it easy to underestimate. In a 3-node cluster, losing two nodes means losing quorum — all writes fail until a node is restored. With Vigilmon monitoring each node individually at a 1-minute interval, you will detect node failures fast enough to restore quorum before it becomes a data availability incident.

Further Reading

• Monitoring ClickHouse with Vigilmon
• Monitoring InfluxDB with Vigilmon
• Monitoring Redis with Vigilmon
• Docker Container Monitoring with Vigilmon