tutorial

Monitoring TigerBeetle with Vigilmon

TigerBeetle is the financial-grade ledger database built for double-entry bookkeeping at 1M transfers/second. Here's how to monitor replica health, cluster quorum, transfer throughput, and balance integrity with Vigilmon.

TigerBeetle is a purpose-built financial-grade ledger database written in Zig, designed around double-entry accounting invariants at the storage layer. It runs as a cluster of replicas (minimum 3 for production) using the VSR (Viewstamped State Replication) consensus protocol, targeting 1 million transfers per second on commodity hardware. Unlike traditional databases, TigerBeetle exposes no SQL or REST interface — clients use the TigerBeetle client library directly (Go, Node.js, Python, Java, .NET, Rust, C). This means monitoring requires process-level checks, client-side instrumentation, and periodic balance verification through the API. Vigilmon handles the process and TCP-level monitoring while your instrumentation feeds metrics for the rest.

What You'll Set Up

  • Replica process liveness checks on each cluster node
  • Cluster quorum health via TCP port monitoring across all replicas
  • Transfer throughput and commit latency alerting
  • Double-entry balance integrity verification
  • WAL disk usage alerts before storage capacity is exhausted
  • Backup age alerts for financial audit trail compliance

Prerequisites

  • TigerBeetle cluster with 3+ replicas (one primary, two replicas minimum)
  • Each replica exposing port 3000 (or configured replica port)
  • A free Vigilmon account
  • Client library instrumentation to expose metrics (Prometheus or a simple HTTP endpoint)

Step 1: Monitor Each Replica's TCP Port

TigerBeetle replicas listen on a TCP port (default 3000) for client connections. A replica going silent is the first sign of quorum failure. Add a TCP monitor for each replica node:

  1. Log in to vigilmon.online and click Add Monitor.
  2. Set Type to TCP.
  3. Enter the replica host and port: replica-1.internal:3000.
  4. Set Check interval to 30 seconds.
  5. Set Alert threshold to 2 consecutive failures (a single blip may be a transient network hiccup; two means the replica is truly down).
  6. Click Save.

Repeat for every replica in your cluster:

replica-1.internal:3000
replica-2.internal:3000
replica-3.internal:3000

For a 3-replica VSR cluster the quorum minimum is ⌈3/2⌉+1 = 2 replicas. If two or more TCP monitors alert simultaneously, your cluster has lost quorum and all writes are halted.

Step 2: Add an Operations Health Endpoint (Client-Side)

TigerBeetle does not expose an HTTP management API. You need a thin sidecar or wrapper service on each node that pings the TigerBeetle replica and exposes an HTTP /health endpoint Vigilmon can probe. A minimal Go sidecar example:

package main

import (
    "context"
    "net/http"
    "os"

    "github.com/tigerbeetle/tigerbeetle-go"
)

func main() {
    addresses := []string{os.Getenv("TB_ADDRESS")}
    client, err := tigerbeetle_go.NewClient(tigerbeetle_go.ToUint128(0), addresses, 1)
    if err != nil {
        panic(err)
    }
    defer client.Close()

    http.HandleFunc("/health", func(w http.ResponseWriter, r *http.Request) {
        // A zero-item lookup is a lightweight liveness ping
        _, lookupErr := client.LookupAccounts([]tigerbeetle_go.Uint128{})
        if lookupErr != nil {
            http.Error(w, "tigerbeetle unavailable", http.StatusServiceUnavailable)
            return
        }
        w.WriteHeader(http.StatusOK)
        w.Write([]byte(`{"status":"ok"}`))
    })

    http.ListenAndServe(":9090", nil)
}

Run this sidecar on each replica host. Then add HTTP monitors in Vigilmon:

  1. Click Add Monitor → HTTP / HTTPS.
  2. Enter http://replica-1.internal:9090/health.
  3. Set Expected status to 200.
  4. Set Check interval to 1 minute.
  5. Click Save.

Step 3: Monitor Cluster Quorum

Beyond individual replica checks, you want a combined quorum check. Add a separate Vigilmon HTTP monitor for a quorum-aware health endpoint on your primary replica sidecar. This endpoint should return 200 only when the majority of replicas are reachable:

http.HandleFunc("/quorum", func(w http.ResponseWriter, r *http.Request) {
    alive := checkReplicaCount() // your internal probe of all replica sidecars
    quorumMin := (totalReplicas / 2) + 1
    if alive < quorumMin {
        http.Error(w, fmt.Sprintf("quorum lost: %d/%d replicas alive", alive, totalReplicas), http.StatusServiceUnavailable)
        return
    }
    w.WriteHeader(http.StatusOK)
    fmt.Fprintf(w, `{"alive":%d,"quorum_min":%d}`, alive, quorumMin)
})

Add this as an HTTP monitor in Vigilmon with a 1-minute check interval and immediate alerting (1 consecutive failure). A quorum failure blocks all TigerBeetle writes immediately.

Step 4: Alert on Transfer Throughput Drop

Instrument your TigerBeetle client code to count committed transfers per second and expose this as a Prometheus metric or a simple JSON endpoint:

import time
import threading
from tigerbeetle import Client

transfer_counter = 0
transfer_lock = threading.Lock()

def commit_transfers(client, batch):
    global transfer_counter
    errors = client.create_transfers(batch)
    with transfer_lock:
        transfer_counter += len(batch) - len(errors)

def metrics_endpoint():
    # Expose as HTTP endpoint for Vigilmon
    # /metrics returns {"transfers_per_second": <rate>}
    pass

Once you expose a /metrics endpoint, add a Vigilmon HTTP monitor that checks for a healthy throughput value. For critical financial systems, any sustained drop in transfer rate should page on-call immediately:

  1. Click Add Monitor → HTTP / HTTPS.
  2. Enter http://tb-instrumentation.internal:9091/metrics.
  3. Enable Keyword check and verify transfers_per_second appears in the response body.
  4. Set Check interval to 1 minute.
  5. Click Save.

For P99 commit latency, TigerBeetle's VSR consensus adds replica round-trip time to each batch commit. A P99 exceeding 10ms typically indicates replica lag or network pressure between replicas.

Step 5: Balance Integrity Verification (Double-Entry Check)

The core financial guarantee of TigerBeetle is that the sum of all debit postings equals the sum of all credit postings for each account. Schedule a periodic balance check as a cron heartbeat in Vigilmon:

Write a balance verification script that runs on a schedule:

#!/usr/bin/env python3
import requests
from tigerbeetle import Client, AccountFilter

VIGILMON_HEARTBEAT = "https://vigilmon.online/heartbeat/YOUR_HEARTBEAT_ID"

def verify_balance_integrity(client):
    """
    For each account, sum(debits_posted) - sum(credits_posted) must equal
    the account's reported balance. TigerBeetle enforces this at the storage
    layer, so any mismatch is a critical integrity violation.
    """
    accounts = client.lookup_accounts([...])  # fetch accounts to verify
    for account in accounts:
        expected = account.debits_posted - account.credits_posted
        if expected != account.balance:
            raise ValueError(
                f"Balance mismatch on account {account.id}: "
                f"expected {expected}, got {account.balance}"
            )

def main():
    client = Client(cluster_id=0, addresses=["127.0.0.1:3000"])
    try:
        verify_balance_integrity(client)
        # All balances check out — ping Vigilmon heartbeat
        requests.get(VIGILMON_HEARTBEAT, timeout=10)
    except Exception as e:
        print(f"INTEGRITY ERROR: {e}")
        # Do NOT ping heartbeat — Vigilmon will alert on missed ping
    finally:
        client.close()

if __name__ == "__main__":
    main()

Set up the cron heartbeat in Vigilmon:

  1. Click Add Monitor → Cron Heartbeat.
  2. Set the expected ping interval to 15 minutes (or your verification cadence).
  3. Copy the heartbeat URL and paste it as VIGILMON_HEARTBEAT in the script.
  4. Add this script to your system cron:
*/15 * * * * /usr/local/bin/tb-balance-check.py

A missed heartbeat means either the balance check failed (integrity violation) or the script itself crashed — both deserve immediate attention.

Step 6: WAL Disk Usage Alert

TigerBeetle uses an append-only data file (Direct I/O, never overwriting committed data). Disk usage grows as transfers accumulate. Monitor the data file disk usage with a cron heartbeat that checks available space:

#!/bin/bash
# /usr/local/bin/tb-disk-check.sh
HEARTBEAT_URL="https://vigilmon.online/heartbeat/YOUR_DISK_HEARTBEAT_ID"
TB_DATA_DIR="/var/lib/tigerbeetle"
THRESHOLD=80  # alert if disk usage > 80%

USAGE=$(df "$TB_DATA_DIR" | awk 'NR==2 {print $5}' | tr -d '%')

if [ "$USAGE" -lt "$THRESHOLD" ]; then
    curl -s "$HEARTBEAT_URL" > /dev/null
else
    echo "DISK ALERT: TigerBeetle data dir at ${USAGE}% capacity"
    # Do not ping heartbeat — Vigilmon alerts on missed ping
fi

Add a cron heartbeat in Vigilmon with a 10-minute interval, then schedule the script:

*/10 * * * * /usr/local/bin/tb-disk-check.sh

Step 7: Backup Age Alert

TigerBeetle data files must be backed up regularly for financial audit trail integrity. A missed backup should alert before the audit gap becomes a compliance issue:

#!/bin/bash
# /usr/local/bin/tb-backup-check.sh
HEARTBEAT_URL="https://vigilmon.online/heartbeat/YOUR_BACKUP_HEARTBEAT_ID"
BACKUP_MARKER="/var/backups/tigerbeetle/last_backup_success"
MAX_AGE_HOURS=25  # alert if last backup > 25 hours ago

if [ -f "$BACKUP_MARKER" ]; then
    LAST_BACKUP=$(stat -c %Y "$BACKUP_MARKER")
    NOW=$(date +%s)
    AGE_HOURS=$(( (NOW - LAST_BACKUP) / 3600 ))
    if [ "$AGE_HOURS" -lt "$MAX_AGE_HOURS" ]; then
        curl -s "$HEARTBEAT_URL" > /dev/null
    fi
fi
# If marker missing or too old, heartbeat is not pinged → Vigilmon alerts

Your actual backup script should touch "$BACKUP_MARKER" on successful completion.

Step 8: Configure Alert Channels

  1. Go to Alert Channels in Vigilmon and connect your team's Slack, PagerDuty, or email.
  2. For replica TCP monitors: set consecutive failures to 2 before alerting.
  3. For the quorum health endpoint: set to 1 consecutive failure — quorum loss is immediate.
  4. For balance integrity heartbeats: set to 1 missed ping — any missed integrity check is critical.
  5. Use Vigilmon's escalation feature to page the on-call engineer immediately for quorum and balance alerts while routing replica alerts to a lower-urgency channel.

Summary

| Monitor | Type | Interval | Alert Threshold | |---|---|---|---| | Replica TCP (×3) | TCP | 30s | 2 failures | | Replica HTTP health (×3) | HTTP | 1m | 2 failures | | Cluster quorum endpoint | HTTP | 1m | 1 failure | | Transfer throughput | HTTP keyword | 1m | 2 failures | | Balance integrity heartbeat | Cron | 15m | 1 missed ping | | WAL disk usage heartbeat | Cron | 10m | 1 missed ping | | Backup age heartbeat | Cron | 1h | 1 missed ping |

TigerBeetle's architecture — VSR consensus, append-only storage, and enforced double-entry invariants — makes it one of the safest financial databases available. Vigilmon fills the observability gap by wrapping process liveness, TCP reachability, and cron-based integrity checks into a single alerting dashboard. When your next transfer batch hits 1M/s, you'll know the moment anything in that pipeline deviates.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles