Monitoring FoundationDB with Vigilmon: Distributed Key-Value Availability, Cluster Health, and External Uptime Checks

FoundationDB is a distributed ordered key-value store built for strict ACID transactions at scale, the engine behind Apple's CloudKit and Snowflake's metadata layer. Its architecture gives you extraordinary reliability guarantees — but those guarantees depend on a healthy cluster of coordinators, log servers, and storage servers all working in concert. Coordinator election failures, storage server lag behind the transaction log, and degraded mode operation (running below the configured replication factor) can all reduce throughput or block writes long before any process-level monitor fires. Vigilmon adds an independent external layer — HTTP health checks on your FoundationDB-connected services that verify real cluster connectivity, surface degraded mode operation, and alert on application-level failures before your users notice.

This tutorial shows you how to build meaningful monitoring for FoundationDB-backed applications using Vigilmon.

What You'll Build

• A health endpoint that probes real FoundationDB cluster connectivity and checks cluster status
• Degraded mode and replication health detection
• A Vigilmon HTTP monitor with appropriate timeout and assertion settings
• A heartbeat monitor for FoundationDB-backed background workers
• A cluster-aware alerting strategy for coordinator and storage server tiers

Prerequisites

• A running FoundationDB cluster (single-process dev or multi-node production)
• An application connected to FoundationDB (any runtime)
• A free account at vigilmon.online

Step 1: Add a Health Endpoint to Your FoundationDB Service

Add a /health endpoint that performs a lightweight FoundationDB read/write to verify real connectivity and checks cluster status via fdb.getStatus().

Node.js (foundationdb — official Node binding)

// routes/health.js
import fdb from 'foundationdb';

fdb.setAPIVersion(720);

const db = fdb.open(process.env.FDB_CLUSTER_FILE || '/etc/foundationdb/fdb.cluster');

export async function healthHandler(req, res) {
  const checks = {};
  let ok = true;
  const start = Date.now();

  // Check 1: Read/write probe with a sentinel key
  try {
    await db.doTransaction(async (tn) => {
      const probeKey = Buffer.from('vigilmon.probe');
      tn.set(probeKey, Buffer.from(String(Date.now())));
      const val = await tn.get(probeKey);
      if (!val) throw new Error('Probe key read returned null');
    });
    checks.connectivity = 'ok';
    checks.pingLatencyMs = Date.now() - start;
  } catch (err) {
    checks.connectivity = `error: ${err.message}`;
    ok = false;
  }

  // Check 2: Cluster status via status JSON
  try {
    const statusStart = Date.now();
    const statusJson = await db.doTransaction(async (tn) => {
      // FoundationDB exposes cluster status at the special \xff\xff/status/json key
      const statusKey = Buffer.from('\xff\xff/status/json');
      const raw = await tn.get(statusKey);
      return raw ? JSON.parse(raw.toString()) : null;
    });

    if (!statusJson) {
      checks.clusterStatus = 'status JSON unavailable';
      ok = false;
    } else {
      const cluster = statusJson.cluster || {};
      const health = cluster.data?.state?.healthy;
      const description = cluster.data?.state?.description || '';
      const replicationFactor = cluster.configuration?.redundancy_mode || 'unknown';
      const degraded = cluster.degraded_processes || 0;

      checks.clusterHealthy = health ?? false;
      checks.clusterDescription = description;
      checks.replicationMode = replicationFactor;
      checks.degradedProcesses = degraded;
      checks.statusLatencyMs = Date.now() - statusStart;

      if (health === false) {
        checks.clusterStatus = `unhealthy: ${description}`;
        ok = false;
      } else if (degraded > 0) {
        checks.clusterStatus = `degraded: ${degraded} processes degraded`;
        // Degraded but not fully down — decide per your tolerance
      } else {
        checks.clusterStatus = 'ok';
      }
    }
  } catch (err) {
    checks.clusterStatus = `error: ${err.message}`;
    ok = false;
  }

  res.status(ok ? 200 : 503).json({
    status: ok ? 'ok' : 'degraded',
    service: 'foundationdb-service',
    checks,
  });
}

Python (fdb — official Python binding)

# routers/health.py
import os, time, json
import fdb
from fastapi import APIRouter
from fastapi.responses import JSONResponse

fdb.api_version(720)
router = APIRouter()

db = fdb.open(os.environ.get('FDB_CLUSTER_FILE', '/etc/foundationdb/fdb.cluster'))

@fdb.transactional
def probe_read_write(tr):
    probe_key = b'vigilmon.probe'
    tr[probe_key] = str(int(time.time() * 1000)).encode()
    val = tr[probe_key]
    return val.present()

@fdb.transactional
def get_status(tr):
    status_key = b'\xff\xff/status/json'
    raw = tr[status_key]
    return json.loads(bytes(raw).decode()) if raw.present() else None

@router.get("/health")
async def health():
    checks = {}
    ok = True
    start = time.monotonic()

    # Check 1: Read/write probe
    try:
        result = probe_read_write(db)
        checks['connectivity'] = 'ok' if result else 'probe read failed'
        checks['pingLatencyMs'] = int((time.monotonic() - start) * 1000)
        if not result:
            ok = False
    except Exception as e:
        checks['connectivity'] = f'error: {e}'
        ok = False

    # Check 2: Cluster status
    try:
        status_start = time.monotonic()
        status = get_status(db)
        checks['statusLatencyMs'] = int((time.monotonic() - status_start) * 1000)

        if not status:
            checks['clusterStatus'] = 'status JSON unavailable'
            ok = False
        else:
            cluster = status.get('cluster', {})
            state = cluster.get('data', {}).get('state', {})
            healthy = state.get('healthy', False)
            description = state.get('description', '')
            degraded = cluster.get('degraded_processes', 0)
            replication = cluster.get('configuration', {}).get('redundancy_mode', 'unknown')

            checks['clusterHealthy'] = healthy
            checks['clusterDescription'] = description
            checks['replicationMode'] = replication
            checks['degradedProcesses'] = degraded

            if not healthy:
                checks['clusterStatus'] = f'unhealthy: {description}'
                ok = False
            elif degraded > 0:
                checks['clusterStatus'] = f'degraded: {degraded} processes degraded'
            else:
                checks['clusterStatus'] = 'ok'
    except Exception as e:
        checks['clusterStatus'] = f'error: {e}'
        ok = False

    return JSONResponse(
        status_code=200 if ok else 503,
        content={'status': 'ok' if ok else 'degraded', 'service': 'foundationdb-service', 'checks': checks},
    )

Go (apple/foundationdb Go binding)

// internal/health/handler.go
package health

import (
    "encoding/json"
    "net/http"
    "os"
    "time"

    "github.com/apple/foundationdb/bindings/go/src/fdb"
)

func Handler(database fdb.Database) http.HandlerFunc {
    return func(w http.ResponseWriter, r *http.Request) {
        checks := map[string]any{}
        ok := true

        // Check 1: Read/write probe
        start := time.Now()
        _, err := database.Transact(func(tr fdb.Transaction) (any, error) {
            probeKey := fdb.Key("vigilmon.probe")
            tr.Set(probeKey, []byte(time.Now().String()))
            val, err := tr.Get(probeKey).Get()
            if err != nil {
                return nil, err
            }
            if val == nil {
                return nil, fmt.Errorf("probe key read returned nil")
            }
            return val, nil
        })
        if err != nil {
            checks["connectivity"] = "error: " + err.Error()
            ok = false
        } else {
            checks["connectivity"] = "ok"
            checks["pingLatencyMs"] = time.Since(start).Milliseconds()
        }

        // Check 2: Cluster status
        statusStart := time.Now()
        rawStatus, err := database.Transact(func(tr fdb.Transaction) (any, error) {
            statusKey := fdb.Key("\xff\xff/status/json")
            raw, err := tr.Get(statusKey).Get()
            return raw, err
        })
        if err != nil {
            checks["clusterStatus"] = "error: " + err.Error()
            ok = false
        } else if rawStatus == nil {
            checks["clusterStatus"] = "status JSON unavailable"
            ok = false
        } else {
            checks["statusLatencyMs"] = time.Since(statusStart).Milliseconds()
            var statusJSON map[string]any
            if jsonErr := json.Unmarshal(rawStatus.([]byte), &statusJSON); jsonErr == nil {
                cluster, _ := statusJSON["cluster"].(map[string]any)
                data, _ := cluster["data"].(map[string]any)
                state, _ := data["state"].(map[string]any)
                healthy, _ := state["healthy"].(bool)
                description, _ := state["description"].(string)
                degraded, _ := cluster["degraded_processes"].(float64)

                checks["clusterHealthy"] = healthy
                checks["clusterDescription"] = description
                checks["degradedProcesses"] = int(degraded)

                if !healthy {
                    checks["clusterStatus"] = "unhealthy: " + description
                    ok = false
                } else if degraded > 0 {
                    checks["clusterStatus"] = fmt.Sprintf("degraded: %.0f processes degraded", degraded)
                } else {
                    checks["clusterStatus"] = "ok"
                }
            } else {
                checks["clusterStatus"] = "error parsing status JSON"
                ok = false
            }
        }

        w.Header().Set("Content-Type", "application/json")
        if !ok {
            w.WriteHeader(http.StatusServiceUnavailable)
        }
        _ = json.NewEncoder(w).Encode(map[string]any{
            "status":  map[bool]string{true: "ok", false: "degraded"}[ok],
            "service": "foundationdb-service",
            "checks":  checks,
        })
    }
}

Deploy and verify manually before wiring it into Vigilmon:

curl -i https://your-app.example.com/health
# HTTP/1.1 200 OK
# {"status":"ok","service":"foundationdb-service","checks":{"connectivity":"ok","pingLatencyMs":3,"clusterStatus":"ok","clusterHealthy":true,"degradedProcesses":0}}

Step 2: Configure the Vigilmon HTTP Monitor

1. Log in to Vigilmon → Add Monitor → HTTP.
2. URL: your service's health endpoint (e.g. https://api.example.com/health).
3. Check interval: 60 seconds.
4. Response timeout: 15 seconds (FoundationDB status reads can take up to 5 seconds on a loaded cluster).
5. Expected status: 200.
6. JSON assertion: path status, expected value ok.
7. Save the monitor.

Vigilmon probes from multiple geographic regions simultaneously. A single-region blip will not page you; multi-region consensus is required before an incident opens. This is particularly important for FoundationDB because coordinator leader elections and log server failovers cause brief transient errors that resolve within seconds — you want durable alerts, not pager fatigue.

Tip: Add a second JSON assertion on checks.clusterHealthy equals true. This fires specifically when the FoundationDB cluster reports itself as unhealthy, giving you a separate signal from connectivity failures.

Step 3: Heartbeat Monitoring for FoundationDB-Backed Workers

FoundationDB is commonly used as the transactional backbone for layers built on top of it (record store layers, document layers, queue implementations). Workers that depend on FoundationDB will stall silently when the cluster enters degraded mode or when transaction throughput drops below the workload's requirements. Vigilmon's heartbeat monitors detect silent worker death: your worker pings Vigilmon after each successful processing cycle, and if pings stop arriving, Vigilmon fires an alert.

Set Up the Heartbeat Monitor

1. In Vigilmon, go to Monitors → New Monitor → Heartbeat.
2. Set the name: foundationdb-worker.
3. Set the expected interval: 2 minutes (adjust to your job frequency).
4. Set the grace period: 5 minutes.
5. Save — copy the unique heartbeat URL.

Wire It Into Your Worker

Node.js:

import fdb from 'foundationdb';
import fetch from 'node-fetch';

fdb.setAPIVersion(720);
const db = fdb.open(process.env.FDB_CLUSTER_FILE || '/etc/foundationdb/fdb.cluster');

const jobQueue = db.at(fdb.tuple.pack(['jobs']));

async function runWorkerCycle() {
  await db.doTransaction(async (tn) => {
    // Read and clear a batch of queued jobs
    const range = await tn.getRangeAll(jobQueue.range(), { limit: 50 });
    for (const [key, value] of range) {
      const job = JSON.parse(value.toString());
      await processJob(job);
      tn.clear(key);
    }
  });

  // Only ping Vigilmon after successful transaction commit
  await fetch(process.env.VIGILMON_HEARTBEAT_URL).catch(() => {});
}

setInterval(runWorkerCycle, 60_000);

Python:

import os, time, json, requests
import fdb

fdb.api_version(720)
db = fdb.open(os.environ.get('FDB_CLUSTER_FILE', '/etc/foundationdb/fdb.cluster'))

JOB_PREFIX = fdb.Subspace(('jobs',))

@fdb.transactional
def process_batch(tr):
    jobs = list(tr[JOB_PREFIX.range()])
    for key, value in jobs[:50]:
        job = json.loads(value.decode())
        process_job(job)
        del tr[key]

def run_worker_cycle():
    process_batch(db)
    requests.get(os.environ['VIGILMON_HEARTBEAT_URL'], timeout=5)

while True:
    try:
        run_worker_cycle()
    except Exception as e:
        print(f'Worker cycle failed — heartbeat NOT sent: {e}')
    time.sleep(60)

Step 4: Using fdbcli for Direct Cluster Health Checks

For infrastructure-level monitoring alongside Vigilmon's application-layer checks, fdbcli provides the canonical cluster status view:

# Check cluster health from any machine with access to the cluster file
fdbcli --exec "status json" | python3 -m json.tool

# Key fields to monitor:
# .cluster.data.state.healthy — true/false
# .cluster.data.state.description — human-readable status
# .cluster.degraded_processes — count of degraded processes
# .cluster.qos.worst_queue_bytes_storage_server — storage queue depth
# .cluster.workload.transactions.committed.hz — committed transactions/sec

You can wrap fdbcli in a cron job or a dedicated monitoring probe that pushes metrics to your observability stack — and use Vigilmon for the independent external HTTP check that confirms your application's perspective.

Step 5: Alerting Strategy

| Alert channel | When to use | |---|---| | PagerDuty | Connectivity probe fails (cluster unreachable) | | PagerDuty | clusterHealthy: false (cluster reports unhealthy state) | | Slack webhook | degradedProcesses > 0 (below replication factor, writes may block) | | Slack + Email | Transaction latency > 50ms (cluster under abnormal load) | | PagerDuty | Heartbeat missed (worker pipeline stopped) | | Status page | Any user-facing service backed by FoundationDB |

Set response time thresholds as an early warning:

• Alert at 10ms for transactional probe latency (simple read/write should be < 5ms on a healthy cluster)
• Alert at 5000ms for health endpoint response time (status JSON reads can take up to 5s under load — anything beyond that signals serious cluster distress)

What Vigilmon Catches That Internal Monitoring Misses

| Scenario | fdbcli / internal metrics | Vigilmon | |---|---|---| | Cluster reachable from ops host but not from app servers | No | HTTP probe from app's perspective | | Degraded mode (below replication factor — writes may stall) | fdbcli status only | degradedProcesses check fires | | Coordinator leader election (brief write unavailability) | Cluster log only | Probe transaction fails — 503 fires | | Worker pipeline silently stopped | Process appears running | Heartbeat grace period expires → alert | | Storage queue depth growing (approaching throughput limit) | Metric only | Transaction probe latency threshold fires | | Monitoring pipeline (Prometheus/Grafana) fails | Alerts silently lost | Vigilmon is external — unaffected |

FoundationDB's ACID guarantees are only as strong as the cluster serving them. When degraded processes reduce the replication factor or a coordinator election blocks writes, your application pays the cost immediately. External health checks with Vigilmon give you the independent signal you need — before transaction failures cascade to your users.

Start monitoring your FoundationDB applications in under 5 minutes — register free at vigilmon.online.