tutorial

How to Monitor CouchDB Uptime and Health with Vigilmon

CouchDB's eventual consistency model and compaction cycles can mask availability problems. Learn how to monitor CouchDB uptime externally with Vigilmon HTTP probes, heartbeat monitors for replication jobs, and alert routing for cluster quorum health.

CouchDB is unique among NoSQL databases in that it exposes a native HTTP API — but that very API being responsive does not mean your data is healthy. A CouchDB cluster can pass its own /_up check while compaction is stalling writes, a replication job is diverging, or quorum has dropped below the write majority your application requires. An out-of-sync node in a three-node cluster quietly returns stale reads without surfacing an error.

Vigilmon gives you external visibility into CouchDB health through HTTP probe monitoring and heartbeat monitoring for continuous replication tasks and background jobs. This tutorial walks through both.

Why CouchDB Monitoring Needs More Than Process Checks

systemd, Docker health checks, and the Fauxton dashboard tell you couchdb is running. They cannot tell you:

  • Whether CouchDB is reachable from your application servers across the network
  • Whether the cluster quorum (write majority) is intact after a node failure
  • Whether a continuous replication job has errored out or fallen behind
  • Whether database compaction is consuming I/O and degrading write throughput
  • Whether pending database changes have grown beyond what your replication lag budget allows

These are the failure modes that produce slow, degraded experiences without clean error signals. External monitoring through Vigilmon catches them by probing the actual connectivity and logic paths your application relies on.

Step 1: Build a CouchDB Health Endpoint

CouchDB already exposes /_up and /_node/_local/_stats, but you need an application-side endpoint that checks the conditions your application actually cares about and returns a meaningful response to Vigilmon.

Node.js / Express Example

// health/couchdb.js
const express = require('express');
const nano = require('nano')(process.env.COUCHDB_URL);

const app = express();

app.get('/health/couchdb', async (req, res) => {
  try {
    // 1. Verify basic connectivity with a known document round-trip
    const db = nano.use(process.env.COUCHDB_DB);
    const probeId = '_health_probe';
    try {
      await db.insert({ ts: Date.now() }, probeId);
    } catch (err) {
      if (err.statusCode !== 409) throw err; // 409 = already exists, acceptable
    }
    const doc = await db.get(probeId);
    if (!doc || !doc._id) {
      return res.status(503).json({ status: 'degraded', reason: 'probe_read_mismatch' });
    }

    // 2. Check cluster membership (quorum)
    const membership = await nano.request({ db: '_membership', method: 'GET' });
    const allNodes = membership.all_nodes?.length ?? 0;
    const clusterNodes = membership.cluster_nodes?.length ?? 0;
    if (clusterNodes < Math.ceil(allNodes / 2) + 1) {
      return res.status(503).json({
        status: 'degraded',
        reason: 'quorum_lost',
        cluster_nodes: clusterNodes,
        all_nodes: allNodes,
      });
    }

    // 3. Check for active replication errors
    const replicatorDb = nano.use('_replicator');
    let failedReplications = 0;
    try {
      const result = await replicatorDb.list({ include_docs: true });
      failedReplications = result.rows.filter(
        r => r.doc?.['_replication_state'] === 'error'
      ).length;
    } catch {
      // _replicator not used — skip
    }

    if (failedReplications > 0) {
      return res.status(503).json({
        status: 'degraded',
        reason: 'replication_error',
        failed_replications: failedReplications,
      });
    }

    return res.status(200).json({ status: 'ok', cluster_nodes: clusterNodes });
  } catch (err) {
    return res.status(503).json({ status: 'down', error: err.message });
  }
});

app.listen(3002);

Python (FastAPI) Example

# health_couchdb.py
from fastapi import FastAPI
from fastapi.responses import JSONResponse
import httpx
import os
import time

app = FastAPI()

COUCHDB_URL = os.environ["COUCHDB_URL"]  # e.g. http://user:pass@localhost:5984
COUCHDB_DB = os.environ["COUCHDB_DB"]

@app.get("/health/couchdb")
async def couchdb_health():
    try:
        async with httpx.AsyncClient(timeout=5) as client:
            # 1. Verify read/write round-trip
            probe_url = f"{COUCHDB_URL}/{COUCHDB_DB}/_health_probe"
            put_resp = await client.put(probe_url, json={"ts": int(time.time() * 1000)})
            if put_resp.status_code not in (201, 202, 409):
                return JSONResponse(status_code=503, content={"status": "degraded", "reason": "write_failed"})

            get_resp = await client.get(probe_url)
            if get_resp.status_code != 200:
                return JSONResponse(status_code=503, content={"status": "degraded", "reason": "read_failed"})

            # 2. Cluster quorum check
            membership_resp = await client.get(f"{COUCHDB_URL}/_membership")
            membership = membership_resp.json()
            all_nodes = len(membership.get("all_nodes", []))
            cluster_nodes = len(membership.get("cluster_nodes", []))
            quorum = (all_nodes // 2) + 1
            if cluster_nodes < quorum:
                return JSONResponse(status_code=503, content={
                    "status": "degraded",
                    "reason": "quorum_lost",
                    "cluster_nodes": cluster_nodes,
                    "all_nodes": all_nodes,
                })

            # 3. Replication error check
            repl_resp = await client.get(f"{COUCHDB_URL}/_replicator/_all_docs?include_docs=true")
            if repl_resp.status_code == 200:
                rows = repl_resp.json().get("rows", [])
                failed = [r for r in rows if r.get("doc", {}).get("_replication_state") == "error"]
                if failed:
                    return JSONResponse(status_code=503, content={
                        "status": "degraded",
                        "reason": "replication_error",
                        "failed_replications": len(failed),
                    })

        return {"status": "ok", "cluster_nodes": cluster_nodes}
    except Exception as e:
        return JSONResponse(status_code=503, content={"status": "down", "error": str(e)})

Verify the endpoint before wiring up Vigilmon:

curl -i https://your-app.example.com/health/couchdb
# HTTP/1.1 200 OK
# {"status":"ok","cluster_nodes":3}

Step 2: Configure Vigilmon HTTP Monitor for CouchDB

  1. Log in to vigilmon.online and go to Monitors → New Monitor
  2. Choose HTTP / HTTPS
  3. Set the URL to your health endpoint: https://your-app.example.com/health/couchdb
  4. Set the check interval to 1 minute
  5. Under Expected response, configure:
    • Status code: 200
    • Response body contains: "status":"ok"
    • Response time threshold: 3000ms
  6. Under Alert channels, assign your Slack or PagerDuty channel
  7. Save the monitor

Vigilmon probes from multiple geographic regions simultaneously, requiring multi-region consensus before opening an incident. This eliminates false alarms from transient single-probe network hiccups.

Monitoring Individual Cluster Nodes

In a CouchDB cluster, create per-node monitors to detect partial failures:

  • [couchdb-node-1] /health/couchdb — P1 alert on node failure
  • [couchdb-node-2] /health/couchdb — P1 alert on node failure
  • [couchdb-node-3] /health/couchdb — P1 alert on node failure

Use Vigilmon's status page grouping to surface all CouchDB monitors in a single pane for on-call engineers.

Step 3: Heartbeat Monitoring for Continuous Replication and Background Jobs

CouchDB continuous replication jobs — used for database mirroring, offline-first sync, and cross-region data distribution — can enter an error state silently. The _replicator database marks the job as error but no alert fires automatically. Document processing workers that consume the _changes feed can also stall indefinitely without exiting.

Vigilmon heartbeat monitors detect silent stalls: your job pings Vigilmon after each successful cycle. If pings stop arriving within the expected window, Vigilmon fires an alert.

Set Up the Heartbeat Monitor

  1. In Vigilmon, go to Monitors → New Monitor → Heartbeat
  2. Set the name: couchdb-changes-consumer
  3. Set the expected interval: 5 minutes (adjust to your feed's throughput)
  4. Set the grace period: 10 minutes
  5. Save — copy the unique heartbeat URL, e.g. https://vigilmon.online/heartbeat/abc123xyz

Wire It Into a Changes Feed Consumer

// changes-consumer.js
const nano = require('nano')(process.env.COUCHDB_URL);
const axios = require('axios');

async function run() {
  const db = nano.use(process.env.COUCHDB_DB);
  let lastSeq = 'now';

  async function poll() {
    try {
      const changes = await db.changes({ since: lastSeq, limit: 100, include_docs: true });
      for (const change of changes.results) {
        await processChange(change.doc);
      }
      lastSeq = changes.last_seq;

      // Ping Vigilmon after each successful polling cycle
      await axios.get(process.env.VIGILMON_HEARTBEAT_URL, { timeout: 5000 }).catch(() => {});
    } catch (err) {
      console.error('Changes feed error:', err);
      // Heartbeat stops — Vigilmon alerts within grace period
    }
    setTimeout(poll, 30_000);
  }

  await poll();
}

run();

For batch replication verification jobs:

# verify_replication.py
import requests
import os

def check_replication_health():
    couchdb_url = os.environ["COUCHDB_URL"]
    resp = requests.get(
        f"{couchdb_url}/_replicator/_all_docs",
        params={"include_docs": "true"},
        timeout=10,
    )
    resp.raise_for_status()
    rows = resp.json().get("rows", [])
    failed = [r for r in rows if r.get("doc", {}).get("_replication_state") == "error"]
    if failed:
        raise RuntimeError(f"Replication errors: {[r['id'] for r in failed]}")

    print(f"All {len(rows)} replication jobs healthy")
    requests.get(os.environ["VIGILMON_HEARTBEAT_URL"], timeout=5)

if __name__ == "__main__":
    check_replication_health()

Step 4: Compaction and Alert Routing

CouchDB stores document revisions as append-only B-tree entries. Without regular compaction, disk usage grows unbounded and read performance degrades as revision trees deepen. Monitor compaction status and disk usage in your health endpoint:

// Add to your health endpoint:
const dbInfo = await db.info();
const diskSize = dbInfo.sizes?.file ?? 0;
const dataSize = dbInfo.sizes?.active ?? 0;
const wasteRatio = diskSize > 0 ? ((diskSize - dataSize) / diskSize) : 0;

if (wasteRatio > 0.7) {
  return res.status(503).json({
    status: 'degraded',
    reason: 'compaction_needed',
    waste_ratio: wasteRatio.toFixed(2),
  });
}

Configure alert routing in Vigilmon:

| Monitor | Alert Channel | Priority | |---|---|---| | CouchDB cluster /health/couchdb | Slack + PagerDuty | P1 | | Per-node /health/couchdb | Slack | P1 | | Heartbeat: changes consumer | Slack + email | P2 | | Heartbeat: replication verification | Email | P3 |

Set response time thresholds as early warnings:

  • Alert at 1000ms for the health endpoint (basic CouchDB document fetches should be fast)
  • Alert at 5000ms for application endpoints backed by CouchDB views (signals missing design doc or compaction backlog)

Summary

CouchDB failures surface in subtle ways — quorum loss, silent replication errors, compaction-stalled writes — long before your users see errors. Vigilmon gives you external visibility across the full failure surface:

| Monitor Type | What It Covers | |---|---| | HTTP monitor on /health/couchdb | Read/write round-trip, quorum, replication errors | | HTTP monitor per cluster node | Per-node availability | | Heartbeat monitor | Changes feed consumer liveness, batch job completion |

Get started free at vigilmon.online — your first CouchDB monitor is running in under two minutes.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles