tutorial

How to Monitor Couchbase Uptime and Health with Vigilmon

Couchbase rebalances, failovers, and DCP stream stalls can degrade your cluster silently. Learn how to monitor Couchbase uptime externally with Vigilmon HTTP probes, heartbeat monitors for DCP consumers, and alert routing for bucket-level health.

Couchbase is built for high-throughput, low-latency workloads — but auto-rebalancing after a node failure, DCP stream consumer lag, and vBucket map inconsistencies can degrade your cluster silently. A node marked healthy in the Couchbase UI might be serving stale data to your application while rebalancing completes. A DCP consumer feeding your search index might stall without throwing a process-level error.

Vigilmon gives you external visibility into Couchbase health through HTTP probe monitoring and heartbeat monitoring for DCP-based consumers and background jobs. This tutorial walks through both.

Why Couchbase Monitoring Needs More Than Process Checks

systemd, Docker health checks, and the Couchbase Web Console tell you cluster nodes are running. They cannot tell you:

  • Whether Couchbase is reachable from your application servers across the network
  • Whether a rebalance or failover is in progress and affecting read/write latency
  • Whether a DCP stream consumer (Elasticsearch connector, search indexer, event processor) has stalled
  • Whether bucket quota is near exhaustion and memcached evictions are spiking
  • Whether N1QL query service is degraded and returning timeout errors

These are the failure modes that produce slow, degraded experiences without clean error signals. External monitoring through Vigilmon catches them by probing the actual connectivity and logic paths your application relies on.

Step 1: Build a Couchbase Health Endpoint

Couchbase exposes a REST management API on port 8091, but your application servers may not have direct access to that port, and internal metrics don't verify network-layer reachability. Add a thin health route to your existing application server.

Node.js / Express Example

// health/couchbase.js
const express = require('express');
const couchbase = require('couchbase');

const app = express();
let cluster, bucket, collection;

async function connect() {
  cluster = await couchbase.connect(process.env.COUCHBASE_URL, {
    username: process.env.COUCHBASE_USER,
    password: process.env.COUCHBASE_PASS,
  });
  bucket = cluster.bucket(process.env.COUCHBASE_BUCKET);
  collection = bucket.defaultCollection();
}
connect().catch(console.error);

app.get('/health/couchbase', async (req, res) => {
  try {
    // Upsert a known probe document to verify read/write round-trip
    const probeKey = '_health_probe';
    await collection.upsert(probeKey, { ts: Date.now() }, { timeout: 3000 });
    const result = await collection.get(probeKey, { timeout: 3000 });

    if (!result.content || !result.content.ts) {
      return res.status(503).json({ status: 'degraded', reason: 'probe_read_mismatch' });
    }

    // Check for rebalance in progress via management REST API
    let rebalancing = false;
    try {
      const resp = await fetch(`http://${process.env.COUCHBASE_HOST}:8091/pools/default`, {
        headers: {
          Authorization: 'Basic ' + Buffer.from(
            `${process.env.COUCHBASE_USER}:${process.env.COUCHBASE_PASS}`
          ).toString('base64'),
        },
        signal: AbortSignal.timeout(2000),
      });
      const data = await resp.json();
      rebalancing = data.rebalanceStatus !== 'none';
    } catch {
      // Management API unavailable — non-fatal
    }

    if (rebalancing) {
      return res.status(503).json({ status: 'degraded', reason: 'rebalance_in_progress' });
    }

    return res.status(200).json({ status: 'ok' });
  } catch (err) {
    return res.status(503).json({ status: 'down', error: err.message });
  }
});

app.listen(3002);

Python (FastAPI) Example

# health_couchbase.py
from fastapi import FastAPI
from fastapi.responses import JSONResponse
from couchbase.cluster import Cluster
from couchbase.options import ClusterOptions
from couchbase.auth import PasswordAuthenticator
import httpx
import base64
import os
import time

app = FastAPI()

_cluster = None
_collection = None

def get_collection():
    global _cluster, _collection
    if _collection is None:
        auth = PasswordAuthenticator(os.environ["COUCHBASE_USER"], os.environ["COUCHBASE_PASS"])
        _cluster = Cluster(os.environ["COUCHBASE_URL"], ClusterOptions(auth))
        bucket = _cluster.bucket(os.environ["COUCHBASE_BUCKET"])
        _collection = bucket.default_collection()
    return _collection

@app.get("/health/couchbase")
async def couchbase_health():
    try:
        col = get_collection()

        # Write + read probe
        probe_key = "_health_probe"
        col.upsert(probe_key, {"ts": int(time.time() * 1000)})
        result = col.get(probe_key)
        if not result.content_as[dict].get("ts"):
            return JSONResponse(status_code=503, content={"status": "degraded", "reason": "probe_read_mismatch"})

        # Check rebalance status
        rebalancing = False
        try:
            creds = base64.b64encode(
                f"{os.environ['COUCHBASE_USER']}:{os.environ['COUCHBASE_PASS']}".encode()
            ).decode()
            async with httpx.AsyncClient(timeout=2) as client:
                resp = await client.get(
                    f"http://{os.environ['COUCHBASE_HOST']}:8091/pools/default",
                    headers={"Authorization": f"Basic {creds}"},
                )
                data = resp.json()
                rebalancing = data.get("rebalanceStatus", "none") != "none"
        except Exception:
            pass

        if rebalancing:
            return JSONResponse(status_code=503, content={"status": "degraded", "reason": "rebalance_in_progress"})

        return {"status": "ok"}
    except Exception as e:
        return JSONResponse(status_code=503, content={"status": "down", "error": str(e)})

Verify the endpoint before wiring up Vigilmon:

curl -i https://your-app.example.com/health/couchbase
# HTTP/1.1 200 OK
# {"status":"ok"}

Step 2: Configure Vigilmon HTTP Monitor for Couchbase

  1. Log in to vigilmon.online and go to Monitors → New Monitor
  2. Choose HTTP / HTTPS
  3. Set the URL to your health endpoint: https://your-app.example.com/health/couchbase
  4. Set the check interval to 1 minute
  5. Under Expected response, configure:
    • Status code: 200
    • Response body contains: "status":"ok"
    • Response time threshold: 3000ms
  6. Under Alert channels, assign your Slack or PagerDuty channel
  7. Save the monitor

Vigilmon probes from multiple geographic regions simultaneously, requiring multi-region consensus before opening an incident. This eliminates false alarms from transient single-probe network hiccups.

Monitoring Multiple Nodes and Buckets

For multi-node Couchbase clusters, create separate monitors:

  • [couchbase-node-1] /health/couchbase — P1 alert on failure
  • [couchbase-node-2] /health/couchbase — P1 alert on failure
  • [couchbase-bucket-orders] /health/couchbase?bucket=orders — P2 alert for data-plane health

Use Vigilmon's status page grouping to collect all Couchbase monitors into a single pane for on-call engineers.

Step 3: Heartbeat Monitoring for DCP Consumers and Background Jobs

Couchbase DCP (Database Change Protocol) streams power real-time event pipelines, full-text search indexers, Elasticsearch connectors, and analytics consumers. When a DCP consumer stalls — due to a network partition, rebalance event, or unhandled exception — it stops processing silently. No process exits; no error is thrown.

Vigilmon heartbeat monitors detect silent stalls: your consumer pings Vigilmon after each successful processing cycle. If pings stop arriving within the expected window, Vigilmon fires an alert.

Set Up the Heartbeat Monitor

  1. In Vigilmon, go to Monitors → New Monitor → Heartbeat
  2. Set the name: couchbase-dcp-consumer
  3. Set the expected interval: 2 minutes (adjust to your stream's throughput)
  4. Set the grace period: 5 minutes
  5. Save — copy the unique heartbeat URL, e.g. https://vigilmon.online/heartbeat/abc123xyz

Wire It Into a DCP Consumer

// dcp-consumer.js
const couchbase = require('couchbase');
const axios = require('axios');

async function run() {
  const cluster = await couchbase.connect(process.env.COUCHBASE_URL, {
    username: process.env.COUCHBASE_USER,
    password: process.env.COUCHBASE_PASS,
  });

  // Using eventing or a polling loop as a stand-in for DCP processing
  const bucket = cluster.bucket(process.env.COUCHBASE_BUCKET);
  const collection = bucket.defaultCollection();

  setInterval(async () => {
    try {
      // ... your DCP/eventing processing logic here ...
      await processEvents(collection);

      // Ping Vigilmon after each successful processing cycle
      await axios.get(process.env.VIGILMON_HEARTBEAT_URL, { timeout: 5000 }).catch(() => {});
    } catch (err) {
      console.error('DCP consumer error:', err);
      // Heartbeat stops — Vigilmon alerts within grace period
    }
  }, 60_000);
}

run();

For batch jobs such as nightly N1QL aggregations:

# nightly_aggregation.py
import requests
import os
from couchbase.cluster import Cluster
from couchbase.options import ClusterOptions, QueryOptions
from couchbase.auth import PasswordAuthenticator

def run_aggregation():
    auth = PasswordAuthenticator(os.environ["COUCHBASE_USER"], os.environ["COUCHBASE_PASS"])
    cluster = Cluster(os.environ["COUCHBASE_URL"], ClusterOptions(auth))

    result = cluster.query(
        "SELECT COUNT(*) as total FROM `orders` WHERE status = 'pending'",
        QueryOptions(timeout=60)
    )
    for row in result:
        print(f"Pending orders: {row['total']}")

    # Ping Vigilmon on successful completion
    requests.get(os.environ["VIGILMON_HEARTBEAT_URL"], timeout=5)

if __name__ == "__main__":
    run_aggregation()

Step 4: Bucket Quota and Alert Routing

Couchbase evicts data from RAM when a bucket approaches its quota ceiling — if eviction is set to NRU (Not Recently Used), hot data can be silently ejected, causing full-document fetches from disk or cache misses. Expose bucket memory usage in your health endpoint:

// Add to your health endpoint:
const bucketResp = await fetch(
  `http://${process.env.COUCHBASE_HOST}:8091/pools/default/buckets/${process.env.COUCHBASE_BUCKET}`,
  { headers: { Authorization: authHeader }, signal: AbortSignal.timeout(2000) }
);
const bucketData = await bucketResp.json();
const memUsed = bucketData.basicStats?.memUsed ?? 0;
const quota = bucketData.quota?.ram ?? 1;
const memPct = Math.round((memUsed / quota) * 100);

if (memPct > 90) {
  return res.status(503).json({ status: 'degraded', reason: 'bucket_quota_near_limit', mem_pct: memPct });
}

Configure alert routing in Vigilmon:

| Monitor | Alert Channel | Priority | |---|---|---| | Couchbase node /health/couchbase | Slack + PagerDuty | P1 | | Bucket quota threshold | Slack | P2 | | Heartbeat: DCP consumer | Slack + email | P2 | | Heartbeat: nightly N1QL job | Email | P3 |

Set response time thresholds as early warnings:

  • Alert at 1000ms for the health endpoint (Couchbase KV round-trips should be sub-millisecond)
  • Alert at 5000ms for N1QL-backed application endpoints (signals missing index or rebalance impact)

Summary

Couchbase failures surface in subtle ways — silent rebalances, DCP consumer stalls, bucket quota evictions — long before your users see errors. Vigilmon gives you external visibility across the full failure surface:

| Monitor Type | What It Covers | |---|---| | HTTP monitor on /health/couchbase | KV round-trip, rebalance status, bucket quota | | HTTP monitor per cluster node | Per-node availability during rolling restarts | | Heartbeat monitor | DCP consumer liveness, batch job completion |

Get started free at vigilmon.online — your first Couchbase monitor is running in under two minutes.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles