tutorial

Monitoring Google Cloud Bigtable with Vigilmon: Wide-Column Health Checks, Node Availability & Batch Job Heartbeats

Google Cloud Bigtable is a massively scalable NoSQL wide-column store — but its managed nature hides application-layer failures. This guide shows how to build health endpoints for Bigtable-backed services and monitor them independently with Vigilmon.

Google Cloud Bigtable is a fully managed, massively scalable NoSQL wide-column database that powers some of the largest workloads on Google Cloud — from real-time analytics to time-series storage to ML feature serving. But "fully managed" doesn't mean immune to failure from your application's perspective. Instance configuration errors, app profile misconfiguration, row key hotspots, replication lag across clusters, and IAM permission failures can all degrade your Bigtable-backed service while Cloud Monitoring metrics show no cluster-level incidents. Vigilmon adds an independent external monitoring layer: HTTP health checks on your Bigtable-connected services that probe real read/write availability and alert before users encounter errors.

What You'll Build

  • A health endpoint that performs real Bigtable reads and writes to verify connectivity
  • A Vigilmon HTTP monitor with timeout and JSON assertion configuration
  • Replication cluster health checks for multi-cluster Bigtable instances
  • Cron heartbeat monitoring for background Bigtable ingestion jobs
  • An alerting strategy tuned to Bigtable's operational patterns

Prerequisites

  • A Google Cloud project with a Bigtable instance (development or production tier)
  • A table with at least one column family
  • Service account credentials with bigtable.data.reader and bigtable.data.writer roles
  • An application service connecting to Bigtable
  • A free account at vigilmon.online

Step 1: Add a Health Endpoint to Your Bigtable Service

Add a /health route that performs a lightweight read-then-write probe against a dedicated health row in your Bigtable table.

Node.js (@google-cloud/bigtable)

// routes/health.js
import { Bigtable } from '@google-cloud/bigtable';

const bigtable = new Bigtable({ projectId: process.env.GCLOUD_PROJECT });
const instance = bigtable.instance(process.env.BIGTABLE_INSTANCE_ID);
const table = instance.table(process.env.BIGTABLE_TABLE_ID);

// Use a dedicated health-check row — never a production row key
const HEALTH_ROW_KEY = '_vigilmon_healthcheck';

export async function healthHandler(req, res) {
  const checks = {};
  let ok = true;
  const start = Date.now();

  // Check 1: Write probe — verify write path
  try {
    const row = table.row(HEALTH_ROW_KEY);
    await row.save({
      cf1: {                                // replace 'cf1' with your column family name
        timestamp: { value: Date.now().toString() },
      },
    });
    checks.write = 'ok';
    checks.writeLatencyMs = Date.now() - start;
  } catch (err) {
    checks.write = `error: ${err.message}`;
    checks.writeCode = err.code ?? 'unknown';
    ok = false;
  }

  // Check 2: Read probe — verify read path
  const readStart = Date.now();
  try {
    const [row] = await table.row(HEALTH_ROW_KEY).get({ decode: false });
    checks.read = row ? 'ok' : 'row not found';
    checks.readLatencyMs = Date.now() - readStart;
    if (!row) ok = false;
  } catch (err) {
    checks.read = `error: ${err.message}`;
    ok = false;
  }

  res.status(ok ? 200 : 503).json({
    status: ok ? 'ok' : 'degraded',
    service: 'bigtable-service',
    checks,
  });
}

Python (google-cloud-bigtable)

# routers/health.py
import os, time
from google.cloud import bigtable
from fastapi import APIRouter

router = APIRouter()

_client = bigtable.Client(project=os.environ['GCLOUD_PROJECT'], admin=False)
_instance = _client.instance(os.environ['BIGTABLE_INSTANCE_ID'])
_table = _instance.table(os.environ['BIGTABLE_TABLE_ID'])

HEALTH_ROW_KEY = b'_vigilmon_healthcheck'
COLUMN_FAMILY = os.environ.get('BIGTABLE_COLUMN_FAMILY', 'cf1')

@router.get("/health")
async def health():
    checks = {}
    ok = True
    start = time.monotonic()

    # Check 1: Write probe
    try:
        row = _table.direct_row(HEALTH_ROW_KEY)
        row.set_cell(
            COLUMN_FAMILY,
            b'timestamp',
            str(int(time.time() * 1000)).encode(),
        )
        row.commit()
        checks['write'] = 'ok'
        checks['writeLatencyMs'] = int((time.monotonic() - start) * 1000)
    except Exception as e:
        checks['write'] = f'error: {e}'
        ok = False

    # Check 2: Read probe
    read_start = time.monotonic()
    try:
        row = _table.read_row(HEALTH_ROW_KEY)
        checks['read'] = 'ok' if row else 'row not found'
        checks['readLatencyMs'] = int((time.monotonic() - read_start) * 1000)
        if not row:
            ok = False
    except Exception as e:
        checks['read'] = f'error: {e}'
        ok = False

    return {
        'status': 'ok' if ok else 'degraded',
        'service': 'bigtable-service',
        'checks': checks,
    }

Important: Use a dedicated row key like _vigilmon_healthcheck that sorts outside your normal data key range. This prevents the health probe row from appearing in your production queries and avoids polluting your row key space.

Step 2: Configure the Vigilmon HTTP Monitor

  1. Log in to Vigilmon → Add Monitor → HTTP.
  2. URL: your service's health endpoint (e.g. https://api.example.com/health).
  3. Check interval: 60 seconds.
  4. Response timeout: 15 seconds.
  5. Expected status: 200.
  6. JSON assertion: path status, expected value ok.
  7. Click Save.

Tip: Bigtable's gRPC client may exhibit connection re-establishment latency when the underlying connection has been idle. If your service has low query traffic, set the response timeout to 15 seconds to accommodate initial connection setup. Consider adding a keepalive ping in your Bigtable client configuration to avoid this on warm paths.

For high-throughput services, add a second monitor targeting a /health/replication endpoint that verifies your Bigtable replication cluster is in sync — useful for multi-cluster instances serving read traffic from a secondary cluster.

Step 3: Heartbeat Monitoring for Background Bigtable Jobs

Bigtable is commonly used as the sink for streaming ingestion jobs (Cloud Dataflow, Apache Beam, Pub/Sub consumers) and batch analytics pipelines. These have no HTTP endpoint to probe. Use Vigilmon's Heartbeat monitor to confirm they run on schedule.

  1. In Vigilmon, click Add Monitor → Cron Heartbeat.
  2. Set the expected ping interval to match your job frequency.
  3. Copy the heartbeat URL.
  4. Ping the heartbeat at the end of each successful job run:
// Node.js — scheduled Bigtable ingestion job
import fetch from 'node-fetch';
import { Bigtable } from '@google-cloud/bigtable';

export async function runIngestionBatch() {
  const bigtable = new Bigtable({ projectId: process.env.GCLOUD_PROJECT });
  const table = bigtable.instance(process.env.BIGTABLE_INSTANCE_ID)
                         .table(process.env.BIGTABLE_TABLE_ID);
  try {
    const rows = await fetchRecordsToIngest();  // your data source
    await table.insert(rows.map(r => ({
      key: r.rowKey,
      data: { cf1: { value: { value: JSON.stringify(r.payload) } } },
    })));

    // Signal success to Vigilmon
    await fetch(process.env.VIGILMON_HEARTBEAT_URL, { method: 'POST' });
    console.log(`Ingested ${rows.length} rows — heartbeat sent.`);
  } catch (err) {
    console.error('Ingestion failed — heartbeat NOT sent:', err.message);
  }
}

# Python — batch Bigtable write job
import os, requests
from google.cloud import bigtable

def run_batch_write():
    client = bigtable.Client(project=os.environ['GCLOUD_PROJECT'])
    table = client.instance(os.environ['BIGTABLE_INSTANCE_ID']).table(
        os.environ['BIGTABLE_TABLE_ID']
    )
    try:
        records = fetch_records_to_ingest()  # your data source
        rows = []
        for r in records:
            row = table.direct_row(r['row_key'].encode())
            row.set_cell('cf1', b'payload', r['payload'].encode())
            rows.append(row)
        table.mutate_rows(rows)

        # Signal success to Vigilmon
        requests.post(os.environ['VIGILMON_HEARTBEAT_URL'], timeout=5)
        print(f'Ingested {len(rows)} rows — heartbeat sent.')
    except Exception as e:
        print(f'Batch write failed — heartbeat NOT sent: {e}')

Step 4: Alert Routing

| Monitor type | Target | Alert channel | Priority | |---|---|---|---| | HTTP | /health (write + read probe) | Email + PagerDuty | Critical | | HTTP | /health (read latency assertion) | Slack | Warning | | HTTP | /health/replication | Slack + Email | High | | Cron Heartbeat | Ingestion pipeline | Email + PagerDuty | Critical | | Cron Heartbeat | Batch analytics job | Slack | Warning |

Set consecutive failures to 2 before alerting. Bigtable's gRPC connection layer occasionally retries transparently, and a single probe timeout during connection re-establishment is not a real service failure.

What Vigilmon Catches That Cloud Monitoring Misses

| Scenario | Cloud Monitoring | Vigilmon | |---|---|---| | IAM permission change blocks app reads | No app-layer metric | Health read probe fails → 503 fires | | Table deleted or column family removed | No app-layer alert | Health write probe fails immediately | | App profile routing to wrong cluster | Metric shows healthy | Health probe reveals replication lag | | Ingestion job silently stopped | No job-level metric | Heartbeat grace period expires → alert | | Write latency spike (hot row key / hotspot) | Latency metric exists | Write latency assertion triggers | | External network to Bigtable endpoint lost | Not externally observable | Vigilmon probe is external and independent |

Bigtable's managed infrastructure is highly reliable, but your application's relationship to that infrastructure — credentials, app profiles, schema, and ingestion pipelines — adds layers that managed monitoring cannot see. Vigilmon closes that gap with external health checks that verify what your users actually experience.

Start monitoring your Bigtable applications in under 5 minutes — register free at vigilmon.online.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles