tutorial

Monitoring Google AlloyDB with Vigilmon: PostgreSQL Availability, Read Pool Health & Application-Layer Uptime Checks

Practical guide to uptime and health monitoring for Google AlloyDB — HTTP health endpoints for AlloyDB-connected services, primary and read pool endpoint probing, replica lag detection, and independent external monitoring with Vigilmon.

Google AlloyDB delivers up to 4× the throughput of standard PostgreSQL and 100× faster analytical queries while remaining fully PostgreSQL-compatible, but your application's uptime depends on more than AlloyDB's infrastructure guarantees. Primary instance failover lag, read pool replica lag, connection pool exhaustion after an HA event, and IAM authentication failures can all degrade your application silently while Google Cloud Monitoring metrics remain within bounds. Vigilmon adds an independent external layer: HTTP health checks on your AlloyDB-connected services that probe real database connectivity, detect replica lag, and alert on application-level failures before users notice.

This tutorial shows you how to build meaningful monitoring for AlloyDB-backed applications using Vigilmon.

What You'll Build

  • A health endpoint that probes real AlloyDB primary and read pool connectivity
  • Read pool replica lag detection to catch stale reads
  • A Vigilmon HTTP monitor with appropriate timeout and assertion settings
  • A heartbeat monitor for AlloyDB-backed background jobs
  • A failover-aware alerting strategy

Prerequisites

  • A Google Cloud project with an AlloyDB cluster (primary + optional read pool)
  • A service account or Workload Identity with roles/alloydb.client
  • An application connected to AlloyDB (any runtime)
  • A free account at vigilmon.online

Step 1: Add a Health Endpoint to Your AlloyDB Service

AlloyDB uses standard PostgreSQL wire protocol, so any PostgreSQL client works. Add a /health endpoint that probes both the primary instance and the read pool endpoint.

Node.js (pg — node-postgres)

// routes/health.js
import pg from 'pg';

// AlloyDB primary instance IP (from Cloud Console)
const primaryPool = new pg.Pool({
  host: process.env.ALLOYDB_PRIMARY_IP,
  user: process.env.DB_USER,
  password: process.env.DB_PASSWORD,
  database: process.env.DB_NAME,
  port: 5432,
  max: 2,
  connectionTimeoutMillis: 6000,
  idleTimeoutMillis: 10000,
});

// AlloyDB read pool IP (optional — omit if no read pool configured)
const readerPool = process.env.ALLOYDB_READ_POOL_IP
  ? new pg.Pool({
      host: process.env.ALLOYDB_READ_POOL_IP,
      user: process.env.DB_USER,
      password: process.env.DB_PASSWORD,
      database: process.env.DB_NAME,
      port: 5432,
      max: 2,
      connectionTimeoutMillis: 6000,
      idleTimeoutMillis: 10000,
    })
  : null;

export async function healthHandler(req, res) {
  const checks = {};
  let ok = true;
  const start = Date.now();

  // Check 1: Primary instance — verify write path and HA status
  try {
    const result = await primaryPool.query(
      "SELECT pg_is_in_recovery() AS is_standby, version() AS pg_version, now() AS server_time"
    );
    const row = result.rows[0];
    checks.primaryLatencyMs = Date.now() - start;
    checks.postgresVersion = row.pg_version;
    checks.serverTime = row.server_time;

    if (row.is_standby) {
      checks.primary = 'unexpectedly in recovery — HA failover may be in progress';
      ok = false;
    } else {
      checks.primary = 'ok';
    }
  } catch (err) {
    checks.primary = `error: ${err.message}`;
    ok = false;
  }

  // Check 2: Read pool — replica lag
  if (readerPool) {
    try {
      const readerStart = Date.now();
      const result = await readerPool.query(`
        SELECT
          pg_is_in_recovery() AS is_standby,
          EXTRACT(EPOCH FROM (now() - pg_last_xact_replay_timestamp())) AS replica_lag_seconds
      `);
      const row = result.rows[0];
      checks.readerLatencyMs = Date.now() - readerStart;
      checks.replicaLagSeconds = Math.round((row.replica_lag_seconds || 0) * 10) / 10;

      const maxLag = parseFloat(process.env.MAX_REPLICA_LAG_SECONDS || '30');
      if (row.replica_lag_seconds > maxLag) {
        checks.readPool = `replica lag high: ${checks.replicaLagSeconds}s`;
        ok = false;
      } else {
        checks.readPool = 'ok';
      }
    } catch (err) {
      checks.readPool = `error: ${err.message}`;
      ok = false;
    }
  } else {
    checks.readPool = 'not configured';
  }

  res.status(ok ? 200 : 503).json({
    status: ok ? 'ok' : 'degraded',
    service: 'alloydb-service',
    checks,
  });
}

Python (psycopg2)

# routers/health.py
import os, time, psycopg2
from psycopg2 import pool as pg_pool
from fastapi import APIRouter
from fastapi.responses import JSONResponse

router = APIRouter()

primary_pool = pg_pool.SimpleConnectionPool(
    1, 2,
    host=os.environ['ALLOYDB_PRIMARY_IP'],
    user=os.environ['DB_USER'],
    password=os.environ['DB_PASSWORD'],
    dbname=os.environ['DB_NAME'],
    port=5432,
    connect_timeout=6,
)

read_pool_ip = os.environ.get('ALLOYDB_READ_POOL_IP')
reader_pool = pg_pool.SimpleConnectionPool(
    1, 2,
    host=read_pool_ip,
    user=os.environ['DB_USER'],
    password=os.environ['DB_PASSWORD'],
    dbname=os.environ['DB_NAME'],
    port=5432,
    connect_timeout=6,
) if read_pool_ip else None

@router.get("/health")
async def health():
    checks = {}
    ok = True
    start = time.monotonic()

    # Check 1: Primary instance
    conn = None
    try:
        conn = primary_pool.getconn()
        with conn.cursor() as cur:
            cur.execute("SELECT pg_is_in_recovery(), version()")
            is_standby, version = cur.fetchone()
            checks['primaryLatencyMs'] = int((time.monotonic() - start) * 1000)
            checks['postgresVersion'] = version

            if is_standby:
                checks['primary'] = 'unexpectedly in recovery — failover may be in progress'
                ok = False
            else:
                checks['primary'] = 'ok'
    except Exception as e:
        checks['primary'] = f'error: {e}'
        ok = False
    finally:
        if conn:
            primary_pool.putconn(conn)

    # Check 2: Read pool
    if reader_pool:
        conn = None
        try:
            reader_start = time.monotonic()
            conn = reader_pool.getconn()
            with conn.cursor() as cur:
                cur.execute("""
                    SELECT
                        pg_is_in_recovery(),
                        EXTRACT(EPOCH FROM (now() - pg_last_xact_replay_timestamp())) AS lag
                """)
                is_standby, lag = cur.fetchone()
                checks['readerLatencyMs'] = int((time.monotonic() - reader_start) * 1000)
                checks['replicaLagSeconds'] = round(lag or 0, 1)

                max_lag = float(os.environ.get('MAX_REPLICA_LAG_SECONDS', '30'))
                if lag and lag > max_lag:
                    checks['readPool'] = f'replica lag high: {lag:.1f}s'
                    ok = False
                else:
                    checks['readPool'] = 'ok'
        except Exception as e:
            checks['readPool'] = f'error: {e}'
            ok = False
        finally:
            if conn:
                reader_pool.putconn(conn)
    else:
        checks['readPool'] = 'not configured'

    status_code = 200 if ok else 503
    return JSONResponse(status_code=status_code, content={
        'status': 'ok' if ok else 'degraded',
        'service': 'alloydb-service',
        'checks': checks,
    })

Go (pgx)

// internal/health/handler.go
package health

import (
    "context"
    "encoding/json"
    "net/http"
    "time"

    "github.com/jackc/pgx/v5/pgxpool"
)

func Handler(primary *pgxpool.Pool, reader *pgxpool.Pool) http.HandlerFunc {
    return func(w http.ResponseWriter, r *http.Request) {
        checks := map[string]any{}
        ok := true
        ctx, cancel := context.WithTimeout(r.Context(), 8*time.Second)
        defer cancel()

        // Check 1: Primary instance
        start := time.Now()
        var isRecovery bool
        var version string
        err := primary.QueryRow(ctx, "SELECT pg_is_in_recovery(), version()").Scan(&isRecovery, &version)
        if err != nil {
            checks["primary"] = "error: " + err.Error()
            ok = false
        } else {
            checks["primaryLatencyMs"] = time.Since(start).Milliseconds()
            checks["postgresVersion"] = version
            if isRecovery {
                checks["primary"] = "unexpectedly in recovery — failover in progress"
                ok = false
            } else {
                checks["primary"] = "ok"
            }
        }

        // Check 2: Read pool replica lag
        if reader != nil {
            readerStart := time.Now()
            var readIsRecovery bool
            var lagSeconds *float64
            err = reader.QueryRow(ctx,
                "SELECT pg_is_in_recovery(), EXTRACT(EPOCH FROM (now() - pg_last_xact_replay_timestamp()))").
                Scan(&readIsRecovery, &lagSeconds)
            if err != nil {
                checks["readPool"] = "error: " + err.Error()
                ok = false
            } else {
                checks["readerLatencyMs"] = time.Since(readerStart).Milliseconds()
                if lagSeconds != nil {
                    checks["replicaLagSeconds"] = *lagSeconds
                    if *lagSeconds > 30 {
                        checks["readPool"] = "replica lag high"
                        ok = false
                    } else {
                        checks["readPool"] = "ok"
                    }
                } else {
                    checks["readPool"] = "ok"
                }
            }
        } else {
            checks["readPool"] = "not configured"
        }

        w.Header().Set("Content-Type", "application/json")
        if !ok {
            w.WriteHeader(http.StatusServiceUnavailable)
        }
        _ = json.NewEncoder(w).Encode(map[string]any{
            "status":  map[bool]string{true: "ok", false: "degraded"}[ok],
            "service": "alloydb-service",
            "checks":  checks,
        })
    }
}

Verify the endpoint before configuring Vigilmon:

curl -i https://your-app.example.com/health
# HTTP/1.1 200 OK
# {"status":"ok","service":"alloydb-service","checks":{"primary":"ok","primaryLatencyMs":4,"readPool":"ok","replicaLagSeconds":0.1}}

Step 2: Configure the Vigilmon HTTP Monitor

  1. Log in to Vigilmon → Add Monitor → HTTP.
  2. URL: your service's health endpoint (e.g. https://api.example.com/health).
  3. Check interval: 60 seconds.
  4. Response timeout: 10 seconds.
  5. Expected status: 200.
  6. JSON assertion: path status, expected value ok.
  7. Save the monitor.

Create a second monitor targeting /health/reader for read pool isolation — this lets you distinguish total unavailability from degraded read performance during AlloyDB HA events.

AlloyDB connection note: AlloyDB instances are not publicly accessible by default. Your health endpoint must run within the same VPC, or you must use AlloyDB Auth Proxy for secure connectivity. The external Vigilmon probe hits your application's public health endpoint — not the database directly. This is the correct architecture.

Step 3: Heartbeat Monitoring for AlloyDB-Backed Workers

Background jobs that process AlloyDB data (report generation, ETL pipelines, cache warming) will stall silently during an AlloyDB HA failover — the new primary takes over but DNS propagation and connection pool reconnection adds latency. Vigilmon's heartbeat monitors detect silent worker death.

Set Up the Heartbeat Monitor

  1. In Vigilmon, go to Monitors → New Monitor → Heartbeat.
  2. Set the name: alloydb-worker.
  3. Set the expected interval: 5 minutes (adjust to your job frequency).
  4. Set the grace period: 15 minutes (allow for HA failover recovery time).
  5. Save — copy the unique heartbeat URL.

Wire It Into Your Worker

Node.js:

import fetch from 'node-fetch';
import { primaryPool } from './db.js';

async function runBatchJob() {
  const client = await primaryPool.connect();
  try {
    await client.query('BEGIN');
    const result = await client.query(
      'SELECT id FROM pending_jobs WHERE processed_at IS NULL LIMIT 100 FOR UPDATE SKIP LOCKED'
    );

    for (const row of result.rows) {
      await processJob(row.id, client);
    }

    await client.query('COMMIT');

    // Ping Vigilmon only after successful commit
    await fetch(process.env.VIGILMON_HEARTBEAT_URL).catch(() => {});
  } catch (err) {
    await client.query('ROLLBACK');
    console.error('Batch job failed — heartbeat NOT sent:', err.message);
  } finally {
    client.release();
  }
}

setInterval(runBatchJob, 5 * 60 * 1000);

Python:

import os, time, requests, psycopg2

def run_batch_job(conn):
    with conn.cursor() as cur:
        cur.execute("""
            SELECT id FROM pending_jobs
            WHERE processed_at IS NULL
            LIMIT 100
            FOR UPDATE SKIP LOCKED
        """)
        jobs = cur.fetchall()
        for (job_id,) in jobs:
            process_job(job_id, cur)
        conn.commit()

    # Ping Vigilmon after successful commit
    requests.get(os.environ['VIGILMON_HEARTBEAT_URL'], timeout=5)

while True:
    try:
        with psycopg2.connect(...) as conn:
            run_batch_job(conn)
    except Exception as e:
        print(f"Batch job failed — heartbeat NOT sent: {e}")
    time.sleep(300)

Step 4: AlloyDB Columnar Engine Monitoring

AlloyDB's columnar engine accelerates analytical queries by automatically managing columnar storage in memory. When the columnar engine is under memory pressure, queries fall back to row-based scans — a significant performance cliff for OLAP workloads.

Surface columnar engine status in a /health/analytics endpoint:

-- Check columnar engine status
SELECT
  table_schema,
  table_name,
  column_store_pages_total,
  column_store_pages_loaded,
  column_store_memory_used_bytes
FROM
  pg_catalog.pg_stats_ext
WHERE
  column_store_pages_total > 0;

Configure a separate Vigilmon monitor for this endpoint to alert when columnar page load ratios drop below your threshold.

Step 5: Alerting Strategy

| Alert channel | When to use | |---|---| | PagerDuty | Primary returns 503 (unavailable or in HA failover) | | Slack + Email | Primary is unexpectedly in recovery (failover detected) | | Slack webhook | Read pool lag > 30 seconds (stale reads possible) | | PagerDuty | Heartbeat missed (background worker stopped) | | Slack webhook | Columnar engine under pressure (analytical query slowdown) | | Status page | Any user-facing service backed by AlloyDB |

Configure alert escalation in Vigilmon: page immediately on first failure, re-notify after 3 minutes. AlloyDB HA failover typically completes in under 60 seconds, so a 3-minute escalation gives time for automatic recovery without delaying human response to genuine outages.

What Vigilmon Catches That Google Cloud Monitoring Misses

| Scenario | Cloud Monitoring | Vigilmon | |---|---|---| | Primary in recovery during HA failover | No application-layer check | /health catches pg_is_in_recovery() | | Read pool lag growing silently | ReplicationLag metric exists | HTTP assertion on replicaLagSeconds | | Connection pool exhausted | No pool-level check | Health endpoint probe fails → 503 | | Worker stalled after failover | No end-to-end check | Heartbeat grace period expires → alert | | Cloud Monitoring alerts fail to deliver | Alerts silently lost | Vigilmon is external — unaffected | | Auth Proxy token expiry breaks connections | No application-layer check | Health endpoint probe catches error | | Network path to AlloyDB blocked by VPC rules | Internal metric only | External check from app's perspective |

AlloyDB delivers exceptional performance, but application-layer failures during HA failover, read pool lag, and connection pool issues can degrade user experience silently. External health checks with Vigilmon give you the independent signal you need — before your users notice.

Start monitoring your AlloyDB applications in under 5 minutes — register free at vigilmon.online.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles