Google Cloud Spanner offers 99.999% SLA on multi-region instances, but your application's availability depends on more than Spanner itself — it depends on session pool health, IAM permissions, network latency from your service to Spanner, and your own query logic. GCP Cloud Monitoring can alert on Spanner-side metrics, but it can't tell you whether your application can actually execute a query. Vigilmon adds an independent external layer: HTTP health checks on your Spanner-connected services that probe real database connectivity, session pool status, and latency bounds.
This tutorial shows you how to build meaningful monitoring for Cloud Spanner-backed applications using Vigilmon.
What You'll Build
- A health endpoint that probes real Spanner connectivity with a lightweight read
- A Vigilmon HTTP monitor with appropriate timeout and assertion settings
- A heartbeat pattern for Spanner-backed batch jobs
- A latency-aware health check that degrades gracefully under high load
- An alerting strategy for Spanner unavailability and session exhaustion
Prerequisites
- A Google Cloud project with a Spanner instance and database
- A service account or Workload Identity with
spanner.databases.readpermission - An application connected to Spanner (Node.js, Python, Go, or Java)
- A free account at vigilmon.online
Step 1: Add a Health Endpoint to Your Spanner Service
Add a /health endpoint that executes a minimal Spanner read to verify real connectivity — not just that the client library loaded.
Node.js (@google-cloud/spanner)
// routes/health.js
import { Spanner } from '@google-cloud/spanner';
const spanner = new Spanner({ projectId: process.env.GCP_PROJECT_ID });
const instance = spanner.instance(process.env.SPANNER_INSTANCE_ID);
const database = instance.database(process.env.SPANNER_DATABASE_ID);
export async function healthHandler(req, res) {
const checks = {};
let ok = true;
const start = Date.now();
try {
// Lightweight read: SELECT 1 verifies session acquisition and query execution
const [rows] = await database.run({ sql: 'SELECT 1 AS probe', timeout: 5000 });
const latencyMs = Date.now() - start;
checks.spanner = 'ok';
checks.latencyMs = latencyMs;
const warnLatencyMs = parseInt(process.env.SPANNER_LATENCY_WARN_MS || '2000', 10);
if (latencyMs > warnLatencyMs) {
checks.spanner = 'slow';
ok = false;
}
} catch (err) {
checks.spanner = `error: ${err.message}`;
ok = false;
}
// Also check session pool health
try {
const pool = database.pool_;
if (pool) {
checks.sessionPoolSize = pool.size;
checks.sessionPoolAvailable = pool.available;
if (pool.available === 0 && pool.size >= (pool.max || 100)) {
checks.sessionPool = 'exhausted';
ok = false;
} else {
checks.sessionPool = 'ok';
}
}
} catch (_) {
// Pool introspection is optional — don't fail health check for this
}
res.status(ok ? 200 : 503).json({
status: ok ? 'ok' : 'degraded',
service: 'spanner-service',
checks,
});
}
Python (google-cloud-spanner)
# routers/health.py
import os, time
from google.cloud import spanner
from fastapi import APIRouter
router = APIRouter()
_client = spanner.Client(project=os.environ['GCP_PROJECT_ID'])
_instance = _client.instance(os.environ['SPANNER_INSTANCE_ID'])
_database = _instance.database(os.environ['SPANNER_DATABASE_ID'])
@router.get("/health")
async def health():
checks = {}
ok = True
start = time.monotonic()
try:
with _database.snapshot() as snapshot:
results = list(snapshot.execute_sql("SELECT 1 AS probe"))
latency_ms = int((time.monotonic() - start) * 1000)
checks['spanner'] = 'ok'
checks['latencyMs'] = latency_ms
warn_ms = int(os.environ.get('SPANNER_LATENCY_WARN_MS', '2000'))
if latency_ms > warn_ms:
checks['spanner'] = 'slow'
ok = False
except Exception as e:
checks['spanner'] = f'error: {str(e)}'
ok = False
return {
'status': 'ok' if ok else 'degraded',
'service': 'spanner-service',
'checks': checks,
}
Go (cloud.google.com/go/spanner)
// internal/health/handler.go
package health
import (
"context"
"encoding/json"
"net/http"
"os"
"time"
"cloud.google.com/go/spanner"
"google.golang.org/api/iterator"
)
func Handler(client *spanner.Client) http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
checks := map[string]any{}
ok := true
start := time.Now()
ctx, cancel := context.WithTimeout(r.Context(), 5*time.Second)
defer cancel()
iter := client.Single().Query(ctx, spanner.Statement{SQL: "SELECT 1 AS probe"})
defer iter.Stop()
_, err := iter.Next()
if err != nil && err != iterator.Done {
checks["spanner"] = "error: " + err.Error()
ok = false
} else {
latencyMs := time.Since(start).Milliseconds()
checks["spanner"] = "ok"
checks["latencyMs"] = latencyMs
if latencyMs > 2000 {
checks["spanner"] = "slow"
ok = false
}
}
w.Header().Set("Content-Type", "application/json")
if !ok {
w.WriteHeader(http.StatusServiceUnavailable)
}
json.NewEncoder(w).Encode(map[string]any{
"status": map[bool]string{true: "ok", false: "degraded"}[ok],
"service": "spanner-service",
"checks": checks,
})
}
}
Step 2: Configure the Vigilmon HTTP Monitor
- Log in to Vigilmon → Add Monitor → HTTP.
- URL: your service's health endpoint (e.g.
https://api.example.com/health). - Check interval: 60 seconds.
- Response timeout: 10 seconds — Spanner cold session acquisition can take 2–3 s.
- Expected status:
200. - JSON assertion: path
status, expected valueok.
Tip: For multi-region Spanner instances, deploy your health endpoint in the same region as your application. A health check from the wrong region may show healthy Spanner but hide regional network issues affecting your users.
Step 3: Heartbeat Monitoring for Spanner Batch Jobs
Spanner-backed batch jobs (data exports, aggregation pipelines, replication jobs) are invisible to HTTP monitors. Use Vigilmon's Heartbeat monitor and ping it after each successful batch completion.
# batch/spanner_export_job.py
import os, requests
from google.cloud import spanner
def run_export():
client = spanner.Client(project=os.environ['GCP_PROJECT_ID'])
instance = client.instance(os.environ['SPANNER_INSTANCE_ID'])
database = instance.database(os.environ['SPANNER_DATABASE_ID'])
rows_exported = 0
with database.snapshot() as snapshot:
results = snapshot.execute_sql(
"SELECT * FROM events WHERE processed = FALSE LIMIT 10000"
)
for row in results:
export_row(row)
rows_exported += 1
# Only ping if export completed successfully
if rows_exported > 0:
requests.post(os.environ['VIGILMON_HEARTBEAT_URL'], timeout=5)
print(f"Exported {rows_exported} rows — heartbeat sent")
run_export()
In Vigilmon, create a Heartbeat monitor:
- Set the grace period to 2× your expected job interval.
- A missed heartbeat means the job failed, Spanner was unreachable, or the export produced zero rows — all conditions worth alerting on.
Step 4: Latency Degradation Alert
Spanner latency can increase during hot-spotting, schema changes, or regional failovers. Set a tighter latency threshold for your health check to catch degradation before users feel it.
// Tighten the health check for SLA-sensitive services
const LATENCY_LEVELS = {
healthy: parseInt(process.env.LATENCY_OK_MS || '500', 10),
degraded: parseInt(process.env.LATENCY_WARN_MS || '2000', 10),
critical: parseInt(process.env.LATENCY_ERROR_MS || '5000', 10),
};
function classifyLatency(ms) {
if (ms < LATENCY_LEVELS.healthy) return { level: 'healthy', httpStatus: 200 };
if (ms < LATENCY_LEVELS.degraded) return { level: 'slow', httpStatus: 200 };
if (ms < LATENCY_LEVELS.critical) return { level: 'degraded', httpStatus: 503 };
return { level: 'critical', httpStatus: 503 };
}
Configure Vigilmon to assert status equals ok — the slow level stays green in Vigilmon while your application-level metrics track latency trends separately.
Step 5: Alerting Strategy
| Alert channel | When to use | |---|---| | Email / PagerDuty | Service returns 503 (Spanner unreachable or session pool exhausted) | | Slack webhook | Heartbeat missed (batch job stalled) | | Slack + Email | Latency critical threshold — user-facing queries >5 s | | Status page | Public API backed by Cloud Spanner |
Configure alert escalation in Vigilmon: notify immediately, re-notify after 5 minutes. Cloud Spanner unavailability in a multi-region instance is rare but severe when it occurs.
What Vigilmon Catches That GCP Cloud Monitoring Misses
| Scenario | GCP Cloud Monitoring | Vigilmon | |---|---|---| | Application can't acquire a session | No built-in application-layer check | HTTP monitor catches 503 immediately | | IAM permission revoked | No alerting for auth failures at app layer | Health endpoint catches on next check | | Session pool exhausted | Needs custom metric + alert | Health endpoint reports pool saturation | | Batch job silently stops | Needs custom metric pipeline | Heartbeat grace period fires alert | | GCP Console or Monitoring is down | Your alert path may also fail | Vigilmon is external — unaffected | | Cross-region network latency spike | Spanner metrics look fine | Latency-aware health check catches it |
Cloud Spanner's globally distributed design makes your database resilient, but resilience at the database layer doesn't protect you from application-level failures. Session exhaustion, IAM misconfigurations, and network latency spikes all look like silence from the outside. External health checks from Vigilmon give you the independent signal you need.
Start monitoring your Cloud Spanner applications in under 5 minutes — register free at vigilmon.online.