tutorial

How to Monitor CloudNativePG PostgreSQL Clusters with Vigilmon

CloudNativePG is a Kubernetes operator that manages the full lifecycle of PostgreSQL clusters — primary election, replication, failover, backups, and connect...

CloudNativePG is a Kubernetes operator that manages the full lifecycle of PostgreSQL clusters — primary election, replication, failover, backups, and connection pooling — entirely inside your cluster. It's a battle-tested way to run production Postgres on Kubernetes, but the operator model introduces a layer of abstraction that makes database health monitoring more complex, not simpler.

In this tutorial you'll set up external monitoring for your CloudNativePG clusters using Vigilmon — free tier, no credit card required.

Why CloudNativePG needs external monitoring

CloudNativePG ships with a Prometheus metrics endpoint and integrates with the standard Kubernetes health probe system. So why add external monitoring?

Because the Kubernetes control plane and CloudNativePG operator can report everything as healthy while your database is unreachable from applications:

  • Primary failover in progress — a failover takes 10–30 seconds; during this window all writes fail, but the operator reports the cluster as Healthy once the new primary is elected
  • Connection pooler (PgBouncer) failure — the CloudNativePG-managed PgBouncer pod crashes; the database is fine but no application can connect
  • LoadBalancer or Service routing broken — the ClusterIP or LoadBalancer service for the cluster stops routing traffic after a node replacement
  • Replica lag exceeds threshold — a replica falls hours behind the primary; read traffic routed there returns stale data
  • Backup job failure — WAL archiving or scheduled base backups silently fail; you discover this during a recovery

For each of these, CloudNativePG may show Cluster: Healthy while your application is either down or silently degraded. External monitoring catches what the operator can't see.

What you'll need

  • A Kubernetes cluster with CloudNativePG installed (v1.20+ recommended)
  • A running Cluster resource with at least one instance
  • A free Vigilmon account (sign up takes 30 seconds)

Step 1: Deploy a CloudNativePG cluster

Here's a production-ready CloudNativePG cluster manifest with connection pooling and monitoring enabled:

# postgres-cluster.yaml
apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
  name: pg-production
  namespace: databases
spec:
  instances: 3
  primaryUpdateStrategy: unsupervised

  postgresql:
    parameters:
      max_connections: "200"
      shared_buffers: "256MB"
      log_min_duration_statement: "1000"

  bootstrap:
    initdb:
      database: appdb
      owner: appuser
      secret:
        name: pg-production-credentials

  storage:
    size: 50Gi
    storageClass: fast-ssd

  monitoring:
    enablePodMonitor: true

  backup:
    retentionPolicy: "30d"
    barmanObjectStore:
      destinationPath: "s3://your-bucket/postgres-backups"
      s3Credentials:
        accessKeyId:
          name: s3-creds
          key: ACCESS_KEY_ID
        secretAccessKey:
          name: s3-creds
          key: SECRET_ACCESS_KEY
---
# connection pooler
apiVersion: postgresql.cnpg.io/v1
kind: Pooler
metadata:
  name: pg-production-rw
  namespace: databases
spec:
  cluster:
    name: pg-production
  instances: 2
  type: rw
  pgbouncer:
    poolMode: transaction
    parameters:
      max_client_conn: "1000"
      default_pool_size: "25"

Apply and verify:

kubectl apply -f postgres-cluster.yaml

# Watch cluster come up
kubectl get clusters -n databases -w

# Check instance status
kubectl get pods -n databases -l cnpg.io/cluster=pg-production

# Get the primary pod name
kubectl get pods -n databases -l cnpg.io/cluster=pg-production \
  -l role=primary -o jsonpath='{.items[0].metadata.name}'

Step 2: Deploy a health-check sidecar for external probing

CloudNativePG doesn't expose an HTTP endpoint by default — Postgres speaks TCP. The cleanest way to enable Vigilmon HTTP monitoring is to deploy a lightweight health-check proxy alongside the cluster.

# pg-health-proxy.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: pg-health-proxy
  namespace: databases
spec:
  replicas: 1
  selector:
    matchLabels:
      app: pg-health-proxy
  template:
    metadata:
      labels:
        app: pg-health-proxy
    spec:
      containers:
        - name: proxy
          image: wrouesnel/postgres_exporter:latest
          env:
            - name: DATA_SOURCE_NAME
              valueFrom:
                secretKeyRef:
                  name: pg-production-credentials
                  key: uri
          ports:
            - containerPort: 9187
              name: metrics
          livenessProbe:
            httpGet:
              path: /metrics
              port: 9187
            initialDelaySeconds: 10
            periodSeconds: 30
---
apiVersion: v1
kind: Service
metadata:
  name: pg-health-proxy
  namespace: databases
spec:
  type: LoadBalancer
  selector:
    app: pg-health-proxy
  ports:
    - port: 9187
      targetPort: 9187
      protocol: TCP

Alternatively, deploy a minimal Go health server that actually queries the database:

// pg-health/main.go
package main

import (
    "database/sql"
    "encoding/json"
    "log"
    "net/http"
    "os"
    "time"

    _ "github.com/lib/pq"
)

var db *sql.DB

func healthHandler(w http.ResponseWriter, r *http.Request) {
    ctx := r.Context()
    var version string

    err := db.QueryRowContext(ctx, "SELECT version()").Scan(&version)
    status := "ok"
    httpStatus := http.StatusOK

    if err != nil {
        status = "error"
        httpStatus = http.StatusServiceUnavailable
    }

    w.Header().Set("Content-Type", "application/json")
    w.WriteHeader(httpStatus)
    json.NewEncoder(w).Encode(map[string]interface{}{
        "status":    status,
        "checked":   time.Now().UTC(),
        "pg_version": version,
    })
}

func main() {
    var err error
    db, err = sql.Open("postgres", os.Getenv("PG_DSN"))
    if err != nil {
        log.Fatal(err)
    }
    db.SetMaxOpenConns(2)
    db.SetConnMaxLifetime(30 * time.Second)

    http.HandleFunc("/health", healthHandler)
    log.Fatal(http.ListenAndServe(":8080", nil))
}

Deploy it and expose it externally so Vigilmon can reach it from outside the cluster.

Step 3: Set up HTTP monitoring in Vigilmon

With your health endpoint accessible, add it to Vigilmon:

  1. Log in to vigilmon.online and go to Monitors → New Monitor
  2. Choose HTTP / HTTPS as the monitor type
  3. Set the URL to your health proxy: https://pg-health.example.com/health
  4. Set the check interval to 1 minute
  5. Under Expected response, configure:
    • Status code: 200
    • Response body contains: "status":"ok"
  6. Save the monitor

What this catches

This monitor fires if:

  • The health proxy pod is down (connection pooler or network failure)
  • The query to PostgreSQL fails (primary crashed, failover in progress)
  • The Kubernetes service stops routing traffic

Step 4: Monitor the TCP layer

In addition to HTTP, monitor the raw TCP port of the CloudNativePG service. This catches load balancer or service routing failures independently of the health proxy:

# Get the external IP of the CloudNativePG read-write service
kubectl get svc -n databases pg-production-rw \
  -o jsonpath='{.status.loadBalancer.ingress[0].ip}'

In Vigilmon:

  1. Go to Monitors → New Monitor
  2. Choose TCP Port
  3. Enter the external IP and port 5432
  4. Save the monitor

If the TCP monitor fires but the HTTP monitor is healthy, you likely have a service routing issue rather than a database failure.

Step 5: Monitor backup health with a heartbeat

CloudNativePG backup failures are silent. Add a heartbeat monitor that your backup job pings on success:

# backup-checker.yaml — a CronJob that verifies recent backups exist
apiVersion: batch/v1
kind: CronJob
metadata:
  name: backup-health-checker
  namespace: databases
spec:
  schedule: "0 * * * *"  # hourly
  jobTemplate:
    spec:
      template:
        spec:
          containers:
            - name: checker
              image: bitnami/kubectl:latest
              command:
                - /bin/sh
                - -c
                - |
                  LATEST=$(kubectl get backup -n databases \
                    -l cnpg.io/cluster=pg-production \
                    --sort-by=.metadata.creationTimestamp \
                    -o jsonpath='{.items[-1].status.phase}')
                  if [ "$LATEST" = "completed" ]; then
                    curl -fsS "https://vigilmon.online/api/v1/heartbeat/YOUR_HEARTBEAT_ID"
                  else
                    echo "Latest backup not completed: $LATEST"
                    exit 1
                  fi
          restartPolicy: Never

Set up the heartbeat in Vigilmon:

  1. Go to Monitors → New Monitor
  2. Choose Heartbeat
  3. Set interval to 2 hours (allows for backup job delay)
  4. Copy the heartbeat URL into your CronJob manifest
  5. Save the monitor

If backups stop completing, you'll get an alert within 2 hours instead of discovering it during a disaster recovery.

Step 6: Configure alert channels and runbook integration

When Vigilmon detects a PostgreSQL cluster issue, you need immediate context. Configure a webhook to enrich alerts with cluster state:

#!/bin/bash
# on-alert.sh — called by your webhook handler
CLUSTER="pg-production"
NS="databases"

echo "=== CloudNativePG Cluster Status ==="
kubectl get cluster $CLUSTER -n $NS -o yaml | grep -A 20 "status:"

echo "=== Pods ==="
kubectl get pods -n $NS -l cnpg.io/cluster=$CLUSTER

echo "=== Recent Events ==="
kubectl get events -n $NS --field-selector involvedObject.name=$CLUSTER \
  --sort-by='.lastTimestamp' | tail -20

echo "=== Replication Lag ==="
kubectl exec -n $NS \
  $(kubectl get pods -n $NS -l cnpg.io/cluster=$CLUSTER,role=primary \
    -o jsonpath='{.items[0].metadata.name}') \
  -- psql -U postgres -c \
  "SELECT client_addr, state, sent_lsn, replay_lsn,
          (sent_lsn - replay_lsn) AS replay_lag
   FROM pg_stat_replication;"

To receive alerts:

  1. In Vigilmon, go to Alert Channels → Add Channel
  2. Choose Email for on-call and Webhook for your automated runbook handler
  3. Assign both channels to all PostgreSQL monitors

Step 7: Create a database status page

Add a status page that gives your application teams visibility into database health:

  1. In Vigilmon, go to Status Pages → New Status Page
  2. Name it "Database Infrastructure"
  3. Add all your CloudNativePG monitors: HTTP health, TCP port, backup heartbeat
  4. Group by cluster name if you run multiple clusters
  5. Publish the URL and share with your backend engineering team

Monitoring reference

| Monitor | Type | What it catches | |---------|------|-----------------| | https://pg-health.example.com/health | HTTP | Primary crashes, failover windows, query failures | | External IP:5432 | TCP | Service routing failures, load balancer issues | | Backup CronJob | Heartbeat | Silent backup failures, WAL archiving errors | | PgBouncer endpoint | HTTP | Connection pooler failures |

Useful CloudNativePG debugging commands when alerts fire:

# Check overall cluster health
kubectl get cluster pg-production -n databases

# Check which pod is primary
kubectl get pods -n databases -l cnpg.io/cluster=pg-production,role=primary

# View operator logs
kubectl logs -n cnpg-system \
  $(kubectl get pods -n cnpg-system -l app.kubernetes.io/name=cloudnative-pg \
    -o jsonpath='{.items[0].metadata.name}') -f

# View recent backups
kubectl get backup -n databases -l cnpg.io/cluster=pg-production \
  --sort-by=.metadata.creationTimestamp

# Trigger a manual failover
kubectl cnpg promote pg-production -n databases

# Check replication status from primary
kubectl exec -n databases \
  $(kubectl get pods -n databases -l cnpg.io/cluster=pg-production,role=primary \
    -o jsonpath='{.items[0].metadata.name}') \
  -- psql -U postgres -c "SELECT * FROM pg_stat_replication;"

What's next

  • SSL certificate monitoring — if you expose PgBouncer via TLS, Vigilmon will alert you before the certificate expires
  • Replica monitoring — add separate monitors for read replicas to detect replication lag before it affects your read-only workloads
  • Multi-cluster coverage — if you run CloudNativePG clusters in multiple namespaces or environments, create a monitor group for each and aggregate them on a single status page

Get started free at vigilmon.online — no credit card, monitors start running in under a minute.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles