Monitoring CloudNativePG: K8s-Native PostgreSQL Cluster Health and Vigilmon Integration

CloudNativePG (CNPG) brings first-class PostgreSQL lifecycle management to Kubernetes through a custom operator. It handles HA failover, WAL archiving, and scheduled backups — but only if you're watching. This guide walks through what to monitor in a production CNPG deployment, from pod health to WAL archiving failures, and how Vigilmon adds an external verification layer.

CloudNativePG Architecture Refresher

A CNPG cluster consists of:

• A primary PostgreSQL instance (read/write)
• One or more replica instances (read-only, streaming replication from primary)
• A pooler (optional PgBouncer sidecar)
• Scheduled backup CronJobs

The operator reconciles the desired state and performs switchovers when the primary fails. Your monitoring must track all layers.

Cluster Status via kubectl

The fastest health check is the CNPG cluster resource status:

kubectl get cluster my-cluster -n postgres -o yaml | yq '.status'

Key fields to watch:

status:
  phase: Cluster in healthy state      # alert if not this
  currentPrimary: my-cluster-1         # track for switchover events
  readyInstances: 3                     # should equal instances count
  instances: 3
  conditions:
    - type: Ready
      status: "True"                    # alert if False
    - type: ContinuousArchiving
      status: "True"                    # alert if False
    - type: LastBackupSucceeded
      status: "True"                    # alert if False

Script this into a Kubernetes readiness probe or a monitoring sidecar.

Pod and Instance Health

Each CNPG instance runs as a Pod. Monitor them like any other K8s workload:

# Check all cluster pods
kubectl get pods -n postgres -l cnpg.io/cluster=my-cluster

# Expected output for a healthy 3-node cluster
NAME             READY   STATUS    RESTARTS   AGE
my-cluster-1     1/1     Running   0          5d
my-cluster-2     1/1     Running   0          5d
my-cluster-3     1/1     Running   0          5d

Alert on:

• Any pod not in Running state
• RESTARTS counter increasing (indicates crash-looping)
• READY showing 0/1

For Prometheus monitoring, CNPG ships with a built-in metrics endpoint. Enable it in your cluster spec:

apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
  name: my-cluster
spec:
  enablePodMonitor: true   # creates a PodMonitor for Prometheus Operator
  monitoring:
    enableDefaultQueries: true

Prometheus Metrics

With enablePodMonitor: true, CNPG exposes metrics on port 9187. Key metrics:

# Is the instance primary or replica?
cnpg_pg_replication_in_recovery == 0  # primary
cnpg_pg_replication_in_recovery == 1  # replica

# Replication lag in bytes
cnpg_pg_replication_lag

# WAL archiving status (0 = ok, 1 = failing)
cnpg_pg_stat_archiver_failed_count

# Connections
cnpg_pg_stat_activity_count

# Transaction rate
cnpg_pg_database_xact_commit_total

Prometheus alert rules for a production cluster:

groups:
  - name: cnpg
    rules:
      - alert: CNPGPrimaryMissing
        expr: count(cnpg_pg_replication_in_recovery == 0) by (cluster) < 1
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "No primary found in CNPG cluster {{ $labels.cluster }}"

      - alert: CNPGReplicationLagHigh
        expr: cnpg_pg_replication_lag > 10485760  # 10 MB
        for: 5m
        labels:
          severity: warning

      - alert: CNPGWALArchivingFailing
        expr: increase(cnpg_pg_stat_archiver_failed_count[5m]) > 0
        labels:
          severity: critical

Switchover Detection

CNPG records switchover events as Kubernetes Events. Monitor them:

kubectl get events -n postgres \
  --field-selector reason=SwitchoverRequired \
  --sort-by='.lastTimestamp'

For automated alerting, watch for SwitchoverRequired and InstancePromoted events in your event router (e.g., event-exporter to Prometheus or Loki).

WAL Archiving Health

WAL archiving is critical for point-in-time recovery. If archiving fails, your backup window shrinks to zero. Monitor pg_stat_archiver:

SELECT 
  archived_count,
  failed_count,
  last_archived_wal,
  last_archived_time,
  last_failed_wal,
  last_failed_time,
  EXTRACT(EPOCH FROM (NOW() - last_archived_time)) AS seconds_since_last_archive
FROM pg_stat_archiver;

Alert when:

• failed_count increases
• seconds_since_last_archive exceeds your WAL segment interval (typically > 300s for active databases)
• last_failed_time is recent

Backup Job Monitoring

CNPG ScheduledBackup resources create CronJobs. Check their status:

kubectl get scheduledbackup -n postgres
kubectl get backup -n postgres --sort-by='.metadata.creationTimestamp' | tail -5

A failed backup looks like:

NAME                      AGE   CLUSTER      METHOD      PHASE
my-cluster-backup-1234    1h    my-cluster   barmanObjectStore   failed

Alert on any backup in failed phase or if no successful backup exists in the last 25 hours.

Vigilmon Integration

CNPG runs inside Kubernetes, but Vigilmon gives you an external vantage point — checking the database endpoint from outside the cluster, independent of your internal monitoring stack.

Step 1 — expose a health endpoint

Deploy a lightweight health service that queries CNPG status:

from flask import Flask, jsonify
from kubernetes import client, config
import psycopg2

app = Flask(__name__)

@app.route("/health/cnpg")
def health():
    results = {}

    # Check DB connectivity
    try:
        conn = psycopg2.connect(
            host="my-cluster-rw.postgres.svc.cluster.local",
            dbname="app",
            user="app",
            password="secret",
            connect_timeout=3
        )
        conn.close()
        results["db_connect"] = "ok"
    except Exception as e:
        results["db_connect"] = f"error: {e}"
        return jsonify(results), 503

    # Check replication lag via SQL
    try:
        conn = psycopg2.connect(
            host="my-cluster-ro.postgres.svc.cluster.local",
            dbname="app", user="app", password="secret",
            connect_timeout=3
        )
        cur = conn.cursor()
        cur.execute("""
            SELECT EXTRACT(EPOCH FROM (now() - pg_last_xact_replay_timestamp()))
            AS replication_lag_seconds
        """)
        lag = cur.fetchone()[0]
        results["replication_lag_seconds"] = lag
        conn.close()
        if lag and lag > 30:
            results["replication_lag"] = "lagging"
            return jsonify(results), 503
        results["replication_lag"] = "ok"
    except Exception as e:
        results["replication_lag"] = f"error: {e}"

    return jsonify(results), 200

if __name__ == "__main__":
    app.run(host="0.0.0.0", port=9090)

Step 2 — expose the service externally

apiVersion: v1
kind: Service
metadata:
  name: cnpg-health-svc
  namespace: postgres
spec:
  type: LoadBalancer  # or NodePort + ingress
  selector:
    app: cnpg-health
  ports:
    - port: 9090
      targetPort: 9090

Step 3 — configure Vigilmon monitors

In Vigilmon, create:

1. HTTP monitor → https://your-cluster-health-endpoint/health/cnpg — interval 60s, expects 200.
2. TCP monitor → primary endpoint my-cluster-rw:5432 — verifies the TCP socket is open.
3. TCP monitor → read replica endpoint my-cluster-ro:5432.

Set alert escalation to your on-call rotation and enable status-page incidents for database outages.

Alert Summary

| Check | Threshold | Severity | |-------|-----------|----------| | Cluster phase | Not "healthy" | Critical | | Primary missing | 0 primaries | Critical | | Replication lag | > 10 MB | Warning | | WAL archiving failed | Any failure | Critical | | Last backup | > 25h ago | Warning | | Pod restarts | > 3 in 1h | Warning |

Conclusion

CloudNativePG abstracts a lot of PostgreSQL operational complexity, but it does not provide external observability out of the box. Pair the built-in Prometheus metrics with kubectl event monitoring for internal health, and use Vigilmon HTTP and TCP checks for an external verification layer that's independent of your in-cluster monitoring stack. With both in place, you'll know about primary failures, replication lag, and backup breakage before they become user-facing incidents.