Monitoring CrunchyData PGO (Postgres Operator) with Vigilmon

CrunchyData Postgres Operator (PGO) v5 manages production-grade PostgreSQL clusters on Kubernetes using the postgres-operator.crunchydata.com API group. It handles high availability through Patroni, connection pooling via PgBouncer, and backup scheduling — but it ships no built-in external availability dashboard. When a PgBouncer pod crashes, the primary fails over, or the operator itself stops reconciling, application connections break while Kubernetes shows pods as Running. Vigilmon closes that gap with external health checks, TCP connectivity monitoring, and heartbeat probes for your CrunchyData clusters.

What You'll Set Up

• HTTP health probe for the PGO operator
• TCP connectivity check on the PgBouncer connection pool endpoint
• Direct PostgreSQL primary TCP check as a fallback
• Heartbeat from a periodic query job to confirm end-to-end availability
• Alerting tuned for HA failover and PgBouncer restart timing

Prerequisites

• Kubernetes cluster with PGO v5 installed (postgres-operator.crunchydata.com)
• A PostgresCluster resource deployed (e.g., hippo)
• kubectl access to the postgres-operator namespace
• A free Vigilmon account

Step 1: Expose the PGO Operator Health Endpoint

The PGO operator pod exposes a metrics and health endpoint on port 8080. Expose it for external monitoring:

# Verify the port
kubectl get pod -n postgres-operator -l app.kubernetes.io/name=pgo -o jsonpath='{.items[0].spec.containers[0].ports}'

Create a NodePort service pointing at the operator health path:

apiVersion: v1
kind: Service
metadata:
  name: pgo-operator-health
  namespace: postgres-operator
spec:
  type: NodePort
  selector:
    app.kubernetes.io/name: pgo
  ports:
  - name: health
    port: 8080
    targetPort: 8080
    nodePort: 30083

kubectl apply -f pgo-operator-health-svc.yaml

Add a monitor in Vigilmon:

1. Log in to vigilmon.online and click Add Monitor.
2. Set Type to HTTP / HTTPS.
3. Enter http://<node-ip>:30083/healthz.
4. Set Check interval to 1 minute.
5. Set Expected HTTP status to 200.
6. Click Save.

Step 2: Monitor the PgBouncer Connection Pool

PGO deploys PgBouncer as the recommended entry point for application connections. Its service is named <cluster>-pgbouncer in your cluster namespace. Monitor it on TCP port 5432 to detect pool crashes before they cascade to your application:

# Get PgBouncer service details
kubectl get svc hippo-pgbouncer -n postgres-operator
# NAME               TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE
# hippo-pgbouncer    ClusterIP   10.96.77.200    <none>        5432/TCP   7d

Expose PgBouncer for external TCP monitoring:

kubectl patch svc hippo-pgbouncer \
  -n postgres-operator \
  --type='json' \
  -p='[{"op":"replace","path":"/spec/type","value":"NodePort"},{"op":"add","path":"/spec/ports/0/nodePort","value":30433}]'

In Vigilmon:

1. Click Add Monitor → TCP Port.
2. Enter your node IP and 30433.
3. Set Check interval to 1 minute.
4. Set Alert after to 2 consecutive failures.
5. Click Save.

PgBouncer restarts quickly (under 10 seconds) but can be unavailable long enough for application connection errors to spike. This TCP check will alert within 2 minutes if PgBouncer becomes persistently unreachable.

Step 3: Monitor the Primary PostgreSQL Port Directly

PGO also creates a <cluster>-primary service that resolves directly to the Patroni primary pod, bypassing PgBouncer. Monitoring this service separately helps distinguish between PgBouncer failures and actual PostgreSQL primary failures:

kubectl get svc hippo-primary -n postgres-operator

Expose it:

kubectl patch svc hippo-primary \
  -n postgres-operator \
  --type='json' \
  -p='[{"op":"replace","path":"/spec/type","value":"NodePort"},{"op":"add","path":"/spec/ports/0/nodePort","value":30532}]'

Add a second TCP monitor in Vigilmon for port 30532. If the PgBouncer TCP check fails but the primary TCP check is healthy, the issue is in your connection pool layer. If both fail, you have a PostgreSQL-level issue.

Step 4: Heartbeat from a Periodic Query Job

Configure an end-to-end availability check using a Kubernetes CronJob that connects through PgBouncer, runs a query, and pings Vigilmon's heartbeat URL:

apiVersion: batch/v1
kind: CronJob
metadata:
  name: vigilmon-pgo-heartbeat
  namespace: postgres-operator
spec:
  schedule: "*/5 * * * *"
  jobTemplate:
    spec:
      template:
        spec:
          restartPolicy: Never
          containers:
          - name: heartbeat
            image: postgres:15-alpine
            env:
            - name: PGPASSWORD
              valueFrom:
                secretKeyRef:
                  name: hippo-pguser-hippo
                  key: password
            - name: PGUSER
              valueFrom:
                secretKeyRef:
                  name: hippo-pguser-hippo
                  key: user
            - name: PGDATABASE
              valueFrom:
                secretKeyRef:
                  name: hippo-pguser-hippo
                  key: dbname
            command:
            - sh
            - -c
            - |
              psql -h hippo-pgbouncer -c "SELECT 1" && \
              wget -qO- https://vigilmon.online/api/heartbeat/<YOUR_HEARTBEAT_ID>

Configure the Vigilmon heartbeat

1. In Vigilmon, click Add Monitor → Heartbeat.
2. Set Expected interval to 5 minutes.
3. Set Grace period to 3 minutes.
4. Paste the heartbeat URL into the CronJob manifest.
5. Apply: kubectl apply -f pgo-heartbeat.yaml.

PGO secrets (hippo-pguser-hippo) follow the naming pattern <cluster>-pguser-<username> and contain host, port, user, password, dbname, and uri keys. The CronJob above reads credentials directly from the secret to avoid hardcoding them.

Step 5: Configure Notifications

1. In Vigilmon, go to Settings → Notifications.
2. Add your Slack webhook, PagerDuty key, or on-call email.
3. Assign the channel to all monitors.
4. Recommended delays:
   • Operator health: 2 minutes (operator crash needs immediate attention)
   • PgBouncer TCP: 2 minutes
   • Primary TCP: 2 minutes
   • Heartbeat: 10 minutes (allow scheduling overhead)

What You've Built

| Monitor | Type | Checks | |---|---|---| | PGO operator /healthz | HTTP | Operator process alive and reconciling | | PgBouncer TCP port 5432 | TCP | Connection pool reachable | | Primary TCP port 5432 | TCP | PostgreSQL primary reachable directly | | DB query heartbeat | Heartbeat | End-to-end query path through PgBouncer |

This four-layer setup isolates failures at each layer of the PGO stack: operator, connection pool, primary PostgreSQL, and application-level query path. With Vigilmon alerting, you'll distinguish a PgBouncer crash from a Patroni failover stall from an operator reconciliation loop failure — and know which team needs to act.