Monitoring Zalando Postgres Operator with Vigilmon

Zalando Postgres Operator (also known as postgres-operator or spilo) provisions and manages highly-available PostgreSQL clusters on Kubernetes using Patroni for leader election and failover. It handles the complexity of streaming replication, switchover, and rolling upgrades — but it ships no external availability dashboard. When a Patroni failover stalls, the operator webhook goes silent, or your primary endpoint becomes unreachable, application queries fail while Kubernetes reports pods as Running. Vigilmon closes that gap with external health checks, TCP endpoint monitoring, and dead-man's-switch heartbeats for your database fleet.

What You'll Set Up

• HTTP health probe for the postgres-operator controller
• TCP connectivity check on your primary PostgreSQL endpoint
• Patroni REST API readiness probe for leader health
• Heartbeat from a periodic query job to confirm end-to-end database availability
• Alerting tuned for database failover timing

Prerequisites

• Kubernetes cluster with Zalando Postgres Operator installed (v1.10+)
• A postgresql custom resource deployed (e.g., acid-minimal-cluster)
• kubectl access to the postgres-operator and cluster namespaces
• A free Vigilmon account

Step 1: Expose the Operator Health Endpoint

The postgres-operator controller exposes a /health endpoint on port 8080. Make it reachable for external monitoring:

Option A — NodePort (quick)

kubectl patch svc postgres-operator \
  -n postgres-operator \
  --type='json' \
  -p='[{"op":"replace","path":"/spec/type","value":"NodePort"},{"op":"add","path":"/spec/ports/0/nodePort","value":30080}]'

Then add a monitor in Vigilmon:

1. Log in to vigilmon.online and click Add Monitor.
2. Set Type to HTTP / HTTPS.
3. Enter http://<node-ip>:30080/health.
4. Set Check interval to 1 minute.
5. Set Expected HTTP status to 200.
6. Click Save.

Option B — Ingress (recommended for production)

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: postgres-operator-health
  namespace: postgres-operator
  annotations:
    nginx.ingress.kubernetes.io/backend-protocol: "HTTP"
spec:
  rules:
  - host: pg-operator-health.internal.example.com
    http:
      paths:
      - path: /health
        pathType: Prefix
        backend:
          service:
            name: postgres-operator
            port:
              number: 8080

Apply and add http://pg-operator-health.internal.example.com/health as an HTTP monitor in Vigilmon.

Step 2: Monitor the Primary PostgreSQL TCP Endpoint

Zalando Postgres Operator creates a <cluster-name> service that points to the Patroni primary. Monitor TCP connectivity on port 5432 to detect failovers that leave the primary unreachable:

# Get the cluster service IP
kubectl get svc acid-minimal-cluster -n default
# NAME                    TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE
# acid-minimal-cluster    ClusterIP   10.96.150.42    <none>        5432/TCP   14d

Expose it for external TCP monitoring:

kubectl patch svc acid-minimal-cluster \
  -n default \
  --type='json' \
  -p='[{"op":"replace","path":"/spec/type","value":"NodePort"},{"op":"add","path":"/spec/ports/0/nodePort","value":30432}]'

In Vigilmon:

1. Click Add Monitor → TCP Port.
2. Enter your node IP and 30432.
3. Set Check interval to 1 minute.
4. Set Alert after to 2 consecutive failures (one failure can occur during a normal Patroni switchover).
5. Click Save.

When a Patroni failover stalls — the old primary died but no replica was promoted — this TCP check will alert within 2 minutes, before applications start throwing connection errors.

Step 3: Probe the Patroni REST API

Each PostgreSQL pod runs a Patroni REST API on port 8008 that reports leader/replica state. Monitoring the primary pod's /primary endpoint gives you direct visibility into Patroni's view of cluster health:

# Port-forward to test the endpoint
kubectl port-forward pod/acid-minimal-cluster-0 8008:8008 -n default &
curl http://localhost:8008/primary
# Returns 200 if this pod is the current primary, 503 otherwise

Expose the primary pod's Patroni API via a separate NodePort service:

apiVersion: v1
kind: Service
metadata:
  name: patroni-health
  namespace: default
spec:
  type: NodePort
  selector:
    application: spilo
    cluster-name: acid-minimal-cluster
    spilo-role: master
  ports:
  - name: patroni-api
    port: 8008
    targetPort: 8008
    nodePort: 30808

kubectl apply -f patroni-health-svc.yaml

Add a Vigilmon HTTP monitor for http://<node-ip>:30808/primary:

• Expected status: 200
• Check interval: 1 minute
• Alert after: 2 consecutive failures

This detects split-brain scenarios where TCP to port 5432 is open but Patroni does not consider any pod to be the primary.

Step 4: Heartbeat from a Periodic Database Query

The most valuable database monitor is an end-to-end availability check: a Kubernetes CronJob that connects to PostgreSQL, executes a query, then pings Vigilmon's heartbeat URL. If the ping stops arriving, Vigilmon alerts.

apiVersion: batch/v1
kind: CronJob
metadata:
  name: vigilmon-postgres-heartbeat
  namespace: default
spec:
  schedule: "*/5 * * * *"
  jobTemplate:
    spec:
      template:
        spec:
          restartPolicy: Never
          containers:
          - name: heartbeat
            image: postgres:15-alpine
            env:
            - name: PGPASSWORD
              valueFrom:
                secretKeyRef:
                  name: postgres.acid-minimal-cluster.credentials.postgresql.acid.zalan.do
                  key: password
            command:
            - sh
            - -c
            - |
              psql -h acid-minimal-cluster -U postgres -c "SELECT 1" && \
              wget -qO- https://vigilmon.online/api/heartbeat/<YOUR_HEARTBEAT_ID>

Set up the Vigilmon heartbeat monitor

1. In Vigilmon, click Add Monitor → Heartbeat.
2. Set Expected interval to 5 minutes.
3. Set Grace period to 3 minutes (allows for pod scheduling overhead).
4. Copy the heartbeat URL and paste it into the CronJob manifest above.
5. Apply: kubectl apply -f postgres-heartbeat-cronjob.yaml.

If Patroni fails to elect a primary, the connection fails, the ping never arrives, and Vigilmon fires an alert within 8 minutes — before your application SLA is breached.

Step 5: Configure Notifications

1. In Vigilmon, go to Settings → Notifications.
2. Add your Slack webhook, PagerDuty integration key, or on-call email.
3. Assign the notification channel to all four monitors.
4. Recommended escalation delays:
   • Operator health: 2 minutes (a crashed operator needs immediate attention)
   • Primary TCP: 2 minutes (failover should complete in under 30 seconds normally)
   • Patroni /primary: 2 minutes
   • Heartbeat: 10 minutes (allow for slow job scheduling)

What You've Built

| Monitor | Type | Checks | |---|---|---| | Operator /health | HTTP | Controller process alive and responsive | | Primary TCP port 5432 | TCP | Database socket reachable | | Patroni /primary | HTTP | Patroni leader election healthy | | DB query heartbeat | Heartbeat | End-to-end query path working |

This four-layer setup catches operator crashes, failover stalls, split-brain states, and silent connection failures — the failure modes most likely to cause application downtime without obvious Kubernetes-level alerts. With Vigilmon monitoring your Zalando Postgres Operator clusters, you'll know about database availability issues before your users do.