tutorial

Monitoring Patroni PostgreSQL HA: Switchover Events, DCS Connectivity, and Vigilmon SLA Checks

A practical guide to monitoring Patroni-managed PostgreSQL HA clusters — primary/replica switchover events, etcd/Consul DCS connectivity, replication lag, REST API health polling, and Vigilmon for Postgres HA SLAs.

Patroni automates PostgreSQL failover, but it can't tell your monitoring system when a switchover happened, whether a replica is dangerously behind, or whether the distributed consensus store (etcd or Consul) is healthy enough to run the next election. That gap between "Patroni is running" and "Patroni will fail over correctly under pressure" is exactly where external monitoring earns its keep.

Vigilmon provides external HTTP monitoring for Patroni's REST API, replication lag endpoints, and DCS health — plus heartbeat monitors to confirm scheduled switchover tests and backup jobs completed. This tutorial covers the full HA monitoring stack.

Patroni's REST API

Patroni ships with a REST API (default port 8008) that returns structured cluster state. This is the primary monitoring surface:

# Cluster overview
curl http://patroni-node-1:8008/cluster

# Primary-specific endpoint — returns 200 only on the current primary
curl http://patroni-node-1:8008/primary

# Replica-specific endpoint — returns 200 only on replicas
curl http://patroni-node-1:8008/replica

# General health — returns 200 on any healthy node (primary or replica)
curl http://patroni-node-1:8008/health

These endpoints are the backbone of Patroni monitoring. Load balancers typically route traffic based on /primary and /replica status — the same endpoints Vigilmon can poll externally.

Step 1: Expose Patroni Health via a Proxy Endpoint

Patroni's REST API is usually bound to the private network interface. Proxy it through your application tier for external monitoring:

# FastAPI proxy for Patroni REST API
import httpx
from fastapi import FastAPI
from fastapi.responses import JSONResponse

app = FastAPI()

PATRONI_NODES = [
    "patroni-node-1:8008",
    "patroni-node-2:8008",
    "patroni-node-3:8008",
]

@app.get("/health/patroni")
async def patroni_health():
    primary = None
    replicas = []
    errors = {}

    async with httpx.AsyncClient(timeout=3.0) as client:
        for node in PATRONI_NODES:
            try:
                resp = await client.get(f"http://{node}/patroni")
                if resp.status_code == 200:
                    data = resp.json()
                    role = data.get("role", "unknown")
                    if role == "master":
                        primary = node
                    elif role in ("replica", "standby_leader"):
                        replicas.append(node)
            except Exception as e:
                errors[node] = str(e)

    if not primary:
        return JSONResponse(status_code=503, content={
            "status": "no_primary",
            "replicas": replicas,
            "errors": errors,
        })

    return JSONResponse(status_code=200, content={
        "status": "ok",
        "primary": primary,
        "replicas": replicas,
        "errors": errors,
    })

Replication Lag Endpoint

Patroni exposes lag via its API. The replication_state field on replica nodes includes the lag in bytes:

@app.get("/health/patroni/replication")
async def patroni_replication():
    lag_threshold_bytes = 10 * 1024 * 1024  # 10 MB
    lagging = []

    async with httpx.AsyncClient(timeout=3.0) as client:
        for node in PATRONI_NODES:
            try:
                resp = await client.get(f"http://{node}/patroni")
                if resp.status_code != 200:
                    continue
                data = resp.json()
                if data.get("role") in ("replica", "standby_leader"):
                    lag = data.get("replication_state", {})
                    # patroni reports lag in WAL bytes on newer versions
                    lag_bytes = lag.get("sent_lsn_diff", 0) if isinstance(lag, dict) else 0
                    if lag_bytes > lag_threshold_bytes:
                        lagging.append({"node": node, "lag_bytes": lag_bytes})
            except Exception:
                continue

    status = "ok" if not lagging else "degraded"
    code = 200 if not lagging else 503
    return JSONResponse(status_code=code, content={
        "status": status,
        "lagging_replicas": lagging,
        "threshold_bytes": lag_threshold_bytes,
    })

For a more precise lag measurement, query pg_stat_replication directly on the primary using the standard PostgreSQL approach:

SELECT
    application_name,
    state,
    pg_wal_lsn_diff(pg_current_wal_lsn(), replay_lsn) AS lag_bytes,
    EXTRACT(EPOCH FROM (now() - reply_time)) AS last_seen_seconds
FROM pg_stat_replication;

Step 2: DCS Connectivity Check (etcd / Consul)

Patroni uses a distributed consensus store (etcd or Consul) for leader elections. If the DCS becomes unreachable, Patroni demotes the primary to prevent split-brain — all writes stop. Monitoring DCS health separately from Patroni gives you advance warning.

etcd Health Check

ETCD_ENDPOINTS = [
    "http://etcd-1:2379",
    "http://etcd-2:2379",
    "http://etcd-3:2379",
]

@app.get("/health/etcd")
async def etcd_health():
    healthy = []
    unhealthy = []

    async with httpx.AsyncClient(timeout=2.0) as client:
        for endpoint in ETCD_ENDPOINTS:
            try:
                resp = await client.get(f"{endpoint}/health")
                if resp.status_code == 200 and resp.json().get("health") == "true":
                    healthy.append(endpoint)
                else:
                    unhealthy.append(endpoint)
            except Exception:
                unhealthy.append(endpoint)

    # etcd needs quorum (majority) to function
    quorum = len(ETCD_ENDPOINTS) // 2 + 1
    has_quorum = len(healthy) >= quorum

    status = "ok" if has_quorum else "no_quorum"
    code = 200 if has_quorum else 503
    return JSONResponse(status_code=code, content={
        "status": status,
        "healthy_nodes": healthy,
        "unhealthy_nodes": unhealthy,
        "quorum_required": quorum,
    })

Consul Health Check (alternative DCS)

@app.get("/health/consul")
async def consul_health():
    async with httpx.AsyncClient(timeout=2.0) as client:
        resp = await client.get("http://consul-server:8500/v1/health/state/critical")
    critical_services = resp.json()
    if critical_services:
        return JSONResponse(status_code=503, content={
            "status": "degraded",
            "critical": critical_services,
        })
    return JSONResponse(status_code=200, content={"status": "ok"})

Step 3: Switchover Event Detection

Patroni logs switchover events but doesn't push them. You can detect a switchover by comparing the current primary across polling intervals, or by watching Patroni's /history endpoint:

import json
from pathlib import Path

PRIMARY_STATE_FILE = "/tmp/patroni_primary.json"

@app.get("/health/patroni/switchover")
async def switchover_check():
    async with httpx.AsyncClient(timeout=3.0) as client:
        resp = await client.get(f"http://{PATRONI_NODES[0]}/cluster")
    cluster = resp.json()
    current_primary = next(
        (m["name"] for m in cluster.get("members", []) if m.get("role") == "leader"),
        None,
    )

    state_path = Path(PRIMARY_STATE_FILE)
    previous_primary = None
    if state_path.exists():
        previous_primary = json.loads(state_path.read_text()).get("primary")

    state_path.write_text(json.dumps({"primary": current_primary}))

    if previous_primary and current_primary != previous_primary:
        return JSONResponse(status_code=200, content={
            "status": "switchover_detected",
            "previous": previous_primary,
            "current": current_primary,
        })

    return JSONResponse(status_code=200, content={
        "status": "stable",
        "primary": current_primary,
    })

Vigilmon won't alert on a 200 — but you can log switchover events by watching for the switchover_detected body in Vigilmon's response history.

Step 4: Configure Vigilmon Monitors

Primary Availability Monitor

  1. Log in to vigilmon.online → Monitors → New Monitor → HTTP / HTTPS
  2. URL: https://your-api.example.com/health/patroni
  3. Check interval: 30 seconds (HA clusters warrant frequent checks)
  4. Expected status: 200; body contains: "status":"ok"
  5. Alert channel: Slack + PagerDuty (no primary = immediate writes failing)

Replication Lag Monitor

  1. Monitors → New Monitor → HTTP / HTTPS
  2. URL: https://your-api.example.com/health/patroni/replication
  3. Check interval: 1 minute
  4. Expected status: 200
  5. Alert channel: Slack (P2 — replica falling behind is a pre-failure signal)

DCS (etcd/Consul) Monitor

  1. Monitors → New Monitor → HTTP / HTTPS
  2. URL: https://your-api.example.com/health/etcd
  3. Check interval: 1 minute
  4. Expected status: 200; body contains: "status":"ok"
  5. Alert channel: Slack + PagerDuty (DCS degradation risks automatic demotion)

Step 5: Heartbeat for Scheduled Switchover Tests

Testing failover regularly is the only way to know Patroni will actually work when you need it. Monitor your switchover drills with a Vigilmon heartbeat:

#!/bin/bash
# patroni-switchover-test.sh
set -euo pipefail

VIGILMON_HEARTBEAT="https://vigilmon.online/heartbeat/your-heartbeat-id"

echo "Running scheduled Patroni switchover test..."
patronictl -c /etc/patroni/patroni.yml switchover --master "$(patronictl -c /etc/patroni/patroni.yml list -f json | jq -r '.[] | select(.Role=="Leader") | .Member')" --candidate "$(patronictl -c /etc/patroni/patroni.yml list -f json | jq -r '.[] | select(.Role=="Replica") | .Member' | head -1)" --force

# Wait for new primary to come up
sleep 10
NEW_PRIMARY=$(patronictl -c /etc/patroni/patroni.yml list -f json | jq -r '.[] | select(.Role=="Leader") | .Member')
echo "New primary: $NEW_PRIMARY"

# Ping heartbeat only if switchover succeeded
[ -n "$NEW_PRIMARY" ] && curl -fsS "$VIGILMON_HEARTBEAT" > /dev/null

Set heartbeat interval to 1 week with a 2-day grace period.

Alert Routing Summary

| Monitor | Severity | Channel | |---|---|---| | Patroni primary availability | P1 | Slack + PagerDuty | | Replication lag | P2 | Slack | | etcd/Consul DCS health | P1 | Slack + PagerDuty | | Weekly switchover test | P2 | Email + Slack |

Summary

Patroni monitoring spans three layers: the Patroni REST API, the underlying PostgreSQL replication, and the distributed consensus store. Vigilmon covers all three externally:

| Layer | Vigilmon Monitor | |---|---| | Primary election status | HTTP monitor on /health/patroni | | Replica replication lag | HTTP monitor on /health/patroni/replication | | etcd/Consul quorum | HTTP monitor on /health/etcd | | Failover drill heartbeat | Heartbeat monitor |

Start monitoring your Patroni cluster for free at vigilmon.online.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles