tutorial

How to Monitor etcd with Vigilmon

etcd cluster failures take down the entire Kubernetes control plane. Learn how to monitor etcd cluster health, leader election stability, Raft proposal failure rates, compaction lag, and disk I/O saturation with Vigilmon HTTP probes and heartbeat monitoring.

etcd is the brain of Kubernetes — it stores every cluster state object, every config map, every secret, every Pod spec. When etcd degrades, the Kubernetes API server starts returning 500 errors, controllers stop reconciling, and schedulers freeze. New Pods don't get created, failing Pods don't get restarted, and deployments stall silently. The cluster appears to be running while it's actually in a split-brain holding pattern.

Vigilmon gives you external visibility into etcd cluster health — leader election stability, Raft proposal failures, compaction lag, and disk I/O saturation — so you know about etcd degradation before the Kubernetes control plane becomes the problem your users report.

Why etcd Needs External Monitoring

etcd exposes rich internal metrics via its Prometheus endpoint (/metrics), but these require an active scraper and are only accessible from within the cluster network. External monitoring with Vigilmon adds:

  • Cluster health alerting from outside the cluster network — catches network partitions that isolate your monitoring stack
  • Leader election instability detection — frequent leader changes (more than one per hour in a healthy cluster) indicate disk I/O or network problems that precede split-brain
  • Raft proposal failure rate monitoring — failed proposals mean writes to etcd are being rejected; this is the leading indicator of Kubernetes API server degradation
  • Compaction lag alerting — etcd accumulates MVCC revisions that must be compacted periodically; a stalled compaction causes the database to grow unbounded until etcd runs out of disk space
  • Disk I/O saturation — etcd's fsync latency is extremely sensitive; high disk latency directly causes Raft timeout failures

Step 1: Use etcd's Built-In Health Endpoints

etcd exposes HTTP health endpoints on the client port (default 2379):

# Cluster health — returns 200 if the member is healthy
curl -i http://localhost:2379/health

# Readiness — returns 200 only when the member can serve client requests
curl -i http://localhost:2379/readyz

# Liveness
curl -i http://localhost:2379/livez

# Prometheus metrics — requires access from within the cluster
curl -s http://localhost:2379/metrics | grep -E 'etcd_server_has_leader|etcd_server_leader_changes|etcd_mvcc_db_total_size'

Example responses:

// GET /health — healthy
{"health":"true","reason":""}

// GET /health — unhealthy (quorum loss)
{"health":"false","reason":"NOSPACE"}

// GET /readyz — unhealthy (leader election in progress)
HTTP/1.1 503 Service Unavailable

Step 2: Build an etcd Health Sidecar

etcd's /health endpoint only tells you if the member is reachable. To catch leader instability, proposal failures, and disk pressure before they become outages, build a metrics-based health sidecar.

# etcd_health.py
import os
import time
import requests
from flask import Flask, jsonify

app = Flask(__name__)

ETCD_ENDPOINT   = os.environ.get('ETCD_ENDPOINT', 'http://localhost:2379')
ETCD_METRICS    = os.environ.get('ETCD_METRICS_URL', 'http://localhost:2379/metrics')
# Alert thresholds
MAX_LEADER_CHANGES_PER_HOUR   = int(os.environ.get('MAX_LEADER_CHANGES_PER_HOUR', '2'))
MAX_PROPOSAL_FAILURES         = int(os.environ.get('MAX_PROPOSAL_FAILURES', '10'))
MAX_DB_SIZE_BYTES             = int(os.environ.get('MAX_DB_SIZE_BYTES', str(6 * 1024 * 1024 * 1024)))  # 6GB default
MAX_DISK_FSYNC_MS             = float(os.environ.get('MAX_DISK_FSYNC_MS', '100'))

def parse_prometheus_gauge(text, metric_name):
    for line in text.splitlines():
        if line.startswith(metric_name + ' ') or line.startswith(metric_name + '{'):
            if not line.startswith('#'):
                try:
                    return float(line.rsplit(' ', 1)[-1])
                except ValueError:
                    pass
    return None

@app.route('/health/etcd')
def health():
    issues = []

    # 1. Basic member health
    try:
        health_resp = requests.get(f'{ETCD_ENDPOINT}/health', timeout=5)
        health_data = health_resp.json()
        if health_data.get('health') != 'true':
            return jsonify({
                'status': 'down',
                'reason': 'member_unhealthy',
                'detail': health_data.get('reason', 'unknown'),
            }), 503
    except Exception as e:
        return jsonify({'status': 'down', 'error': str(e)}), 503

    # 2. Prometheus metrics checks
    try:
        metrics_resp = requests.get(ETCD_METRICS, timeout=5)
        if metrics_resp.status_code != 200:
            issues.append({'type': 'metrics_unavailable'})
        else:
            m = metrics_resp.text

            # Check if this member has a leader
            has_leader = parse_prometheus_gauge(m, 'etcd_server_has_leader')
            if has_leader is not None and has_leader < 1:
                return jsonify({'status': 'down', 'reason': 'no_leader'}), 503

            # Leader change rate (total changes; compare across calls for rate)
            leader_changes = parse_prometheus_gauge(m, 'etcd_server_leader_changes_seen_total')
            if leader_changes is not None and leader_changes > MAX_LEADER_CHANGES_PER_HOUR:
                issues.append({
                    'type': 'leader_instability',
                    'leader_changes_total': leader_changes,
                    'threshold': MAX_LEADER_CHANGES_PER_HOUR,
                })

            # Raft proposal failures
            proposal_failures = parse_prometheus_gauge(m, 'etcd_server_proposals_failed_total')
            if proposal_failures is not None and proposal_failures > MAX_PROPOSAL_FAILURES:
                issues.append({
                    'type': 'proposal_failures',
                    'failures_total': proposal_failures,
                    'threshold': MAX_PROPOSAL_FAILURES,
                })

            # Database size (approaching quota = imminent NOSPACE)
            db_size = parse_prometheus_gauge(m, 'etcd_mvcc_db_total_size_in_bytes')
            if db_size is not None and db_size > MAX_DB_SIZE_BYTES:
                issues.append({
                    'type': 'db_size_critical',
                    'db_size_bytes': db_size,
                    'threshold_bytes': MAX_DB_SIZE_BYTES,
                })

            # Disk fsync latency (p99) — high fsync = Raft timeouts coming
            fsync_p99 = parse_prometheus_gauge(m, 'etcd_disk_wal_fsync_duration_seconds{quantile="0.99"}')
            if fsync_p99 is not None:
                fsync_ms = fsync_p99 * 1000
                if fsync_ms > MAX_DISK_FSYNC_MS:
                    issues.append({
                        'type': 'disk_fsync_slow',
                        'fsync_p99_ms': round(fsync_ms, 2),
                        'threshold_ms': MAX_DISK_FSYNC_MS,
                    })

    except Exception as e:
        issues.append({'type': 'metrics_error', 'error': str(e)})

    if issues:
        return jsonify({'status': 'degraded', 'issues': issues}), 503

    return jsonify({'status': 'ok', 'member_healthy': True}), 200

if __name__ == '__main__':
    app.run(port=8093)

Deploy one sidecar per etcd member so Vigilmon can independently check each member's health.

Step 3: Configure Vigilmon Monitors

Monitor 1: etcd Member Health

  1. Log in to vigilmon.online → Monitors → New Monitor
  2. Choose HTTP / HTTPS
  3. URL: https://your-etcd-lb.example.com/health
  4. Check interval: 30 seconds (etcd failures cascade fast)
  5. Expected:
    • Status code: 200
    • Response body contains: "health":"true"
    • Response time threshold: 3000ms
  6. Alert channel: Slack + PagerDuty (etcd down = Kubernetes control plane down)
  7. Save

Repeat for each etcd member if you have direct access to the member endpoints. A 3-member cluster should have 3 monitors.

Monitor 2: Readiness Probe

Add a monitor for the /readyz endpoint which is more strict — it returns 503 during leader elections:

  • URL: https://your-etcd-member.example.com:2379/readyz
  • Expected: 200
  • Interval: 30 seconds
  • Alert channel: Slack (P1 — readyz failure = this member can't serve requests)

Monitor 3: Comprehensive Metrics Health

Add a monitor for the health sidecar that catches degraded states:

  • URL: https://your-etcd-sidecar.example.com:8093/health/etcd
  • Expected: 200, body contains "status":"ok"
  • Interval: 1 minute
  • Alert channel: Slack (P2 — catches warning signs before P1)

Step 4: Monitor Kubernetes API Server Dependency

Since etcd is a Kubernetes control plane dependency, also monitor the Kubernetes API server health as a downstream indicator of etcd degradation:

# Kubernetes API server health endpoint
curl -k https://your-k8s-api:6443/healthz
curl -k https://your-k8s-api:6443/readyz

# etcd-specific liveness sub-check
curl -k https://your-k8s-api:6443/livez/etcd

Configure Vigilmon to probe https://your-k8s-api:6443/livez/etcd directly — this endpoint returns non-200 when the API server loses its etcd connection, giving you a correlated alert that confirms etcd impact.

etcd Compaction Watchdog

etcd compaction keeps the database size manageable. If compaction stalls, the database grows until NOSPACE. Add a heartbeat monitor that your compaction cron job pings after each successful compaction:

  1. In Vigilmon → Monitors → New Monitor → Heartbeat
  2. Name: etcd-compaction-job
  3. Expected interval: 24 hours (or your compaction schedule)
  4. Grace period: 2 hours
  5. Save → copy heartbeat URL

Wire it into your compaction script:

#!/bin/bash
# etcd_compact.sh
ETCD_ENDPOINT=${ETCD_ENDPOINT:-http://localhost:2379}
VIGILMON_HEARTBEAT_URL=${VIGILMON_HEARTBEAT_URL}

# Get current revision
REVISION=$(etcdctl --endpoints=$ETCD_ENDPOINT endpoint status --write-out=json \
  | jq -r '.[0].Status.header.revision')

# Compact to current revision - 1000 (keep recent history)
COMPACT_REV=$((REVISION - 1000))
etcdctl --endpoints=$ETCD_ENDPOINT compact $COMPACT_REV

# Defragment all members
etcdctl --endpoints=$ETCD_ENDPOINT defrag --cluster

# Ping Vigilmon if compaction succeeded
if [ $? -eq 0 ] && [ -n "$VIGILMON_HEARTBEAT_URL" ]; then
    curl -s "$VIGILMON_HEARTBEAT_URL" > /dev/null
fi

Step 5: Alert Routing

| Monitor | Alert Channel | Priority | |---|---|---| | etcd /health (each member) | Slack + PagerDuty | P1 | | etcd /readyz (each member) | Slack + PagerDuty | P1 | | Kubernetes API /livez/etcd | Slack + PagerDuty | P1 | | Health sidecar (metrics-based) | Slack | P2 | | Compaction heartbeat | Email | P2 |

Set response time thresholds on the /health endpoint — etcd member response times above 500ms indicate disk I/O pressure that precedes Raft timeout failures. Alert at 500ms and page at 2 seconds.

Summary

etcd failures are catastrophic for Kubernetes clusters. You need monitoring at multiple layers: member health, leader stability, Raft proposal health, disk pressure, and compaction liveness.

| Monitor Type | What It Covers | |---|---| | HTTP monitor on /health | Member reachability, quorum health | | HTTP monitor on /readyz | Readiness during leader elections | | HTTP monitor on metrics sidecar | Leader stability, proposal failures, disk I/O, DB size | | HTTP monitor on K8s API /livez/etcd | Control plane impact of etcd degradation | | Heartbeat monitor on compaction | MVCC compaction schedule liveness |

Get started free at vigilmon.online — your first etcd monitor is running in under two minutes.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles