tutorial

Monitoring Debezium: CDC Pipeline Health, Connector Lag, and Vigilmon Integration

"A practical guide to monitoring Debezium change data capture pipelines — connector health, snapshot progress, Kafka producer metrics, offset commit failures, and how Vigilmon keeps your CDC pipeline observable end-to-end."

Monitoring Debezium: CDC Pipeline Health, Connector Lag, and Vigilmon Integration

Change data capture (CDC) with Debezium is increasingly mission-critical — feeding data warehouses, powering event-driven microservices, and keeping search indexes in sync. When a Debezium connector stalls, tables diverge silently and downstream consumers start serving stale data. This guide covers what to monitor in a production Debezium deployment and how to wire up Vigilmon to alert you before users notice.

What Can Go Wrong in a Debezium Pipeline

Before diving into metrics, it helps to understand failure modes:

  • Connector task failure: a task exits with an error and Debezium stops producing events without crashing the Kafka Connect worker.
  • Snapshot stall: the initial snapshot of a large table is paused or killed, leaving the connector in an indeterminate state.
  • Replication lag: the connector falls behind the binlog/WAL, growing the lag between the source database and downstream consumers.
  • Offset commit failures: Debezium can't persist its position; a restart replays events, causing duplicates downstream.
  • Kafka producer backpressure: the Connect worker's producer saturates its buffer and begins blocking or dropping records.

Connector and Task Health via the Kafka Connect REST API

The Kafka Connect REST API is your primary health interface:

# Check all connectors
curl http://localhost:8083/connectors

# Check status of a specific connector
curl http://localhost:8083/connectors/inventory-connector/status

A healthy response looks like:

{
  "name": "inventory-connector",
  "connector": { "state": "RUNNING", "worker_id": "connect:8083" },
  "tasks": [
    { "id": 0, "state": "RUNNING", "worker_id": "connect:8083" }
  ]
}

Any state other than RUNNING (e.g. FAILED, PAUSED, UNASSIGNED) needs an alert. Script this check and expose it as a synthetic endpoint:

#!/bin/bash
STATUS=$(curl -sf http://localhost:8083/connectors/inventory-connector/status \
  | jq -r '.tasks[0].state')

if [ "$STATUS" != "RUNNING" ]; then
  echo "CRITICAL: connector task state=$STATUS"
  exit 1
fi
echo "OK: connector running"

Debezium JMX Metrics

Debezium exposes JMX metrics under debezium.mysql:type=connector-metrics. Key metrics to scrape (via Prometheus JMX Exporter or similar):

# prometheus-jmx-exporter config snippet
rules:
  - pattern: 'debezium.mysql<type=connector-metrics, context=snapshot, server=(.+)><>SnapshotCompleted'
    name: debezium_snapshot_completed
    type: GAUGE

  - pattern: 'debezium.mysql<type=connector-metrics, context=streaming, server=(.+)><>MilliSecondsBehindSource'
    name: debezium_lag_ms
    type: GAUGE

  - pattern: 'debezium.mysql<type=connector-metrics, context=streaming, server=(.+)><>NumberOfCommittedTransactions'
    name: debezium_committed_transactions_total
    type: COUNTER

  - pattern: 'debezium.mysql<type=connector-metrics, context=streaming, server=(.+)><>NumberOfErrorsSeen'
    name: debezium_errors_total
    type: COUNTER

The critical streaming metric is MilliSecondsBehindSource — this is your replication lag. Alert when it exceeds your SLA (e.g. > 30 seconds).

Offset Commit Health

Debezium stores offsets in a Kafka topic (default: connect-offsets). You can inspect how fresh the committed offset is:

# Show consumer group lag for the internal offset topic
kafka-consumer-groups.sh \
  --bootstrap-server localhost:9092 \
  --describe \
  --group connect-cluster

Watch for CONSUMER-LAG growing on the __consumer_offsets and connect-offsets topics. If offsets aren't being committed, Debezium will re-read the binlog from the last committed position on restart — potentially producing millions of duplicate events.

Kafka Producer Health

The Kafka Connect worker exposes producer metrics via JMX at kafka.producer:type=producer-metrics:

# Key metrics to watch
kafka.producer:type=producer-metrics,client-id=connector-producer-inventory-connector-0
  record-send-rate         # records per second being sent
  record-error-rate        # errors per second — should be 0
  request-latency-avg      # average produce request latency
  buffer-available-bytes   # drop to 0 means producer is blocked

buffer-available-bytes reaching zero is a hard warning — the producer is backpressured and the connector will stall.

Snapshot Progress

During initial snapshot, monitor SnapshotCompleted (0 = in progress, 1 = done) and RemainingTableCount. A snapshot stuck at the same RemainingTableCount for more than your expected snapshot duration is a problem:

import requests, time

def check_snapshot_progress(connector="inventory-connector"):
    url = f"http://localhost:8083/connectors/{connector}/status"
    r = requests.get(url, timeout=5)
    state = r.json()["connector"]["state"]
    return state  # RUNNING during snapshot is expected; FAILED is not

Vigilmon Integration

Vigilmon's external HTTP check makes it easy to monitor the Kafka Connect health endpoint and your custom health scripts from outside the cluster.

Step 1 — expose a health endpoint

Wrap the connector status check in a minimal HTTP service:

from flask import Flask, jsonify
import requests

app = Flask(__name__)

CONNECTORS = ["inventory-connector", "orders-connector"]

@app.route("/health/debezium")
def health():
    results = {}
    all_ok = True
    for name in CONNECTORS:
        try:
            r = requests.get(
                f"http://localhost:8083/connectors/{name}/status",
                timeout=3
            )
            state = r.json()["tasks"][0]["state"]
            results[name] = state
            if state != "RUNNING":
                all_ok = False
        except Exception as e:
            results[name] = f"ERROR: {e}"
            all_ok = False

    code = 200 if all_ok else 503
    return jsonify({"connectors": results, "healthy": all_ok}), code

if __name__ == "__main__":
    app.run(host="0.0.0.0", port=9090)

Step 2 — add an HTTP monitor in Vigilmon

In your Vigilmon dashboard:

  1. Create a new HTTP monitor pointing to https://your-host:9090/health/debezium.
  2. Set check interval to 60 seconds.
  3. Set expected status code to 200.
  4. Configure an alert escalation policy with a PagerDuty or Slack integration.

Step 3 — add a lag monitor

For replication lag, expose the JMX metric as a Prometheus endpoint and create a separate Vigilmon check that fails when lag exceeds threshold:

@app.route("/health/debezium/lag")
def lag():
    # Read from your Prometheus endpoint or JMX directly
    lag_ms = get_debezium_lag_ms()  # implement this for your setup
    if lag_ms > 30_000:  # 30 seconds
        return jsonify({"lag_ms": lag_ms, "status": "lagging"}), 503
    return jsonify({"lag_ms": lag_ms, "status": "ok"}), 200

Add this as a second Vigilmon HTTP monitor with a tighter alert threshold.

Alert Thresholds to Set

| Metric | Warning | Critical | |--------|---------|----------| | Connector task state | Any non-RUNNING | Any FAILED | | MilliSecondsBehindSource | > 10s | > 60s | | record-error-rate | > 0 | > 5/s | | buffer-available-bytes | < 20% | < 5% | | Snapshot duration | > expected | stalled 2x expected |

Conclusion

Debezium failures are silent by default — a stalled connector doesn't crash, it just stops delivering events. The combination of the Kafka Connect REST API for task health, JMX metrics for lag and producer health, and Vigilmon HTTP monitors for external verification gives you a complete observability layer. Set up the health endpoint, add the Vigilmon checks, and you'll catch CDC pipeline problems before they cascade into data inconsistencies downstream.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles