tutorial

Monitoring Kafka Connect with Vigilmon: Connector Health, Consumer Lag, DLQ Growth & Data Pipeline Uptime

How to monitor Kafka Connect with Vigilmon — connector task health, consumer group lag, worker restart rates, DLQ growth, offset commit failures, and external uptime checks for data integration pipelines.

Kafka Connect is the backbone of data integration pipelines — moving data between databases, object stores, search indices, and streaming platforms at scale. When a connector task fails or a worker goes down, the pipeline stalls silently: events accumulate in source topics, downstream consumers fall behind, and data consistency drifts. Vigilmon gives you external uptime monitoring for your Kafka Connect REST API plus keyword-based alerting on connector states, so you know about task failures before your data warehouse notices the missing rows.

What You'll Build

  • A monitor on Kafka Connect's REST API for cluster-level liveness
  • Per-connector health checks with status keyword alerting
  • DLQ (Dead Letter Queue) growth detection
  • Worker availability monitoring via the workers list endpoint
  • SSL certificate monitoring for your Connect REST API

Prerequisites

  • A running Kafka Connect cluster with the REST API enabled (default port 8083)
  • An accessible endpoint (e.g., https://connect.example.com)
  • At least one configured connector
  • A free account at vigilmon.online

Step 1: Verify the Kafka Connect REST API

Kafka Connect exposes a cluster info endpoint at the root path:

curl https://connect.example.com/

A healthy Connect cluster returns:

{
  "version": "3.6.0",
  "commit": "60e845626d8a465d",
  "kafka_cluster_id": "abc123XYZ"
}

This endpoint confirms the Connect REST API is reachable and the cluster has a valid Kafka broker connection (the kafka_cluster_id is populated only when the cluster communicates successfully with Kafka).

List all registered connectors to verify configuration is intact:

curl https://connect.example.com/connectors

Returns: ["postgres-source", "s3-sink", "elasticsearch-sink"]

Step 2: Create a Vigilmon Cluster Liveness Monitor

  1. Log in to Vigilmon → Add Monitor → HTTP.
  2. URL: https://connect.example.com/.
  3. Check interval: 60 seconds.
  4. Response timeout: 10 seconds.
  5. Expected status: 200.
  6. Keyword: "kafka_cluster_id" (confirms Kafka broker connectivity, not just process liveness).
  7. Click Save.

This distinguishes between two failure modes: the Connect process is down (no response at all), and the Connect process is up but disconnected from Kafka (responds but without kafka_cluster_id).

Step 3: Monitor Connector Task Status

Connector tasks are the workers that actually read from sources and write to sinks. A task can fail while the connector itself appears registered:

curl https://connect.example.com/connectors/postgres-source/status

Returns:

{
  "name": "postgres-source",
  "connector": { "state": "RUNNING", "worker_id": "10.0.0.1:8083" },
  "tasks": [
    { "id": 0, "state": "RUNNING", "worker_id": "10.0.0.1:8083" },
    { "id": 1, "state": "FAILED", "worker_id": "10.0.0.2:8083",
      "trace": "org.apache.kafka.connect.errors.ConnectException: ..." }
  ]
}

Task states:

  • RUNNING: Task is active and processing records
  • PAUSED: Task was deliberately paused (operator action)
  • FAILED: Task crashed; must be restarted manually or via automatic restart policy
  • UNASSIGNED: Task not yet assigned to a worker (transient during startup)

Add a Vigilmon monitor per critical connector:

  1. Add Monitor → HTTP.
  2. URL: https://connect.example.com/connectors/postgres-source/status.
  3. Check interval: 2 minutes.
  4. Expected status: 200.
  5. Keyword: "FAILED" — configure Vigilmon to alert when this keyword is present (inverted check).
  6. Label: Kafka Connect task: postgres-source.

Tip: For connectors with many tasks, you can also check the overall connector state rather than individual tasks: monitor for "state":"FAILED" in the connector object.

Step 4: Monitor All Connectors in Bulk

For clusters with many connectors, use the expanded status endpoint to monitor all connectors in a single check:

curl "https://connect.example.com/connectors?expand=status"

This returns all connectors and their task states in one response. Add a monitor that alerts if any connector enters a FAILED state:

  1. Add Monitor → HTTP.
  2. URL: https://connect.example.com/connectors?expand=status.
  3. Check interval: 2 minutes.
  4. Expected status: 200.
  5. Keyword: "FAILED" — alert when present.
  6. Label: Kafka Connect: any connector FAILED.

This gives you a cluster-wide FAILED alert without needing individual monitors per connector.

Step 5: Monitor Worker Availability

Workers are the JVM processes that host connector tasks. Monitor the worker list to catch worker departures that reduce cluster capacity:

curl https://connect.example.com/workers

Returns the list of active workers. Add a monitor:

  1. Add Monitor → HTTP.
  2. URL: https://connect.example.com/workers.
  3. Check interval: 2 minutes.
  4. Expected status: 200.
  5. Keyword: "10.0.0.1" (or any expected worker IP that should always be present).
  6. Label: Kafka Connect worker availability.

Worker loss reduces task parallelism and triggers task rebalancing, which causes brief pipeline stalls while tasks are reassigned.

Step 6: Monitor Dead Letter Queue (DLQ) Growth

When connectors are configured with DLQ error tolerance (errors.tolerance=all), failed records are routed to a DLQ topic instead of stopping the task. An unchecked DLQ means data is being silently dropped.

Monitor DLQ topics via your Kafka broker's consumer group offset lag, or add a simple check on a custom HTTP endpoint if you expose DLQ metrics. Alternatively, monitor the connector error count via the JMX-to-REST bridge if you run Kafka Connect with a metrics reporter:

# If using Confluent Platform with the REST proxy metrics:
curl https://connect.example.com/connectors/postgres-source/status

Look for "error_count" in the task response — many connectors expose this in their task status. Add a monitor:

  1. Add Monitor → HTTP.
  2. URL: https://connect.example.com/connectors/postgres-source/status.
  3. Check interval: 5 minutes.
  4. Keyword: "error_count":0" — alert when absent (indicates non-zero errors accumulating).
  5. Label: Kafka Connect DLQ sentinel: postgres-source.

Step 7: Monitor Offset Commit Health

Offset commit failures cause connectors to reprocess records after restarts, leading to duplicate data in sinks. Watch for offset-related errors in the task trace:

Add a monitor that checks connector task status and alerts on offset-related error keywords:

  1. Add Monitor → HTTP.
  2. URL: https://connect.example.com/connectors/postgres-source/status.
  3. Check interval: 5 minutes.
  4. Keyword: "OffsetStorageWriter" — this class name appears in stack traces for offset commit failures.
  5. Label: Kafka Connect offset commit: postgres-source.

Step 8: SSL Certificate Monitoring

  1. Add Monitor → SSL Certificate.
  2. Domain: connect.example.com.
  3. Alert when expiry is within: 30 days.
  4. Click Save.

Kafka Connect REST API SSL expiry is frequently overlooked in comparison to application certificates. Many data pipelines (Debezium CDC, JDBC connectors, S3 sinks) use the REST API for health checks and configuration updates — a certificate expiry breaks all external integrations silently.

Step 9: Configure Alert Routing

In Vigilmon under Settings → Notifications:

| Monitor | Trigger | Action | |---|---|---| | Cluster liveness | No 200 or kafka_cluster_id absent | All pipelines down; page immediately | | Any connector FAILED | FAILED present in bulk status | Identify failed connector; restart task | | Worker availability | Expected worker IP absent | Worker lost; check JVM logs and rebalancing | | DLQ sentinel | Error count non-zero | Review DLQ topic; fix source data issues | | Offset commit | OffsetStorageWriter in trace | Risk of duplicate data on restart | | SSL certificate | < 30 days to expiry | Renew certificate |

Alert after: 1 failure for cluster liveness. 2 failures for connector task status (transient task restarts during lag spikes are normal).

Common Kafka Connect Failure Modes and What Vigilmon Catches

| Scenario | Vigilmon monitor | |---|---| | Connect worker JVM OOM | Cluster liveness monitor; alert in < 60 s | | Connector task crashes (source error) | Task status FAILED keyword present | | All workers leave cluster | Worker list empties; liveness monitor fires | | DLQ silently filling with bad records | DLQ sentinel monitor fires | | Kafka broker connectivity lost | kafka_cluster_id absent in root response | | Task restart loop (repeated failures) | FAILED keyword appears repeatedly in monitor history | | SSL certificate expires | SSL monitor alerts at 30-day threshold | | Connector configuration corrupted | 500 response from status endpoint |

Kafka Connect task failures are one of the most common causes of silent data loss in production — tasks fail, the pipeline stalls, and engineers discover the gap hours later when downstream dashboards show missing data. Vigilmon's external monitoring of the Connect REST API and connector status endpoints catches these failures within minutes, long before data consumers notice the gap.

Start monitoring Kafka Connect in under 5 minutes — register free at vigilmon.online.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles