tutorial

Monitoring Karafka with Vigilmon: Kafka Broker Connectivity, Web UI Uptime & Consumer Group Health

How to monitor Karafka Kafka framework for Ruby/Rails with Vigilmon — Kafka broker TCP checks, Karafka Web UI uptime, consumer group health endpoints, and SSL certificate monitoring.

Karafka is the Ruby and Rails framework for building Kafka-based event-driven applications — providing a clean routing DSL, consumer group management, and the Karafka Web UI for inspecting consumer lag, processing throughput, and error rates. When the Kafka broker that Karafka connects to becomes unreachable, consumers stop processing messages, producers start accumulating backlogs, and the business events that drive your application — order placements, payment confirmations, inventory updates — pile up unprocessed. When the Karafka Web UI is down, teams lose their primary visibility into consumer lag and processing health. Vigilmon gives you external monitoring for Karafka's infrastructure dependencies: Kafka broker connectivity, the Web UI, and SSL certificate health.

What You'll Build

  • A TCP monitor on the Kafka broker port to detect broker connectivity failures before consumer lag accumulates
  • An HTTP monitor on the Karafka Web UI to catch dashboard availability failures
  • An SSL certificate monitor for your Kafka broker and Web UI domains
  • Alerting that distinguishes broker-level failures from Web UI availability issues

Prerequisites

  • A Karafka application running with access to a Kafka broker (self-hosted or managed)
  • The Kafka broker accessible on the network with a reachable hostname/IP and port
  • The Karafka Web UI deployed and accessible via HTTPS
  • A free account at vigilmon.online

Step 1: Understand Karafka's Infrastructure Dependencies

Karafka applications depend on two external components that require monitoring:

| Component | Default port | Role | |---|---|---| | Kafka broker | 9092 (plaintext), 9093 (TLS) | Message broker that stores topics and serves consumer groups | | Karafka Web UI | 9999 (default dev), via HTTPS in production | Dashboard for consumer group lag, throughput, and error inspection | | Schema Registry (optional) | 8081 | Validates Avro/JSON schemas if Karafka is configured with schema registry |

External monitoring focuses on the Kafka broker port (the most critical dependency — without it, no messages are consumed or produced) and the Karafka Web UI (the primary operations dashboard for consumer group health).

Step 2: Monitor Kafka Broker Connectivity via TCP Check

Karafka consumers and producers connect to Kafka brokers via TCP. A TCP check on the broker port confirms that the broker is reachable at the network level — the fundamental requirement for any Kafka-dependent service:

# Test Kafka broker TCP connectivity
nc -zv kafka.example.com 9092
# or for TLS-enabled brokers:
nc -zv kafka.example.com 9093
  1. Log in to Vigilmon → Add Monitor → TCP.
  2. Host: kafka.example.com (your Kafka broker hostname or IP).
  3. Port: 9092 (plaintext) or 9093 (TLS-enabled).
  4. Check interval: 60 seconds.
  5. Label: Kafka broker TCP.
  6. Click Save.

This monitor catches:

  • Kafka broker process crashes or OOM kills
  • Network partition between your Rails/Ruby application and the Kafka cluster
  • Firewall rule changes blocking Kafka port access
  • Kafka cluster rolling restarts that take a broker offline longer than expected
  • Cloud provider networking incidents affecting Kafka broker reachability

Alert sensitivity: Set to trigger after 1 consecutive failure. When the Kafka broker is unreachable, Karafka consumers immediately stop processing messages and producers start buffering locally until their buffer limits are exceeded.

Multiple brokers: If your Kafka cluster has multiple brokers, add a TCP monitor for each broker address in your kafka.seed_brokers configuration. Karafka can route around a single failed broker, but monitoring each broker individually catches broker-specific failures before they degrade cluster performance.

Step 3: Monitor the Karafka Web UI

The Karafka Web UI (karafka-web gem) provides a browser dashboard showing consumer group lag, topic throughput, processing errors, and worker health. It is typically deployed as a Rack application mounted inside your Rails app or as a standalone service:

curl https://karafka.example.com
# Returns HTML with the Karafka Web UI dashboard
  1. Add Monitor → HTTP.
  2. URL: https://karafka.example.com (or the path where the Web UI is mounted, e.g., https://app.example.com/karafka).
  3. Check interval: 60 seconds.
  4. Response timeout: 15 seconds.
  5. Expected status: 200.
  6. Keyword: Karafka (appears in the Web UI page title and content).
  7. Label: Karafka Web UI.
  8. Click Save.

Mounted Web UI: If the Karafka Web UI is mounted inside your Rails application at a path (e.g., /karafka), use the full URL including the mount path: https://app.example.com/karafka. The keyword Karafka will still match the page content regardless of mount path.

This monitor catches:

  • Karafka Web UI process failures
  • Rails application failures that take down the mounted Web UI
  • Authentication middleware issues blocking access to the Web UI
  • Deployment failures that leave the Web UI in a broken state

Step 4: Monitor the Karafka Web UI API Endpoint

The Karafka Web UI exposes internal API endpoints that power the dashboard. Monitoring the API endpoint confirms that the Web UI can communicate with its backing data store (Kafka topics used for internal reporting):

curl https://karafka.example.com/api/consumers
# Returns JSON with consumer group status data
  1. Add Monitor → HTTP.
  2. URL: https://karafka.example.com/api/consumers (adjust path for your mount point).
  3. Check interval: 2 minutes.
  4. Response timeout: 10 seconds.
  5. Expected status: 200.
  6. Keyword: consumers (present in the API response JSON).
  7. Label: Karafka Web UI API.
  8. Click Save.

The exact API path depends on your Karafka Web UI version. If /api/consumers is not available, use /api/health or check the Karafka Web UI documentation for health/status endpoints in your deployed version.

Step 5: Monitor Schema Registry (If Configured)

If your Karafka application uses Avro or JSON Schema validation via a Schema Registry (Confluent Schema Registry or AWS Glue Schema Registry), schema registry availability is a production dependency — producers cannot serialize messages if the registry is unreachable:

curl https://schema-registry.example.com/subjects
# Returns JSON list of registered schemas
  1. Add Monitor → HTTP.
  2. URL: https://schema-registry.example.com/subjects.
  3. Check interval: 2 minutes.
  4. Response timeout: 10 seconds.
  5. Expected status: 200.
  6. Keyword: [ (the response is a JSON array).
  7. Label: Schema Registry availability.
  8. Click Save.

Step 6: Monitor SSL Certificates

Kafka brokers configured with TLS (port 9093) and the Karafka Web UI both require valid SSL certificates. Expired certificates cause Karafka consumers and producers to fail with TLS errors and block browser access to the Web UI:

# Check Kafka broker TLS certificate
openssl s_client -connect kafka.example.com:9093 2>/dev/null | openssl x509 -noout -dates

# Check Karafka Web UI certificate
openssl s_client -connect karafka.example.com:443 2>/dev/null | openssl x509 -noout -dates
  1. Add Monitor → SSL Certificate.
  2. Domain: kafka.example.com (for the Kafka broker TLS cert).
  3. Alert when expiry is within: 30 days.
  4. Click Save.
  5. Repeat for karafka.example.com (Web UI domain).

Kafka mTLS: If your Kafka cluster uses mutual TLS (mTLS) where clients present certificates, client certificate expiry is also a failure mode. Karafka's ssl_client_cert and ssl_client_cert_key configuration will fail when the client cert expires — rotate those via your certificate management toolchain before Vigilmon's broker TCP monitor starts catching TLS-related connection failures.

Step 7: Configure Alerting

In Vigilmon under Settings → Notifications, configure your alert channels:

| Monitor | Trigger | Action | |---|---|---| | Kafka broker TCP | Connection refused | Kafka broker down; check broker logs; inspect cluster health | | Karafka Web UI HTTP | Non-200 or Karafka missing | Web UI process failed; check Rails app or standalone Web UI service | | Karafka Web UI API | Non-200 or keyword missing | Web UI backend degraded; check internal Kafka topic connectivity | | Schema Registry | Non-200 or [ missing | Registry unreachable; producers will fail to serialize; check registry service | | Kafka broker SSL | < 30 days to expiry | Renew Kafka broker TLS cert; test Karafka consumer connectivity after renewal | | Karafka Web UI SSL | < 30 days to expiry | Renew Web UI cert; verify browser access after renewal |

Alert after: 1 consecutive failure for the Kafka TCP and Web UI HTTP monitors. 2 consecutive failures for the API and Schema Registry monitors.

Common Karafka Failure Modes and What Vigilmon Catches

| Scenario | Vigilmon monitor | |---|---| | Kafka broker OOM killed | TCP monitor fires; consumers stop processing; alert within 60 s | | Network partition to Kafka cluster | TCP monitor fires; Karafka consumers enter reconnection loop | | Karafka Web UI process crash | HTTP monitor fires; team loses consumer lag visibility | | Kafka broker TLS certificate expired | SSL monitor alerts; all TLS-enabled consumer/producer connections fail | | Schema Registry unreachable | Schema Registry monitor fires; producers fail to serialize new messages | | Rails app deployment failure | Web UI HTTP monitor fires if Web UI is mounted in Rails | | Kafka cluster rolling restart (too slow) | TCP monitor catches individual brokers during restart windows | | Firewall rule change blocks Kafka port | TCP monitor fires immediately | | DNS misconfiguration | All monitors fire simultaneously | | Consumer group rebalancing (extended) | Not caught by external monitoring; visible in Web UI consumer lag view |

Monitoring Consumer Lag (Beyond Infrastructure)

Vigilmon monitors Karafka's infrastructure dependencies — Kafka broker connectivity, Web UI availability, and SSL certificates. Consumer lag and processing health require application-level observability:

  • Karafka Web UI: The primary tool for consumer lag visibility. Consumer group offsets, lag per partition, and processing errors are displayed in the Web UI dashboard — set up Vigilmon to ensure the Web UI is always available when you need it.
  • Prometheus metrics: Karafka exposes Prometheus-compatible metrics via the karafka-statsd or karafka-datadog gems, or via custom listeners. Monitor karafka.consumer.messages.consumed.total and karafka.consumer.lag for application-layer alerting.
  • Dead letter queue monitoring: Configure Karafka's Dead Letter Queue (DLQ) feature to route failed messages to a dead letter topic. Monitor the DLQ topic's message count to detect persistent processing failures.

Vigilmon catches infrastructure-level failures. Consumer lag and processing errors require the Karafka Web UI and application-level metrics.

Karafka is the event-driven backbone for Ruby and Rails applications — when Kafka broker connectivity is lost, every consumer in your application stops processing events, and business-critical messages accumulate in topics until the connection is restored. Vigilmon gives you external visibility into Karafka's infrastructure before consumer lag becomes a customer problem: Kafka broker TCP reachability, Web UI availability, schema registry health, and SSL certificate validity, so you know the moment your event processing pipeline is threatened and can act before message backlogs affect application functionality.

Start monitoring Karafka in under 5 minutes — register free at vigilmon.online.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles