How to Monitor Redpanda Health and Availability with Vigilmon

Redpanda is a Kafka-compatible event streaming platform built from scratch in C++, delivering lower latency and simpler operations than Apache Kafka by eliminating ZooKeeper and the JVM. It speaks the Kafka protocol natively, so existing Kafka clients, producers, and consumers work without modification. But Redpanda's operational simplicity doesn't change the fundamental challenge of streaming infrastructure: broker failures are silent, consumer lag accumulates invisibly, and by the time users notice something is wrong, your data pipeline may be hours behind.

Vigilmon gives you external visibility into Redpanda availability through HTTP probe monitoring of Redpanda's admin API and Pandaproxy, plus heartbeat monitoring for your consumer applications. This tutorial walks through setting up both.

Why Redpanda Needs External Monitoring

Redpanda ships with rich internal metrics via Prometheus scraping and an admin API — but those tools require active watching. External monitoring with Vigilmon adds:

• Proactive alerting when Redpanda's admin API or Pandaproxy returns non-200
• Kafka port reachability checking to confirm the broker is accepting producer/consumer connections
• Heartbeat monitoring that proves your consumer application is processing messages — even when the broker appears healthy
• Multi-region availability verification from outside your network perimeter

Step 1: Use Redpanda's Built-in HTTP Endpoints

Redpanda exposes several HTTP interfaces you can probe directly without writing any sidecar code:

Admin API (Port 9644)

# Broker health check — returns 200 when broker is healthy
curl -i http://redpanda-host:9644/v1/brokers

# Cluster health overview
curl http://redpanda-host:9644/v1/cluster/health_overview

# Partition leadership status
curl http://redpanda-host:9644/v1/partitions

The /v1/cluster/health_overview endpoint returns a JSON body including "is_healthy": true when the cluster is in a good state. Use this for your primary Vigilmon monitor.

Pandaproxy HTTP Interface (Port 8082)

Pandaproxy is Redpanda's HTTP proxy for producing and consuming Kafka messages over REST. Its availability is critical for HTTP-based producers:

# Pandaproxy health check
curl -i http://redpanda-host:8082/

# List topics via Pandaproxy
curl http://redpanda-host:8082/topics

Schema Registry (Port 8081)

If you use Redpanda's Schema Registry:

# Schema Registry health
curl -i http://redpanda-host:8081/

Step 2: Configure Vigilmon HTTP Monitors for Redpanda

Monitor 1: Redpanda Cluster Health

1. Log in to vigilmon.online and go to Monitors → New Monitor
2. Choose HTTP / HTTPS
3. Set the URL to: http://redpanda-host:9644/v1/cluster/health_overview
4. Set the check interval to 1 minute
5. Under Expected response, configure:
   • Status code: 200
   • Response body contains: "is_healthy":true
   • Response time threshold: 3000ms
6. Under Alert channels, assign your Slack or PagerDuty channel
7. Save the monitor

Monitor 2: Pandaproxy Availability

Add a second monitor for Pandaproxy:

• URL: http://redpanda-host:8082/
• Expected: HTTP 200
• Interval: 1 minute
• Response time threshold: 2000ms
• Alert channel: Slack (P2 — REST producers affected, but Kafka protocol still works)

Monitor 3: Kafka Protocol Port Reachability

For Kafka protocol port reachability (port 9092), build a thin health sidecar in your application that attempts to connect and fetch metadata:

// health/redpanda.js
const express = require('express');
const { Kafka } = require('kafkajs'); // KafkaJS works natively with Redpanda

const app = express();
const kafka = new Kafka({
  clientId: 'health-checker',
  brokers: (process.env.REDPANDA_BROKERS || 'localhost:9092').split(','),
});

const admin = kafka.admin();

app.get('/health/redpanda', async (req, res) => {
  try {
    await admin.connect();

    // Fetch cluster metadata — fails immediately if broker is unreachable
    const cluster = await admin.describeCluster();
    const brokerCount = cluster.brokers.length;

    // Check for under-replicated partitions
    const topics = (process.env.REDPANDA_TOPICS || '').split(',').filter(Boolean);
    let underReplicated = [];

    if (topics.length > 0) {
      const metadata = await admin.fetchTopicMetadata({ topics });
      for (const topic of metadata.topics) {
        for (const partition of topic.partitions) {
          if (partition.isr.length < partition.replicas.length) {
            underReplicated.push(`${topic.name}:${partition.partitionId}`);
          }
        }
      }
    }

    await admin.disconnect();

    if (underReplicated.length > 0) {
      return res.status(503).json({
        status: 'degraded',
        reason: 'under_replicated_partitions',
        partitions: underReplicated,
      });
    }

    return res.status(200).json({
      status: 'ok',
      brokers: brokerCount,
    });
  } catch (err) {
    if (admin) await admin.disconnect().catch(() => {});
    return res.status(503).json({ status: 'down', error: err.message });
  }
});

// Consumer lag check endpoint
app.get('/health/redpanda/lag', async (req, res) => {
  try {
    await admin.connect();

    const groupId = process.env.REDPANDA_CONSUMER_GROUP;
    const topic   = process.env.REDPANDA_TOPIC;

    if (!groupId || !topic) {
      await admin.disconnect();
      return res.status(200).json({ status: 'ok', lag_check: 'skipped' });
    }

    const [topicOffsets, groupOffsets] = await Promise.all([
      admin.fetchTopicOffsets(topic),
      admin.fetchOffsets({ groupId, topics: [topic] }),
    ]);

    let maxLag = 0;
    for (const partitionHigh of topicOffsets) {
      const consumerPartition = groupOffsets[0]?.partitions?.find(
        p => p.partition === partitionHigh.partition
      );
      if (consumerPartition) {
        const lag = parseInt(partitionHigh.high) - parseInt(consumerPartition.offset);
        maxLag = Math.max(maxLag, lag);
      }
    }

    await admin.disconnect();

    const threshold = parseInt(process.env.REDPANDA_LAG_THRESHOLD || '5000');
    if (maxLag > threshold) {
      return res.status(503).json({
        status: 'degraded',
        reason: 'consumer_lag',
        max_lag: maxLag,
        threshold,
      });
    }

    return res.status(200).json({ status: 'ok', max_lag: maxLag });
  } catch (err) {
    if (admin) await admin.disconnect().catch(() => {});
    return res.status(503).json({ status: 'down', error: err.message });
  }
});

app.listen(3005);

Verify before wiring up Vigilmon:

curl -i https://your-app.example.com/health/redpanda
# HTTP/1.1 200 OK
# {"status":"ok","brokers":3}

What These Monitors Catch

| Failure | Process Monitor | Vigilmon | |---|---|---| | Redpanda broker crash | ✓ | ✓ | | Admin API unreachable | ✗ | ✓ | | Pandaproxy down | ✗ | ✓ | | Kafka protocol port blocked | ✗ | ✓ | | Under-replicated partitions | ✗ | ✓ | | Consumer lag spike | ✗ | ✓ |

Step 3: Heartbeat Monitoring for Redpanda Consumers

A Redpanda broker can be entirely healthy while your consumer application is stuck in an offset commit loop, blocked on a downstream service, or paused after a partition rebalance. The admin API won't surface any of this.

Vigilmon's heartbeat monitors prove consumer liveness: your consumer pings Vigilmon after each successfully processed message batch, and Vigilmon fires an alert if pings stop arriving within the expected window.

Set Up the Heartbeat Monitor

1. In Vigilmon, go to Monitors → New Monitor → Heartbeat
2. Set the name: redpanda-order-consumer
3. Set the expected interval: 5 minutes (adjust to your throughput)
4. Set the grace period: 10 minutes
5. Save — copy the heartbeat URL, e.g. https://vigilmon.online/heartbeat/abc123xyz

Wire It Into Your Consumer

Node.js (KafkaJS — fully compatible with Redpanda):

import { Kafka } from 'kafkajs';
import axios from 'axios';

const kafka = new Kafka({
  clientId: 'order-consumer',
  brokers: process.env.REDPANDA_BROKERS.split(','),
});

const consumer = kafka.consumer({ groupId: 'order-processor' });

let batchesSinceLastHeartbeat = 0;
const HEARTBEAT_EVERY = 10;

async function run() {
  await consumer.connect();
  await consumer.subscribe({ topic: 'orders', fromBeginning: false });

  await consumer.run({
    eachBatch: async ({ batch, resolveOffset, heartbeat }) => {
      for (const message of batch.messages) {
        await processMessage(message);
        resolveOffset(message.offset);
        await heartbeat(); // Kafka internal heartbeat — keeps session alive
      }

      batchesSinceLastHeartbeat++;
      if (batchesSinceLastHeartbeat >= HEARTBEAT_EVERY) {
        await axios.get(process.env.VIGILMON_HEARTBEAT_URL).catch(() => {});
        batchesSinceLastHeartbeat = 0;
      }
    },
  });
}

run();

Python (confluent-kafka — Redpanda-compatible):

# consumer.py
from confluent_kafka import Consumer
import requests, os, time

conf = {
    'bootstrap.servers': os.environ['REDPANDA_BROKERS'],
    'group.id': 'order-processor',
    'auto.offset.reset': 'latest',
    'enable.auto.commit': True,
}

consumer = Consumer(conf)
consumer.subscribe([os.environ['REDPANDA_TOPIC']])

last_heartbeat = time.time()
HEARTBEAT_INTERVAL = 60  # seconds

try:
    while True:
        msg = consumer.poll(timeout=1.0)
        if msg and not msg.error():
            process_message(msg)

        if time.time() - last_heartbeat > HEARTBEAT_INTERVAL:
            requests.get(os.environ['VIGILMON_HEARTBEAT_URL'], timeout=5)
            last_heartbeat = time.time()
finally:
    consumer.close()

Step 4: Multi-Broker Cluster Monitoring

For Redpanda clusters, monitor each broker node individually alongside the cluster-level health endpoint:

http://redpanda-node-1:9644/v1/brokers
http://redpanda-node-2:9644/v1/brokers
http://redpanda-node-3:9644/v1/brokers

Use a consistent naming convention:

• [redpanda-cluster] node-1 admin /v1/brokers
• [redpanda-cluster] node-2 admin /v1/brokers
• [redpanda-cluster] node-3 admin /v1/brokers
• [redpanda-cluster] health_overview

Group all into a Redpanda Status Page. When two of three nodes fail, your on-call team sees the full cluster picture immediately rather than processing independent alerts.

Step 5: Alert Routing for Redpanda Outages

Redpanda outages follow a cascade: a broker node fails → partition leadership re-elects → brief producer/consumer pause → lag accumulates → downstream services fall behind. Alert routing should front-load broker failure detection.

| Monitor | Alert Channel | Priority | |---|---|---| | Cluster /v1/cluster/health_overview | Slack + PagerDuty | P1 | | Kafka protocol /health/redpanda | Slack + PagerDuty | P1 | | Consumer lag /health/redpanda/lag | Slack | P2 | | Pandaproxy 8082/ | Slack | P2 | | Schema Registry 8081/ | Slack | P3 | | Consumer heartbeat | Slack + email | P2 |

Response time thresholds:

• Alert at 2000ms for the admin API — normal admin API response is under 200ms
• Alert at 5000ms for the Kafka protocol health check — metadata fetch latency signals broker pressure
• Alert at 1000ms for Pandaproxy — HTTP proxy responses should be fast

Summary

Redpanda's Kafka compatibility means existing monitoring patterns apply directly — but its simplified architecture gives you better HTTP endpoints to probe than vanilla Kafka. Vigilmon gives you:

| Monitor Type | What It Covers | |---|---| | HTTP monitor on admin API /v1/cluster/health_overview | Broker cluster health | | HTTP monitor on Kafka protocol sidecar | Port reachability, under-replicated partitions | | HTTP monitor on Pandaproxy 8082/ | REST producer availability | | HTTP monitor on consumer lag sidecar | Consumer group lag thresholds | | Heartbeat monitor | Consumer application liveness |

Get started free at vigilmon.online — your first Redpanda monitor is running in under two minutes.