tutorial

Monitoring KubeBlocks with Vigilmon: Controller Health, Database Cluster Endpoints, Addon Status & Backup Heartbeats

How to monitor KubeBlocks (cloud-native database management platform) with Vigilmon — controller health checks, database cluster endpoint availability, addon status, SSL certificate monitoring, and backup completion heartbeats.

KubeBlocks is the open-source, cloud-native database management platform that unifies management of relational databases, NoSQL stores, vector databases, and streaming platforms on Kubernetes — supporting MySQL, PostgreSQL, Redis, MongoDB, Kafka, Pulsar, and more through a consistent operator framework. When the KubeBlocks controller fails or a managed cluster enters a degraded state, all database lifecycle operations halt and applications lose their data tier. Vigilmon gives you external visibility into the KubeBlocks controller health, database cluster endpoint availability, addon status, backup completion heartbeats, and SSL certificates so you catch failures before they propagate to your applications.

What You'll Build

  • A monitor on the KubeBlocks controller health endpoint
  • TCP monitors for database cluster port availability
  • An HTTP monitor for databases exposing REST health endpoints
  • SSL certificate monitoring for database ingress domains
  • Heartbeat monitors confirming scheduled backups complete on time

Prerequisites

  • A running KubeBlocks installation in Kubernetes (v0.8+) with managed database clusters
  • At least one database cluster service exposed externally via LoadBalancer, NodePort, or Ingress
  • A free account at vigilmon.online

Step 1: Verify the KubeBlocks Controller Health

The KubeBlocks controller (kbcli manager) runs as a deployment and exposes a health endpoint confirming the controller is operational. Verify the controller pod is healthy:

kubectl get pods -n kb-system

Expected output shows the KubeBlocks controller in Running state:

NAME                          READY   STATUS    RESTARTS   AGE
kubeblocks-xxx                1/1     Running   0          3d
kubeblocks-dataprotection-xxx 1/1     Running   0          3d

For external monitoring, the KubeBlocks controller exposes a /readyz and /healthz endpoint on port 8081. Expose this via an Ingress:

curl https://kubeblocks-ctrl.example.com/healthz

A healthy KubeBlocks controller returns ok:

Step 2: Create a Vigilmon HTTP Monitor for the KubeBlocks Controller

  1. Log in to Vigilmon → Add Monitor → HTTP.
  2. URL: https://kubeblocks-ctrl.example.com/healthz.
  3. Check interval: 60 seconds.
  4. Response timeout: 10 seconds.
  5. Expected status: 200.
  6. Keyword: ok.
  7. Label: KubeBlocks Controller Health.
  8. Click Save.

Set alerts after 1 consecutive failure — a KubeBlocks controller outage blocks all cluster provisioning, scaling, version upgrades, and failover operations.

Step 3: Monitor the KubeBlocks Controller Readiness Endpoint

In addition to the /healthz liveness probe, the /readyz endpoint confirms the controller has finished its startup reconciliation and is ready to process requests:

curl https://kubeblocks-ctrl.example.com/readyz
  1. Add Monitor → HTTP.
  2. URL: https://kubeblocks-ctrl.example.com/readyz.
  3. Check interval: 2 minutes.
  4. Expected status: 200.
  5. Keyword: ok.
  6. Label: KubeBlocks Controller Readiness.
  7. Click Save.

Step 4: Monitor Database Cluster Endpoints with TCP Monitors

The most direct measure of a KubeBlocks-managed database cluster is whether its port accepts connections. Vigilmon's TCP monitor confirms connectivity without requiring database credentials:

PostgreSQL cluster (port 5432):

  1. Add Monitor → TCP.
  2. Host: pg-cluster.example.com (your PostgreSQL cluster's external hostname or LoadBalancer IP).
  3. Port: 5432.
  4. Check interval: 60 seconds.
  5. Label: KubeBlocks PostgreSQL Cluster.
  6. Click Save.

MySQL cluster (port 3306):

  1. Add Monitor → TCP.
  2. Host: mysql-cluster.example.com.
  3. Port: 3306.
  4. Check interval: 60 seconds.
  5. Label: KubeBlocks MySQL Cluster.
  6. Click Save.

Redis cluster (port 6379):

  1. Add Monitor → TCP.
  2. Host: redis-cluster.example.com.
  3. Port: 6379.
  4. Check interval: 60 seconds.
  5. Label: KubeBlocks Redis Cluster.
  6. Click Save.

Kafka (port 9092):

  1. Add Monitor → TCP.
  2. Host: kafka.example.com.
  3. Port: 9092.
  4. Check interval: 60 seconds.
  5. Label: KubeBlocks Kafka.
  6. Click Save.

Exposing services: KubeBlocks database clusters use ClusterIP services by default. Expose them externally with kubectl patch service <cluster>-<component> -n default -p '{"spec":{"type":"LoadBalancer"}}' or configure an Ingress with TCP pass-through. Restrict access by IP allowlist where possible.

Step 5: Monitor Kafka and Pulsar with HTTP Health Endpoints

KubeBlocks-managed Kafka (via KRaft mode) and Pulsar expose HTTP health endpoints that provide richer cluster state than a raw TCP check:

Kafka health (if the Kafka REST Proxy addon is enabled):

curl https://kafka-rest.example.com/v3/clusters

Pulsar health:

curl https://pulsar.example.com/admin/v2/brokers/health

A healthy Pulsar broker returns ok:

  1. Add Monitor → HTTP.
  2. URL: https://pulsar.example.com/admin/v2/brokers/health.
  3. Check interval: 2 minutes.
  4. Expected status: 200.
  5. Keyword: ok.
  6. Label: KubeBlocks Pulsar Broker Health.
  7. Click Save.

Step 6: Monitor KubeBlocks Addon Status

KubeBlocks uses addons to extend support for additional database engines. A failed addon means that database type's operator is non-functional and clusters of that type cannot be managed. Monitor addon health via a CronJob that pings Vigilmon only when all enabled addons are in Enabled state:

apiVersion: batch/v1
kind: CronJob
metadata:
  name: kubeblocks-addon-heartbeat
  namespace: kb-system
spec:
  schedule: "*/10 * * * *"
  jobTemplate:
    spec:
      template:
        spec:
          serviceAccountName: kubeblocks
          containers:
            - name: check
              image: bitnami/kubectl:latest
              command:
                - sh
                - -c
                - |
                  FAILED=$(kubectl get addon -n kb-system \
                    -o jsonpath='{.items[?(@.status.phase!="Enabled")].metadata.name}')
                  if [ -z "$FAILED" ]; then
                    curl -fsS -m 10 https://vigilmon.online/ping/addon-ok
                  fi
          restartPolicy: OnFailure

Create a Vigilmon heartbeat with a 15-minute expected interval and a 20-minute grace period, and pair it with this CronJob.

Step 7: Set Up Backup Completion Heartbeat Monitoring

KubeBlocks integrates with its dataprotection component for scheduled backups via BackupPolicy and BackupSchedule resources. A backup that fails silently leaves your clusters unprotected. Use Vigilmon heartbeat monitoring to verify backups complete:

Step 7a — Create a Heartbeat monitor in Vigilmon:

  1. Add Monitor → Heartbeat.
  2. Name: KubeBlocks PostgreSQL daily backup.
  3. Expected interval: 24 hours.
  4. Grace period: 2 hours.
  5. Copy the generated ping URL (e.g., https://vigilmon.online/ping/abc123).

Step 7b — Add a post-backup verification CronJob:

apiVersion: batch/v1
kind: CronJob
metadata:
  name: kubeblocks-backup-heartbeat
  namespace: default
spec:
  schedule: "30 4 * * *"
  jobTemplate:
    spec:
      template:
        spec:
          serviceAccountName: default
          containers:
            - name: check
              image: bitnami/kubectl:latest
              command:
                - sh
                - -c
                - |
                  # Check the latest Backup for the cluster
                  STATUS=$(kubectl get backup -n default \
                    --sort-by=.metadata.creationTimestamp \
                    -o jsonpath='{.items[-1].status.phase}')
                  if [ "$STATUS" = "Completed" ]; then
                    curl -fsS -m 10 https://vigilmon.online/ping/abc123
                  fi
          restartPolicy: OnFailure

This checks the most recent Backup object's phase and pings Vigilmon only on Completed — confirming that actual backup data was written, not just that the schedule executed.

Step 8: Monitor SSL Certificates

Database cluster endpoints and admin UIs exposed via Ingress use TLS. An expired certificate causes connection failures across all TLS-authenticated clients and appears as a database outage to your applications:

  1. Add Monitor → SSL Certificate.
  2. Domain: pg-cluster.example.com (or your database cluster's ingress domain).
  3. Alert when expiry is within: 30 days.
  4. Alert again at: 14 days, 7 days, 3 days, 1 day.
  5. Click Save.

Repeat for each database cluster endpoint exposed over TLS.

Step 9: Configure Alerting

In Vigilmon under Settings → Notifications, configure your alert channels:

| Monitor | Trigger | Action | |---|---|---| | Controller health | Non-200 or ok missing | Check controller pod; kubectl get pods -n kb-system | | Controller readiness | Non-200 | Controller restarting or stuck in reconciliation | | PostgreSQL TCP | Connection refused | Check cluster phase; kubectl describe cluster my-pg -n default | | MySQL TCP | Connection refused | Check pod status and service endpoint | | Redis TCP | Connection refused | Check replica failover status | | Kafka TCP | Connection refused | Check broker pods and KRaft leader election | | Pulsar health | ok absent | Broker failure; check dataprotection and broker logs | | Addon heartbeat | No ping within window | Addon failed; kubectl get addon -n kb-system | | Backup heartbeat | No ping within window | Backup failed; check BackupSchedule and dataprotection logs | | SSL certificate | < 30 days | Renew; verify cert-manager TLS config |

Common KubeBlocks Failure Modes and What Vigilmon Catches

| Scenario | Vigilmon monitor | |---|---| | KubeBlocks controller pod crash | /healthz unreachable; alert within 60 s | | Database cluster pod OOMKilled | TCP port closed; monitor fires within 60 s | | PostgreSQL primary failover stall | TCP port unavailable during election | | Addon fails to enable | Addon heartbeat stops pinging | | Backup job fails silently | Backup heartbeat alerts after grace period | | Kafka broker evicted | TCP port closed; alert within 60 s | | Pulsar broker unhealthy | HTTP health monitor loses ok keyword | | SSL certificate expires | SSL monitor alerts at 30-day threshold | | Cluster in Abnormal phase | TCP port may remain open; pair with phase-check CronJob |

KubeBlocks's unified multi-database management model means the controller is a single operational dependency for dozens of database clusters — and when it fails, every managed database loses lifecycle management without surfacing an obvious error to applications. Vigilmon's layered monitoring of the KubeBlocks controller, database cluster endpoints, addon health, backup completion heartbeats, and SSL certificates gives you complete external visibility across your entire KubeBlocks-managed data platform, so you catch failures before your applications and users do.

Start monitoring KubeBlocks in under 5 minutes — register free at vigilmon.online.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles