tutorial

Monitoring KubeDB with Vigilmon: Operator Health, Database Endpoint Availability, Phase Checks & Backup Heartbeats

How to monitor KubeDB (Kubernetes database management operator) with Vigilmon — operator health checks, database endpoint availability via TCP monitors, phase checks, SSL certificate monitoring, and backup completion heartbeats.

KubeDB is the Kubernetes operator that manages the full lifecycle of production databases — PostgreSQL, MySQL, MongoDB, Redis, Elasticsearch, and more — running natively inside your cluster. It handles provisioning, scaling, version upgrades, automated failover, and backup orchestration so your database team can treat databases as Kubernetes resources. When the KubeDB operator fails or a managed database enters a broken phase, your applications lose their data layer and automated recovery stops working. Vigilmon gives you external visibility into the KubeDB operator health, database endpoint availability, backup completion heartbeats, and SSL certificates so you catch database failures before your applications do.

What You'll Build

  • A monitor on the KubeDB operator health endpoint
  • TCP monitors for database port availability
  • An HTTP monitor for a database with a web UI or health endpoint
  • SSL certificate monitoring for database ingress domains
  • Heartbeat monitors confirming scheduled database backups complete on time

Prerequisites

  • A running KubeDB installation in Kubernetes with managed databases
  • At least one database service exposed via a Service, LoadBalancer, or Ingress
  • A free account at vigilmon.online

Step 1: Verify the KubeDB Operator Health

The KubeDB operator runs as a deployment in the kubedb namespace (or your configured namespace) and exposes a metrics endpoint. To verify the operator pod is healthy:

kubectl get pods -n kubedb

Expected output shows the KubeDB operator and webhook server in Running state:

NAME                       READY   STATUS    RESTARTS   AGE
kubedb-operator-xxx        1/1     Running   0          5d
kubedb-webhook-server-xxx  1/1     Running   0          5d

For external monitoring, expose the KubeDB operator metrics or use a dedicated health check endpoint. KubeDB exposes Prometheus metrics at port 8080 on the operator pod. If you expose this via an Ingress:

curl https://kubedb-metrics.example.com/metrics | grep kubedb_operator

A healthy operator response contains metrics such as kubedb_operator_ready.

Step 2: Monitor KubeDB Operator via Kubernetes API Proxy

If your KubeDB operator is not exposed externally, use a lightweight health probe deployment that reflects operator state via an HTTP endpoint. Alternatively, expose the operator's built-in /readyz endpoint:

curl https://kubedb-operator.example.com/readyz

A healthy KubeDB operator returns ok on its readiness endpoint:

  1. Log in to Vigilmon → Add Monitor → HTTP.
  2. URL: https://kubedb-operator.example.com/readyz.
  3. Check interval: 60 seconds.
  4. Response timeout: 10 seconds.
  5. Expected status: 200.
  6. Keyword: ok.
  7. Label: KubeDB Operator Health.
  8. Click Save.

Set alerts after 1 consecutive failure — a KubeDB operator outage means no database provisioning, scaling, or failover operations can execute.

Step 3: Monitor Database Port Availability with TCP Monitors

The most direct health signal for a KubeDB-managed database is whether its port is open and accepting connections. Vigilmon's TCP monitor checks connectivity without requiring database credentials:

PostgreSQL (port 5432):

  1. Add Monitor → TCP.
  2. Host: postgres.example.com (or your PostgreSQL service's external hostname/IP).
  3. Port: 5432.
  4. Check interval: 60 seconds.
  5. Label: KubeDB PostgreSQL.
  6. Click Save.

MySQL (port 3306):

  1. Add Monitor → TCP.
  2. Host: mysql.example.com.
  3. Port: 3306.
  4. Check interval: 60 seconds.
  5. Label: KubeDB MySQL.
  6. Click Save.

MongoDB (port 27017):

  1. Add Monitor → TCP.
  2. Host: mongodb.example.com.
  3. Port: 27017.
  4. Check interval: 60 seconds.
  5. Label: KubeDB MongoDB.
  6. Click Save.

Redis (port 6379):

  1. Add Monitor → TCP.
  2. Host: redis.example.com.
  3. Port: 6379.
  4. Check interval: 60 seconds.
  5. Label: KubeDB Redis.
  6. Click Save.

Exposing database ports: Database services in Kubernetes are typically ClusterIP. To monitor them externally, expose them via a LoadBalancer service, a NodePort, or an Ingress with a TCP pass-through. Use caution exposing database ports publicly — restrict access by IP allowlist or VPN where possible.

Step 4: Monitor Elasticsearch or OpenSearch with an HTTP Health Check

KubeDB-managed Elasticsearch exposes a REST API with a cluster health endpoint that provides richer health information than a TCP port check:

curl https://elasticsearch.example.com/_cluster/health \
  -u elastic:your-password

A healthy Elasticsearch cluster returns:

{"cluster_name":"my-es","status":"green","number_of_nodes":3}

Add an HTTP monitor with keyword matching:

  1. Add Monitor → HTTP.
  2. URL: https://elasticsearch.example.com/_cluster/health.
  3. Check interval: 2 minutes.
  4. Expected status: 200.
  5. Keyword: green (confirms cluster is fully healthy; yellow means replica shards are unassigned but data is available).
  6. HTTP Headers: Authorization: Basic base64(elastic:your-password).
  7. Label: KubeDB Elasticsearch Cluster Health.
  8. Click Save.

Yellow vs. green: Use green for strict alerting on fully redundant clusters. For single-node dev environments, use yellow to avoid false alerts when replica shards cannot be assigned.

Step 5: Set Up Backup Completion Heartbeat Monitoring

KubeDB integrates with Stash (Appscode's backup operator) to schedule automated database backups. A BackupSession that fails silently leaves your database unprotected. Use Vigilmon heartbeat monitoring to confirm backups complete on schedule:

Step 5a — Create a Heartbeat monitor in Vigilmon:

  1. Add Monitor → Heartbeat.
  2. Name: KubeDB PostgreSQL daily backup.
  3. Expected interval: 24 hours (or match your BackupConfiguration schedule).
  4. Grace period: 2 hours.
  5. Copy the generated ping URL (e.g., https://vigilmon.online/ping/abc123).

Step 5b — Add a post-backup ping to your Stash BackupConfiguration:

Stash supports postBackupActions via hooks. Alternatively, deploy a CronJob that checks the latest BackupSession status:

apiVersion: batch/v1
kind: CronJob
metadata:
  name: kubedb-backup-heartbeat
  namespace: default
spec:
  schedule: "30 3 * * *"
  jobTemplate:
    spec:
      template:
        spec:
          serviceAccountName: default
          containers:
            - name: ping
              image: bitnami/kubectl:latest
              command:
                - sh
                - -c
                - |
                  # Check that the latest BackupSession succeeded
                  STATUS=$(kubectl get backupsession -n default \
                    --sort-by=.metadata.creationTimestamp \
                    -o jsonpath='{.items[-1].status.phase}')
                  if [ "$STATUS" = "Succeeded" ]; then
                    curl -fsS -m 10 https://vigilmon.online/ping/abc123
                  fi
          restartPolicy: OnFailure

This CronJob checks the most recent BackupSession's phase and pings Vigilmon only when the backup succeeded — giving you a signal that actual data was protected, not just that the schedule fired.

Step 6: Monitor KubeDB Database Phase

KubeDB tracks each managed database's operational phase in its custom resource (Ready, Halted, Provisioning, Critical, NotReady). A database stuck in Critical or NotReady indicates a failure the operator detected but cannot automatically recover. You can expose this check via a CronJob that pings a dedicated Vigilmon heartbeat only when all databases are Ready:

# Check all PostgreSQL instances are Ready
UNHEALTHY=$(kubectl get postgres -A \
  -o jsonpath='{.items[?(@.status.phase!="Ready")].metadata.name}')
if [ -z "$UNHEALTHY" ]; then
  curl -fsS -m 10 https://vigilmon.online/ping/db-phase-ok
fi

Pair this with a Vigilmon heartbeat with a 10-minute expected interval and a 15-minute grace period to catch databases entering critical state within one check cycle.

Step 7: Monitor SSL Certificates

Database ingress endpoints and admin UIs often use TLS. An expired certificate blocks all TLS-authenticated database connections and causes application errors that appear as database failures:

  1. Add Monitor → SSL Certificate.
  2. Domain: postgres.example.com (or your database ingress domain).
  3. Alert when expiry is within: 30 days.
  4. Alert again at: 14 days, 7 days, 3 days, 1 day.
  5. Click Save.

Repeat for each database endpoint exposed over TLS.

Step 8: Configure Alerting

In Vigilmon under Settings → Notifications, configure your alert channels:

| Monitor | Trigger | Action | |---|---|---| | KubeDB operator health | Non-200 or ok missing | Check operator pod; kubectl get pods -n kubedb | | PostgreSQL TCP | Connection refused | Check pod status; kubectl describe postgres my-pg | | MySQL TCP | Connection refused | Check pod and service; verify LoadBalancer IP | | MongoDB TCP | Connection refused | Check replica set health and pod logs | | Elasticsearch health | green absent | Check shard allocation; yellow may be normal on small clusters | | Backup heartbeat | No ping within window | BackupSession failed; check Stash logs | | Database phase heartbeat | No ping within window | Database in Critical or NotReady; check events | | SSL certificate | < 30 days | Renew; verify cert-manager TLS config |

Common KubeDB Failure Modes and What Vigilmon Catches

| Scenario | Vigilmon monitor | |---|---| | KubeDB operator pod crash | Operator health endpoint unreachable; alert within 60 s | | Database pod OOMKilled | TCP port closed; monitor fires within 60 s | | PostgreSQL primary election stalls | TCP port unavailable during failover | | MongoDB replica set split | Port may remain open; phase heartbeat detects Critical | | Elasticsearch cluster red | HTTP monitor loses green keyword | | Backup session fails silently | Heartbeat monitor alerts after grace period | | Database in NotReady phase | Phase heartbeat monitor stops pinging | | SSL certificate expires | SSL monitor alerts at 30-day threshold |

KubeDB brings database operations into the Kubernetes control plane, but the operator itself is a single point of failure for database lifecycle management — and managed databases can enter critical states without surfacing obvious alerts to your application tier. Vigilmon's layered monitoring of the operator health, database port availability, Elasticsearch cluster state, backup completion heartbeats, database phase checks, and SSL certificates gives you a complete external view of your KubeDB-managed data layer, so you catch failures before your applications surface them as errors.

Start monitoring KubeDB in under 5 minutes — register free at vigilmon.online.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles