tutorial

How to Monitor BotKube with Vigilmon (Kubernetes ChatOps Reliability)

BotKube is a Kubernetes alerting and ChatOps tool that routes cluster events, pod alerts, and resource notifications into Slack, Microsoft Teams, Discord, an...

BotKube is a Kubernetes alerting and ChatOps tool that routes cluster events, pod alerts, and resource notifications into Slack, Microsoft Teams, Discord, and other messaging platforms. It's the glue between your Kubernetes cluster and your team's communication channels.

When BotKube is healthy, your team gets real-time Kubernetes alerts in their chat tool. When BotKube is down — due to a pod restart, a misconfigured webhook, or a Slack API outage — alerts stop flowing silently. You might not notice until a serious cluster event goes unreported for hours.

This tutorial walks through monitoring BotKube with Vigilmon so you catch BotKube failures independently of BotKube itself.

Why you need external monitoring for BotKube

BotKube runs as a pod in your Kubernetes cluster. Like any pod, it can:

  • Crash and restart — CrashLoopBackOff is possible if BotKube's config is invalid or its webhook receiver can't bind
  • Lose connectivity to the Kubernetes API — BotKube relies on the K8s API server to watch events; a network partition or RBAC change breaks this silently
  • Fail its Slack/Teams integration — bot token expiry, workspace changes, or API rate limiting can silently break the notification pipeline
  • Run but stop processing events — a goroutine deadlock or memory pressure can cause BotKube to appear Running while delivering nothing

If BotKube is your primary Kubernetes alerting system, you need a completely independent external check to confirm it's alive and delivering.

What you'll need

  • BotKube running in your Kubernetes cluster
  • BotKube's API or webhook receiver exposed (via Service/Ingress or port-forward for testing)
  • A free Vigilmon account — sign up takes 30 seconds

Step 1: Identify BotKube's health and API endpoints

BotKube exposes an HTTP metrics/health endpoint on port 2113 by default. Check your BotKube deployment:

kubectl get deployment -n botkube botkube -o yaml | grep -A5 "ports:"

Typical BotKube port configuration:

ports:
  - name: metrics
    containerPort: 2113
    protocol: TCP

BotKube exposes Prometheus metrics at :2113/metrics and a liveness path at :2113/healthz. Expose the health endpoint externally via a Kubernetes Service:

# botkube-health-svc.yaml
apiVersion: v1
kind: Service
metadata:
  name: botkube-health
  namespace: botkube
spec:
  selector:
    app.kubernetes.io/name: botkube
  ports:
    - name: metrics
      port: 2113
      targetPort: 2113
      protocol: TCP
  type: ClusterIP

Then expose it via an Ingress or LoadBalancer for external monitoring. If you use an Ingress:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: botkube-health
  namespace: botkube
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /
spec:
  rules:
    - host: botkube-health.yourdomain.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: botkube-health
                port:
                  number: 2113

Apply and verify:

kubectl apply -f botkube-health-svc.yaml
curl https://botkube-health.yourdomain.com/healthz
# Expected: HTTP 200 with "ok" body

Step 2: Monitor the BotKube health endpoint

Log in to vigilmon.online and go to Monitors → New Monitor.

| Field | Value | |-------|-------| | Type | HTTP / HTTPS | | Name | BotKube Agent Health | | URL | https://botkube-health.yourdomain.com/healthz | | Check interval | 1 minute | | Expected status code | 200 | | Response body contains | ok |

Click Save. Vigilmon will start probing from multiple regions.

Step 3: Monitor the webhook receiver availability

BotKube can also act as a webhook receiver for external systems (like Alertmanager). If you use this feature, the receiver's HTTP endpoint must stay reachable. Check if yours is configured:

kubectl get svc -n botkube

If BotKube exposes an incoming webhook port (default: 2115), add a separate monitor:

| Field | Value | |-------|-------| | Type | HTTP / HTTPS | | Name | BotKube Webhook Receiver | | URL | https://botkube-webhook.yourdomain.com/ | | Check interval | 2 minutes | | Expected status code | 200 or 404 |

A 404 on the root path is acceptable — it confirms the HTTP server is listening even if the root path isn't mapped.

Step 4: Set up a heartbeat to verify alert delivery

The most important check isn't whether BotKube's HTTP server responds — it's whether BotKube is actually processing and delivering alerts. Use Vigilmon's heartbeat monitor to verify the full pipeline.

Create a heartbeat in Vigilmon

  1. Go to Monitors → New Monitor → Heartbeat
  2. Name it: BotKube Alert Pipeline
  3. Expected interval: 10 minutes
  4. Save — copy the ping URL (e.g., https://hb.vigilmon.online/ping/abc123xyz)

Configure a BotKube source to ping the heartbeat

BotKube supports Kubernetes event sources and can execute kubectl commands via the Slack/Teams integration. Configure a cron-based Kubernetes job that pings Vigilmon:

# botkube-heartbeat-cronjob.yaml
apiVersion: batch/v1
kind: CronJob
metadata:
  name: botkube-heartbeat
  namespace: botkube
spec:
  schedule: "*/10 * * * *"
  jobTemplate:
    spec:
      template:
        spec:
          containers:
            - name: ping
              image: curlimages/curl:latest
              command:
                - curl
                - -sf
                - https://hb.vigilmon.online/ping/abc123xyz
          restartPolicy: OnFailure

kubectl apply -f botkube-heartbeat-cronjob.yaml

Now the chain works: if BotKube's associated infrastructure stops running, the heartbeat pings stop, and Vigilmon alerts you after the expected interval passes.

Step 5: Monitor pod restart alerts with Vigilmon's webhook

Configure Vigilmon to receive BotKube's own pod restart notifications via a webhook. This creates a bi-directional monitoring relationship:

  1. In Vigilmon, go to Alert Channels → Add Channel → Webhook
  2. Copy your Vigilmon notification webhook URL

In BotKube's values.yaml (for Helm installs):

communications:
  default-group:
    webhook:
      enabled: true
      url: https://vigilmon.online/webhook/your-channel-id
      bindings:
        sources:
          - k8s-pod-restarts

Now BotKube will notify Vigilmon directly when pods restart, and Vigilmon handles deduplication and alert routing. This supplements the external health checks with event-driven alerts.

Step 6: Configure alert channels

Connect Vigilmon to your notification stack for BotKube failures.

Email

  1. Go to Alert Channels → Add Channel → Email
  2. Enter your platform team email
  3. Assign to: BotKube Agent Health, BotKube Webhook Receiver, BotKube Alert Pipeline

Slack (separate from BotKube)

Connect Vigilmon directly to Slack as a backup channel — this ensures you receive alerts even when BotKube itself is the one that's down:

  1. Go to Alert Channels → Add Channel → Webhook
  2. Enter your Slack incoming webhook URL (for a different channel than BotKube uses)
  3. Assign to BotKube monitors

When BotKube goes down, Vigilmon sends:

{
  "monitor_name": "BotKube Agent Health",
  "status": "down",
  "url": "https://botkube-health.yourdomain.com/healthz",
  "started_at": "2026-07-01T14:00:00Z",
  "duration_seconds": 90
}

Step 7: Create a status page

For platform teams managing Kubernetes clusters, a status page covering your monitoring and alerting infrastructure is valuable during incident response.

  1. Go to Status Pages → New Status Page
  2. Name it: Kubernetes Alerting Infrastructure
  3. Add monitors: BotKube Agent Health, BotKube Webhook Receiver, BotKube Alert Pipeline
  4. Publish

During a cluster incident, your team can check this page to confirm BotKube itself is operational before assuming alerts are being routed correctly.

Monitor summary

| Monitor | Type | Catches | |---------|------|---------| | BotKube /healthz | HTTP | Pod crash, container restart, process hang | | BotKube webhook receiver | HTTP | Webhook handler failure, port binding issues | | BotKube alert pipeline | Heartbeat | Silent delivery failures, Slack token expiry | | BotKube TCP port | TCP | Pod-level network reachability |

Key Kubernetes checks to add alongside BotKube monitoring

These Vigilmon checks complement BotKube and give you full K8s observability coverage:

# Verify BotKube pod is Running and not restarting
kubectl get pod -n botkube -l app.kubernetes.io/name=botkube

# Check BotKube logs for connection errors
kubectl logs -n botkube -l app.kubernetes.io/name=botkube --tail=50

# Confirm BotKube has proper RBAC to watch cluster events
kubectl auth can-i watch pods --as=system:serviceaccount:botkube:botkube -A

What's next

  • SSL certificate monitoring — if your BotKube health endpoint is behind TLS, Vigilmon tracks the cert expiry automatically
  • Multi-environment coverage — run separate BotKube monitors for each cluster (dev, staging, prod)
  • Incident correlation — use Vigilmon's webhook alerts alongside BotKube's Slack messages to cross-reference when cluster events coincide with BotKube downtime

Keep your Kubernetes alerting pipeline reliable with vigilmon.online — free tier, no credit card required.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles