Monitoring Kubewatch with Vigilmon: Event Stream Health, Webhook Availability, Slack/PagerDuty Integration & SSL Certificate Alerts

Kubewatch is the Kubernetes event watcher that listens to cluster resource changes and forwards them to Slack, PagerDuty, webhooks, and other notification channels. When the Kubewatch pod crashes or loses its connection to the Kubernetes API, your team stops receiving pod crash alerts, deployment notifications, and node failure warnings — silently, without any indication that the alerting pipeline itself has gone dark. When the upstream webhook handler your Kubewatch is targeting becomes unavailable, events queue up or get dropped entirely. Vigilmon gives you external visibility into Kubewatch's health endpoint, the availability of its upstream notification targets, and the SSL certificates that protect the entire delivery chain so you know the moment your event alerting goes offline.

What You'll Build

• An HTTP monitor on Kubewatch's health endpoint to detect process failures and API connectivity loss
• HTTP monitors on the upstream webhook and Slack/PagerDuty endpoints Kubewatch forwards events to
• A TCP check to confirm Kubewatch's webhook receiver port is accepting connections
• SSL certificate monitoring for any HTTPS endpoints in the event delivery chain
• An alert runbook mapping Vigilmon findings to specific kubectl remediation commands

Prerequisites

• A Kubewatch installation (v0.1.0+) deployed in the kubewatch namespace via Helm or manifest
• The Kubewatch pod configured with at least one notification handler (Slack webhook, PagerDuty, or custom webhook)
• A Kubernetes cluster with kubectl access
• A free account at vigilmon.online

Step 1: Understand Kubewatch's Observability Surface

Kubewatch exposes a small HTTP health surface and writes structured logs to stdout. Before configuring external monitoring, inspect what is available:

# Check that the Kubewatch pod is running and not restarting
kubectl get pods -n kubewatch

# View live event delivery logs
kubectl logs -n kubewatch deployment/kubewatch --tail=50 -f

# Check the Kubewatch configmap to see which handlers are configured
kubectl get configmap -n kubewatch kubewatch-config -o yaml

# Port-forward to the Kubewatch HTTP endpoint if available
kubectl port-forward -n kubewatch deployment/kubewatch 8080:8080
curl http://localhost:8080/healthz

The key signals are:

• Pod restarts: High restart counts in kubectl get pods indicate the Kubernetes API watcher is disconnecting and the process is crashing.
• Log output: Messages like failed to post to webhook or slack: cannot post message confirm delivery failures even when the pod stays running.
• Health endpoint: Kubewatch exposes /healthz on port 8080 in most Helm chart deployments.

Step 2: Expose the Kubewatch Health Endpoint

The Kubewatch health endpoint is reachable inside the cluster by default. To enable external monitoring from Vigilmon, expose it through an Ingress or NodePort:

# kubewatch-ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: kubewatch-health
  namespace: kubewatch
  annotations:
    nginx.ingress.kubernetes.io/auth-type: basic
    nginx.ingress.kubernetes.io/auth-secret: kubewatch-basic-auth
spec:
  rules:
  - host: kubewatch-health.example.com
    http:
      paths:
      - path: /healthz
        pathType: Exact
        backend:
          service:
            name: kubewatch
            port:
              number: 8080
  tls:
  - hosts:
    - kubewatch-health.example.com
    secretName: kubewatch-health-tls

Apply and verify:

kubectl apply -f kubewatch-ingress.yaml
curl -u monitor:password https://kubewatch-health.example.com/healthz
# Expected: {"status":"ok"} or HTTP 200

Step 3: Create a Vigilmon HTTP Monitor for the Health Endpoint

1. Log in to Vigilmon → Add Monitor → HTTP.
2. URL: https://kubewatch-health.example.com/healthz.
3. Check interval: 2 minutes.
4. Response timeout: 10 seconds.
5. Expected status: 200.
6. Keyword: ok (present in the healthy JSON response).
7. Label: kubewatch health endpoint.
8. Click Save.

This monitor catches:

• Kubewatch pod OOM kills that stop event delivery
• Kubernetes API watcher failures that cause the process to crash-loop
• Deployment rollout failures after Helm upgrades to a new Kubewatch version
• Node evictions that move the pod and cause a brief unavailability window
• Namespace-level network policy changes that isolate the Kubewatch pod

Alert sensitivity: Set to trigger after 2 consecutive failures to suppress false positives during cluster node drains.

Step 4: Monitor the Upstream Webhook Endpoint

If Kubewatch is configured to forward events to a custom webhook URL, that endpoint must stay reachable for events to be delivered. A healthy Kubewatch pod that cannot reach its upstream target silently drops events:

# Typical Kubewatch webhook configuration in kubewatch-config.yaml
handler:
  webhook:
    enabled: true
    url: "https://events-receiver.example.com/kubewatch"

Check your Kubewatch configmap for the configured webhook URL:

kubectl get configmap -n kubewatch kubewatch-config -o jsonpath='{.data.\.kubewatch\.yaml}'

In Vigilmon:

1. Add Monitor → HTTP.
2. URL: https://events-receiver.example.com/kubewatch (or a /health path on the same host).
3. Check interval: 2 minutes.
4. Expected status: 200 (or 405 Method Not Allowed if the endpoint only accepts POST).
5. Label: kubewatch upstream webhook target.
6. Click Save.

Why upstream monitoring matters: Kubewatch logs delivery failures but does not stop processing. If your webhook receiver is down for 30 minutes, Kubewatch will have attempted hundreds of deliveries, logged thousands of errors, and delivered zero events to your team — all while reporting a healthy pod status.

Step 5: Monitor Slack and PagerDuty Integration Health

If Kubewatch is configured to send directly to Slack or PagerDuty, the health of those third-party integration endpoints determines whether your alerts reach anyone:

# Check if Kubewatch logs show Slack delivery errors
kubectl logs -n kubewatch deployment/kubewatch | grep -i "slack\|failed\|error" | tail -20

# Test the Slack webhook manually
curl -X POST -H 'Content-type: application/json' \
  --data '{"text":"Vigilmon connectivity test"}' \
  https://hooks.slack.com/services/YOUR/SLACK/WEBHOOK

For Slack webhook monitoring in Vigilmon:

1. Add Monitor → HTTP.
2. URL: https://hooks.slack.com/services/YOUR/SLACK/WEBHOOK (use a test/canary webhook).
3. Method: POST with body {"text":"vigilmon-ping"}.
4. Expected status: 200.
5. Label: kubewatch slack webhook health.
6. Click Save.

For PagerDuty, monitor the events API endpoint at https://events.pagerduty.com/v2/enqueue the same way using a low-severity test event payload.

Step 6: Monitor SSL Certificates

Kubewatch communicates with HTTPS endpoints for every event delivery. Certificate failures cause delivery errors that look identical to network errors in Kubewatch logs:

# Check the certificate on your webhook endpoint
openssl s_client -connect events-receiver.example.com:443 2>/dev/null | \
  openssl x509 -noout -dates

# Check Kubewatch's own Ingress certificate
openssl s_client -connect kubewatch-health.example.com:443 2>/dev/null | \
  openssl x509 -noout -dates

In Vigilmon:

1. Add Monitor → SSL Certificate.
2. Domain: kubewatch-health.example.com.
3. Alert when expiry is within: 30 days.
4. Alert again: 14 days, 7 days, 3 days, 1 day.
5. Click Save.

Repeat for events-receiver.example.com and any other HTTPS targets in your Kubewatch handler configuration.

Step 7: Configure Alerting and Runbook

In Vigilmon under Settings → Notifications, configure your alert channels:

| Monitor | Trigger | Immediate Action | |---|---|---| | Health endpoint | Non-200 or ok keyword missing | kubectl get pods -n kubewatch → check restart count; kubectl logs -n kubewatch deployment/kubewatch --tail=30 | | Upstream webhook | Non-200 or connection refused | Check webhook receiver service; verify network path from cluster to receiver | | Slack webhook | Non-200 | Validate Slack app token hasn't been revoked; check Slack status page | | SSL certificate | < 30 days to expiry | Renew certificate on Ingress; check cert-manager CertificateRequest status |

Escalation policy: Route Kubewatch health alerts to the same channel Kubewatch itself would send alerts to — but via a separate path (e.g., Vigilmon → PagerDuty directly). This prevents the meta-problem: Kubewatch being down means you can't use Kubewatch to alert about Kubewatch being down.

Common Kubewatch Failure Modes and What Vigilmon Catches

| Scenario | Vigilmon monitor | |---|---| | Kubewatch pod OOM killed by node | Health endpoint returns connection error; alert within 2 min | | Kubernetes API server upgrade causes watcher disconnect | Pod crash-loops; health endpoint goes down; event delivery stops | | Slack API token revoked or expired | Health endpoint stays green; Slack monitor catches delivery failure | | Upstream webhook SSL certificate expired | Health endpoint stays green; webhook target SSL monitor catches 30 days early | | Network policy blocks Kubewatch egress to Slack | Health endpoint green; Slack webhook monitor catches at next check | | Kubewatch namespace deleted accidentally | Health endpoint returns connection refused; alert within 2 min | | Helm upgrade introduces bad configuration | Pod fails to start; health endpoint down during rollout | | Webhook receiver service horizontal scaling event | Brief TCP connection errors; TCP monitor catches transient failures |

Kubernetes events are your cluster's primary signal for failures, deployments, and capacity problems — but that signal is useless if Kubewatch is silently failing to deliver it. A crashed Kubewatch pod means your team is blind to pod OOM kills, failed deployments, and node pressure events happening in real time. Vigilmon closes this blind spot by checking Kubewatch's own health endpoint alongside the upstream notification targets it depends on, so your event alerting pipeline is always independently verified and you are never left debugging a silent cluster alone.

Start monitoring Kubewatch in under 5 minutes — register free at vigilmon.online.