tutorial

Monitoring Kubernetes Event Exporter with Vigilmon: Exporter Health, Event Pipeline Uptime, Backend Connectivity & Config Reload Alerts

How to monitor Kubernetes Event Exporter with Vigilmon — health endpoint checks, event pipeline uptime, backend connectivity to Elasticsearch/Loki/webhooks, and config reload detection to ensure your cluster event stream is always flowing.

Kubernetes Event Exporter reads the full Kubernetes event stream — pod evictions, scheduling failures, node pressure warnings, OOM kills — and routes those events to Elasticsearch, Loki, Kafka, webhooks, and other observability backends. When the exporter process crashes or its connection to the Kubernetes API drops, the event stream stops flowing silently: your SIEM, log aggregation platform, and alerting rules receive no cluster events and produce no alerts, with no indication that the pipeline itself has broken. When the backend it writes to becomes unavailable, events are dropped and audit gaps appear in your cluster history. Vigilmon gives you external visibility into the exporter's health endpoint, backend connectivity, and the SSL certificates protecting the delivery path so you know the moment your event pipeline goes dark.

What You'll Build

  • An HTTP monitor on Kubernetes Event Exporter's metrics endpoint to detect process and watcher failures
  • HTTP or TCP monitors on the backend targets the exporter writes events to
  • A config-reload detection monitor that alerts when the exporter configuration has not been refreshed
  • SSL certificate monitoring for HTTPS backends in the event delivery chain
  • An alert runbook mapping Vigilmon findings to specific remediation commands

Prerequisites

  • Kubernetes Event Exporter (v1.0+) deployed in the monitoring namespace via Helm or manifest
  • At least one backend configured (Elasticsearch, Loki, webhook, or Kafka)
  • kubectl access to the cluster
  • A free account at vigilmon.online

Step 1: Understand the Exporter's Observability Surface

Kubernetes Event Exporter emits Prometheus metrics and writes structured logs to stdout. Inspect both before configuring external monitors:

# Confirm the exporter pod is running
kubectl get pods -n monitoring -l app=kubernetes-event-exporter

# View event processing logs
kubectl logs -n monitoring deployment/kubernetes-event-exporter --tail=50

# Check the exporter configuration
kubectl get configmap -n monitoring kubernetes-event-exporter -o yaml

# Port-forward to the metrics endpoint
kubectl port-forward -n monitoring deployment/kubernetes-event-exporter 2112:2112
curl http://localhost:2112/metrics | grep event_exporter

Key metrics to watch:

  • event_exporter_sent_events_total — increments each time an event is successfully forwarded to a backend; a stale counter means the pipeline has stopped.
  • event_exporter_failed_events_total — increments on backend write failures; non-zero sustained values indicate backend connectivity problems.
  • event_exporter_watched_namespaces — confirms the Kubernetes API watch is active.

Step 2: Expose the Metrics Endpoint for External Monitoring

The exporter's Prometheus metrics are only reachable inside the cluster by default. Expose them via an Ingress:

# event-exporter-ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: event-exporter-metrics
  namespace: monitoring
  annotations:
    nginx.ingress.kubernetes.io/auth-type: basic
    nginx.ingress.kubernetes.io/auth-secret: event-exporter-metrics-auth
spec:
  rules:
  - host: event-exporter.example.com
    http:
      paths:
      - path: /metrics
        pathType: Exact
        backend:
          service:
            name: kubernetes-event-exporter
            port:
              number: 2112
  tls:
  - hosts:
    - event-exporter.example.com
    secretName: event-exporter-tls

Apply and verify:

kubectl apply -f event-exporter-ingress.yaml
curl -u monitor:password https://event-exporter.example.com/metrics | grep event_exporter_sent

A healthy response returns current counters. A connection refused or empty response indicates the exporter process has stopped.

Step 3: Create a Vigilmon HTTP Monitor for the Metrics Endpoint

  1. Log in to Vigilmon → Add Monitor → HTTP.
  2. URL: https://event-exporter.example.com/metrics.
  3. Check interval: 2 minutes.
  4. Response timeout: 15 seconds.
  5. Expected status: 200.
  6. Keyword: event_exporter_sent_events_total (present whenever the exporter is running and connected).
  7. Label: kubernetes-event-exporter metrics.
  8. Click Save.

This monitor catches:

  • Exporter pod OOM kills or evictions that halt event forwarding
  • Kubernetes API watcher disconnects that cause the process to crash-loop
  • ConfigMap version mismatches after Helm upgrades that prevent the exporter from starting
  • Resource quota violations that prevent the pod from rescheduling
  • Kubernetes API server certificate rotations that break the watcher connection

Alert sensitivity: 2 consecutive failures before alerting to suppress transient failures during API server maintenance.

Step 4: Monitor Backend Connectivity

The exporter forwards events to one or more backends. Each backend must be reachable for events to flow. Check the configured backends in your configmap:

kubectl get configmap -n monitoring kubernetes-event-exporter -o jsonpath='{.data.config\.yaml}'

A typical configuration targeting Elasticsearch and a webhook:

# event-exporter-config.yaml
receivers:
  - name: elasticsearch
    elasticsearch:
      hosts:
        - https://elasticsearch.example.com:9200
      index: kube-events
      indexFormat: "kube-events-{2006-01-02}"
  - name: audit-webhook
    webhook:
      endpoint: https://audit-receiver.example.com/k8s-events
      headers:
        Authorization: "Bearer ${WEBHOOK_TOKEN}"

For each backend, create a Vigilmon HTTP monitor:

Elasticsearch backend:

  1. Add Monitor → HTTP.
  2. URL: https://elasticsearch.example.com:9200/_cluster/health.
  3. Expected status: 200.
  4. Keyword: green or yellow (both accept writes).
  5. Label: event-exporter elasticsearch backend.

Webhook backend:

  1. Add Monitor → HTTP.
  2. URL: https://audit-receiver.example.com/health.
  3. Expected status: 200.
  4. Label: event-exporter webhook backend.

Why separate backend monitors matter: The exporter retries failed deliveries in memory. If Elasticsearch is down for 10 minutes and your exporter memory limit is 256Mi, the retry buffer fills up and events are dropped permanently — with no Kubernetes event recording the loss.

Step 5: Detect Configuration Reload Failures

Kubernetes Event Exporter supports live configuration reload via a SIGHUP signal when running in reload mode. A failed reload (due to YAML syntax errors or missing secret references) reverts to the previous config silently. You can expose a config-status endpoint or use a canary strategy:

# Trigger a config reload and check exporter logs for reload confirmation
kubectl rollout restart deployment/kubernetes-event-exporter -n monitoring
kubectl rollout status deployment/kubernetes-event-exporter -n monitoring

# Check for recent config-related errors
kubectl logs -n monitoring deployment/kubernetes-event-exporter | grep -i "config\|reload\|error" | tail -20

Create a lightweight HTTP monitor that verifies the exporter is serving fresh metrics after a reload:

  1. In Vigilmon, Add Monitor → HTTP.
  2. URL: https://event-exporter.example.com/metrics.
  3. Keyword: event_exporter_config_reload_total (if your exporter version exposes this counter).
  4. Label: event-exporter config reload health.
  5. Click Save.

If the reload counter does not increment after a configuration change, the exporter is running with stale routing rules that may be missing new backends or namespace filters.

Step 6: Monitor SSL Certificates

The exporter connects to HTTPS backends using TLS. Certificate failures on any backend cause write errors that surface as event_exporter_failed_events_total increments — but with no specific indication that TLS is the cause:

# Check the certificate on your Elasticsearch backend
openssl s_client -connect elasticsearch.example.com:9200 2>/dev/null | \
  openssl x509 -noout -dates

# Check the webhook backend certificate
openssl s_client -connect audit-receiver.example.com:443 2>/dev/null | \
  openssl x509 -noout -dates

# Check your Ingress certificate for the metrics endpoint
openssl s_client -connect event-exporter.example.com:443 2>/dev/null | \
  openssl x509 -noout -dates

In Vigilmon, add an SSL Certificate monitor for each HTTPS endpoint:

  1. Add Monitor → SSL Certificate.
  2. Domain: elasticsearch.example.com.
  3. Alert when expiry is within: 30 days.
  4. Click Save.

Repeat for audit-receiver.example.com and event-exporter.example.com.

Step 7: Configure Alerting

In Vigilmon under Settings → Notifications, configure your alert channels:

| Monitor | Trigger | Immediate Action | |---|---|---| | Metrics endpoint | Non-200 or keyword missing | kubectl get pods -n monitoring -l app=kubernetes-event-exporter → check restarts; kubectl describe pod for events | | Elasticsearch backend | Non-green cluster health | Check ES cluster state; verify node count and shard allocation | | Webhook backend | Non-200 | Check receiver service logs; verify Kubernetes egress network policy | | SSL certificate | < 30 days to expiry | Renew backend certificate; check cert-manager pipeline for managed certs |

Escalation policy: Events pipeline failures are high-severity for compliance and audit teams. Route alerts to both the platform engineering team (for fix) and the security/compliance channel (for SLA tracking). Include the affected backend name in the alert message so responders know which pipeline segment failed.

Common Kubernetes Event Exporter Failure Modes and What Vigilmon Catches

| Scenario | Vigilmon monitor | |---|---| | Exporter pod OOM killed | Metrics endpoint returns connection error; alert within 2 min | | Kubernetes API server restart breaks watcher | Pod crash-loops; metrics endpoint down; event gap created | | Elasticsearch cluster goes red | Backend health monitor returns non-200; events being dropped | | Webhook receiver certificate expired | SSL monitor catches 30 days early; exporter logs TLS errors | | ConfigMap updated with YAML syntax error | Exporter fails to start after reload; metrics endpoint down | | Namespace added without updating exporter filter | Events from new namespace silently dropped; requires canary check | | Kafka topic deletion breaks exporter pipeline | Backend connectivity check catches topic-level failures | | Node pressure causes pod eviction | Metrics endpoint down; brief event gap during rescheduling |

Kubernetes events are your cluster's audit trail — pod restarts, scheduler decisions, OOM kills, and node evictions all flow through the event stream. When Kubernetes Event Exporter stops forwarding them, your SIEM has no cluster context, your alerting rules fire on silence, and your compliance team discovers the gap weeks later during an audit. Vigilmon closes this blind spot by independently checking the exporter's metrics endpoint and the backends it writes to, so your event pipeline is always verified from the outside and you know immediately when events stop flowing.

Start monitoring Kubernetes Event Exporter in under 5 minutes — register free at vigilmon.online.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles