How to Monitor KubeShark with Vigilmon (HTTP + TCP + Alerts)

KubeShark is a Kubernetes API traffic analyzer and network sniffer that deploys a DaemonSet across your cluster nodes, capturing all inter-pod traffic at the kernel level using eBPF or AF_PACKET and decoding protocols including HTTP/1, HTTP/2, gRPC, Kafka, AMQP, and Redis. Its central hub pod aggregates traffic streams from worker pods on every node and serves a real-time web UI on port 8080. When the hub goes down — whether from an OOM kill under heavy traffic, a kernel compatibility issue with eBPF probes, or network policies blocking hub-to-worker communication — your team loses the ability to diagnose live network-level issues in the cluster entirely. Vigilmon's free tier gives you external HTTP and TCP monitoring for the KubeShark hub so you know the moment it becomes unavailable, before your engineers start chasing ghosts.

Why KubeShark needs external monitoring

• Hub pod OOMKilled in large clusters: In clusters with high inter-pod traffic volume, the hub pod must aggregate and buffer streams from dozens of worker pods simultaneously. Under sustained load this can exhaust the hub's memory limit, triggering an OOMKilled restart. During the restart window — which can take 30–90 seconds — all live traffic capture is lost with no visible warning to users browsing the UI.

• eBPF probe fails on kernel version mismatch: KubeShark's worker pods attach eBPF programs to the host kernel to intercept packets. If a node runs a kernel version that the probe was not compiled for — common after node image upgrades or mixed node pools — the probe silently fails to load. Traffic from that node never reaches the hub, creating blind spots that look like normal operation until someone notices missing flows.

• Hub loses connection to worker pods on specific nodes: The hub maintains persistent connections to each worker pod to stream captured traffic. If a worker pod restarts, a node becomes temporarily unreachable, or network policies are tightened, the hub may lose that worker's stream without surfacing an error in the UI. Traffic from affected nodes disappears silently while the hub continues to appear healthy.

• Hub storage fills up causing capture drops: KubeShark buffers recent traffic in its hub pod's ephemeral storage. On clusters processing high request rates, this buffer can fill faster than it is consumed, causing the hub to drop capture frames. The UI remains accessible and appears functional while actually returning incomplete or stale data — a failure mode that external monitoring of response integrity can detect.

• Network policies blocking hub-to-worker communication: Clusters with strict NetworkPolicy rules may inadvertently block traffic between the hub pod and the worker DaemonSet pods after a policy change or namespace migration. The hub pod stays running and serves the web UI, but all traffic streams from workers are cut. Kubernetes liveness probes report the hub as healthy while it captures nothing at all.

What you'll need

• KubeShark installed in a Kubernetes cluster, started with kubeshark tap and running in the kubeshark namespace with at least the hub pod and worker DaemonSet active
• The KubeShark hub endpoint accessible externally, either via kubectl port-forward for ad-hoc checks or a stable Ingress hostname for production monitoring
• A free Vigilmon account — no credit card required, monitors start running in under a minute

Step 1: Expose KubeShark's health endpoint

First, confirm that the KubeShark hub pod is running and identify which port it is listening on.

# Check the hub pod in the kubeshark namespace
kubectl get pods -n kubeshark

# Expected output similar to:
# NAME                             READY   STATUS    RESTARTS   AGE
# kubeshark-hub-7d9f8b6c4-xkp2m    1/1     Running   0          12m
# kubeshark-worker-daemonset-4jvnq  1/1     Running   0          12m
# kubeshark-worker-daemonset-9r2ks  1/1     Running   0          12m
# kubeshark-worker-daemonset-bwlzt  1/1     Running   0          12m

# Check the hub service and its port
kubectl get svc -n kubeshark

# Expected output:
# NAME             TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE
# kubeshark-hub    ClusterIP   10.96.143.201   <none>        8080/TCP   12m

# Test the hub health endpoint via port-forward
kubectl port-forward -n kubeshark svc/kubeshark-hub 8080:8080 &
curl http://localhost:8080/echo
# Expected output: "KubeShark" or a JSON health response

# Check DaemonSet worker status across all nodes
kubectl get daemonset -n kubeshark
# NAME                         DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE
# kubeshark-worker-daemonset   3         3         3       3            3

For production monitoring you need the hub reachable without a port-forward. Create an Ingress to expose it externally:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: kubeshark-hub-ingress
  namespace: kubeshark
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /
spec:
  ingressClassName: nginx
  rules:
    - host: kubeshark.example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: kubeshark-hub
                port:
                  number: 8080

Apply it with kubectl apply -f kubeshark-ingress.yaml. Once DNS resolves, confirm the health endpoint is reachable:

curl http://kubeshark.example.com/echo
# Expected: "KubeShark"

Step 2: Set up HTTP monitoring in Vigilmon

With the hub endpoint accessible, add an HTTP monitor in Vigilmon that polls the /echo health endpoint continuously.

1. Log in to Vigilmon and click Add Monitor from your dashboard.
2. Select HTTP as the monitor type.
3. Set the URL to http://kubeshark.example.com/echo (or https:// if your Ingress terminates TLS).
4. Set the check interval to 60 seconds for standard coverage, or 30 seconds if continuous capture availability is critical to your team.
5. Under Expected response, confirm the keyword match is set to KubeShark so Vigilmon validates the body and not just the HTTP 200 status code. Click Save.

Monitoring the DaemonSet agent endpoint

If your cluster exposes individual worker node metrics or status pages (for example via a NodePort service for debugging), you can add a second HTTP monitor pointing at a worker endpoint:

1. Add a second monitor of type HTTP targeting http://<node-ip>:<worker-nodeport>/echo on one or more representative nodes.
2. Set the expected response keyword to match the worker's health response.
3. Label this monitor clearly (for example, KubeShark Worker – Node A) so alerts identify which node the failure came from.

Why external monitoring catches what Kubernetes misses

Kubernetes liveness and readiness probes check whether the hub process is alive and accepting connections — not whether it is actually capturing traffic. A hub pod that has lost all worker connections, that is dropping frames due to storage exhaustion, or whose eBPF probes have silently failed on some nodes will still pass its liveness probe and report Running in kubectl get pods. Vigilmon's external HTTP monitor validates the hub's /echo response from outside the cluster, catching TLS termination failures, Ingress misconfigurations, and hub process hangs that internal probes never surface.

Step 3: Monitor KubeShark's TCP port

An HTTP health check confirms the application layer is responding, but a TCP-layer monitor catches lower-level failures — such as the hub process crashing outright, the Ingress controller becoming unreachable, or a firewall rule change blocking port 8080 — before an HTTP check can even attempt a connection.

1. In Vigilmon, click Add Monitor and select TCP.
2. Set the host to kubeshark.example.com (or the raw cluster IP if monitoring internally).
3. Set the port to 8080 if monitoring the hub directly, or 443 if your Ingress terminates TLS.
4. Set the check interval to 60 seconds.
5. Click Save.

This TCP monitor acts as a first line of detection: if it fires, the hub or Ingress is entirely unreachable and the HTTP monitor will follow within the same check cycle.

Step 4: Configure alert channels

Email alerts

1. In Vigilmon, navigate to Alert Channels and click Add Channel.
2. Select Email and enter the address of the person or team responsible for cluster network observability.
3. Assign this channel to both the HTTP monitor and the TCP monitor created in steps 2 and 3.
4. Set the alert threshold to trigger after 2 consecutive failures to avoid noise from transient network blips.

Webhook alerts

For integration with Slack, PagerDuty, or a custom incident handler, add a webhook channel:

1. In Vigilmon, click Add Channel and select Webhook.
2. Paste your endpoint URL (for example, a Slack incoming webhook or PagerDuty Events API v2 URL).
3. Vigilmon will POST a JSON payload on each alert event. A representative payload looks like:

{
  "monitor_name": "KubeShark Hub – HTTP",
  "monitor_type": "http",
  "status": "down",
  "url": "http://kubeshark.example.com/echo",
  "checked_at": "2026-07-11T14:32:00Z",
  "reason": "Connection refused",
  "previous_status": "up",
  "downtime_started_at": "2026-07-11T14:32:00Z"
}

4. Assign this channel to both monitors and click Save.

Diagnostic runbook

When a KubeShark alert fires, run the following commands in order to isolate the cause:

# 1. Check hub pod status — look for OOMKilled or CrashLoopBackOff
kubectl get pod -n kubeshark -l app=kubeshark-hub -o wide

# 2. Inspect hub pod events and resource usage
kubectl describe pod -n kubeshark -l app=kubeshark-hub
kubectl top pod -n kubeshark

# 3. Check all DaemonSet worker pods across nodes
kubectl get pods -n kubeshark -l app=kubeshark-worker -o wide

# 4. Check for eBPF probe errors in worker logs (kernel mismatch shows here)
kubectl logs -n kubeshark -l app=kubeshark-worker --tail=50 | grep -i "ebpf\|probe\|error\|failed"

# 5. Check hub logs for worker connection drops or storage pressure
kubectl logs -n kubeshark -l app=kubeshark-hub --tail=100 | grep -i "worker\|disconnect\|storage\|drop"

# 6. Verify NetworkPolicy is not blocking hub-to-worker traffic
kubectl get networkpolicy -n kubeshark
kubectl describe networkpolicy -n kubeshark

# 7. Check hub ephemeral storage usage
kubectl exec -n kubeshark deployment/kubeshark-hub -- df -h /tmp

# 8. Restart the hub if it is stuck (workers reconnect automatically)
kubectl rollout restart deployment/kubeshark-hub -n kubeshark

# 9. Full reset if workers are in a broken probe state
kubeshark clean && kubeshark tap

Step 5: Create a public status page

A public status page lets your cluster operators and application teams check KubeShark availability without logging into Vigilmon or the cluster.

1. In Vigilmon, navigate to Status Pages and click Create Status Page.
2. Give it a name such as KubeShark Network Observability and choose a public URL slug like kubeshark-status.
3. Add the monitors created in steps 2 and 3 — the HTTP health check and the TCP port monitor — to this status page.
4. Optionally add a description such as: "Real-time availability of KubeShark's traffic capture hub. Downtime on this page means live network-level traffic visibility in the cluster is degraded."
5. Click Publish. Share the URL with your platform engineering and SRE teams so anyone can check KubeShark health without cluster access.

Putting it all together

| Monitor | Type | What it catches | |---|---|---| | KubeShark Hub – HTTP /echo | HTTP | Hub process down, OOMKilled restart, Ingress misconfiguration, hub returning unexpected content due to storage or worker loss | | KubeShark Hub – TCP 8080 | TCP | Total hub process crash, firewall blocking port 8080, Ingress controller failure, node-level network partition | | KubeShark Worker – HTTP (sample node) | HTTP | Worker pod crash on a specific node, eBPF probe failure preventing traffic capture on that node | | KubeShark Hub TLS – TCP 443 | TCP | TLS termination failure at the Ingress, certificate expiry breaking external access to the hub UI |

# Key diagnostic commands — run these when Vigilmon fires an alert

# Overall cluster state for KubeShark
kubectl get all -n kubeshark

# Hub pod resource consumption (OOM risk)
kubectl top pod -n kubeshark

# Worker DaemonSet coverage — DESIRED should equal number of schedulable nodes
kubectl get daemonset kubeshark-worker-daemonset -n kubeshark

# eBPF probe status per worker node
kubectl logs -n kubeshark -l app=kubeshark-worker --prefix | grep -i "ebpf\|attach\|failed"

# Hub storage pressure
kubectl exec -n kubeshark deploy/kubeshark-hub -- df -h

# Network policy audit
kubectl get networkpolicy -A | grep kubeshark

# Full reset when probes are broken
kubeshark clean && kubeshark tap

What's next

• Monitor KubeShark's eBPF probe health per node by scraping worker pod logs or metrics and alerting when a probe fails to attach on a kernel version mismatch — this catches silent capture gaps on individual nodes before they affect investigations
• Add alerts when traffic capture volume drops unexpectedly below a baseline threshold, distinguishing between a genuine quiet period in the cluster and a capture failure where packets are being missed without error
• Integrate with Prometheus to track KubeShark capture statistics over time — hub and worker pods can expose metrics that feed dashboards for captured frame counts, dropped frames, and active worker connections, giving you trending data to catch degradation before it becomes an outage

Get started free at vigilmon.online — no credit card, monitors start running in under a minute.