How to Monitor Kubetail with Vigilmon

Kubetail is a bash script that lets you tail logs from multiple Kubernetes pods simultaneously, color-coding each pod's output so you can track requests across a deployment in a single terminal window. In production it is routinely used during deployments, incident response, and canary rollouts to watch log streams from every replica at once. When the underlying infrastructure Kubetail depends on fails — the Kubernetes API server, RBAC permissions, or the kubeconfig credential chain — Kubetail silently stops producing output and engineers lose real-time log visibility without any warning.

In this tutorial you'll set up uptime monitoring for the Kubernetes API access layer that Kubetail depends on using Vigilmon — free tier, no credit card required.

Why Kubetail's dependencies need external monitoring

Kubetail itself is a thin bash wrapper around kubectl logs. Its failure modes all involve the infrastructure it calls rather than the script itself:

• Kubernetes API server unavailability — Kubetail cannot connect to any pod logs when the API server is down or unreachable from the ops workstation; there is no error surfaced by the tool itself, just an empty or stalled stream
• Expired kubeconfig credentials — OIDC tokens and short-lived certificates expire silently; Kubetail starts but returns authentication errors after the first log line, which is easy to miss in a busy terminal
• RBAC permission drift — when the service account or user binding for get pods and pods/log is removed, Kubetail fails on next run with a permissions error that is not forwarded to any alert channel
• Label selector returning no pods — if a deployment is renamed or migrated to a new namespace, the Kubetail invocation that engineers rely on during incidents returns nothing without indicating a configuration mismatch
• Kubernetes API health endpoint degradation — the /readyz and /livez endpoints degrade before log streaming fails; monitoring them provides early warning

External monitoring from Vigilmon watches the API server health endpoints your Kubetail workflow depends on and alerts you before log streaming becomes unavailable.

What you'll need

• A Kubernetes cluster with Kubetail installed or available as a script
• kubectl access with permissions to read pods and pod logs in your target namespaces
• An exposed Kubernetes API health endpoint or a custom status exporter (see Step 1)
• A free Vigilmon account

Step 1: Expose a health endpoint for Kubetail's dependencies

Kubetail itself does not expose HTTP endpoints. The practical approach is to create a lightweight CronJob that runs a Kubetail-equivalent connectivity check and writes results to a small HTTP status server.

# kubetail-health.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: kubetail-health
  namespace: observability
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: kubetail-health-reader
rules:
  - apiGroups: [""]
    resources: ["pods", "pods/log", "namespaces"]
    verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: kubetail-health-binding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: kubetail-health-reader
subjects:
  - kind: ServiceAccount
    name: kubetail-health
    namespace: observability
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: kubetail-health
  namespace: observability
spec:
  replicas: 1
  selector:
    matchLabels:
      app: kubetail-health
  template:
    metadata:
      labels:
        app: kubetail-health
    spec:
      serviceAccountName: kubetail-health
      containers:
        - name: checker
          image: bitnami/kubectl:latest
          command:
            - /bin/sh
            - -c
            - |
              while true; do
                # Test pod listing (Kubetail's first operation)
                if kubectl get pods --all-namespaces -o name > /tmp/pods 2>&1; then
                  POD_COUNT=$(wc -l < /tmp/pods)
                  # Test log access on the first pod found
                  FIRST_POD=$(head -1 /tmp/pods | cut -d/ -f2)
                  FIRST_NS=$(kubectl get pods --all-namespaces -o jsonpath='{.items[0].metadata.namespace}' 2>/dev/null)
                  if kubectl logs -n "$FIRST_NS" "$FIRST_POD" --tail=1 > /dev/null 2>&1; then
                    echo "healthy pod_count=$POD_COUNT" > /tmp/status
                  else
                    echo "degraded pod_listing=ok log_access=failed" > /tmp/status
                  fi
                else
                  echo "unhealthy api_unreachable=true" > /tmp/status
                fi
                sleep 30
              done
        - name: health-server
          image: busybox:latest
          command:
            - /bin/sh
            - -c
            - |
              while true; do
                STATUS=$(cat /tmp/status 2>/dev/null || echo "starting")
                if echo "$STATUS" | grep -q "^healthy"; then
                  printf "HTTP/1.1 200 OK\r\nContent-Type: text/plain\r\nContent-Length: ${#STATUS}\r\n\r\n$STATUS" | nc -l -p 8080 -q 1
                elif echo "$STATUS" | grep -q "^degraded"; then
                  printf "HTTP/1.1 200 OK\r\nContent-Type: text/plain\r\nContent-Length: ${#STATUS}\r\n\r\n$STATUS" | nc -l -p 8080 -q 1
                else
                  printf "HTTP/1.1 503 Service Unavailable\r\nContent-Type: text/plain\r\nContent-Length: ${#STATUS}\r\n\r\n$STATUS" | nc -l -p 8080 -q 1
                fi
              done
          ports:
            - containerPort: 8080
              name: health
          volumeMounts:
            - name: status
              mountPath: /tmp
      volumes:
        - name: status
          emptyDir: {}
---
apiVersion: v1
kind: Service
metadata:
  name: kubetail-health
  namespace: observability
spec:
  type: NodePort
  selector:
    app: kubetail-health
  ports:
    - name: health
      port: 8080
      targetPort: 8080
      nodePort: 30081

kubectl apply -f kubetail-health.yaml

# Verify the health endpoint
NODE_IP=$(kubectl get nodes -o jsonpath='{.items[0].status.addresses[?(@.type=="InternalIP")].address}')
curl http://$NODE_IP:30081/
# healthy pod_count=42

Step 2: Verify Kubetail API access across namespaces

Before setting up monitoring, confirm that the log access permissions Kubetail requires are intact for your target namespaces:

#!/bin/bash
# check-kubetail-access.sh
NAMESPACES=$(kubectl get namespaces -o jsonpath='{.items[*].metadata.name}')

for NS in $NAMESPACES; do
  # Check pod listing
  if ! kubectl auth can-i list pods -n "$NS" > /dev/null 2>&1; then
    echo "MISSING list pods: $NS"
    continue
  fi
  # Check log access
  if ! kubectl auth can-i get pods/log -n "$NS" > /dev/null 2>&1; then
    echo "MISSING get pods/log: $NS"
    continue
  fi
  echo "OK: $NS"
done

chmod +x check-kubetail-access.sh
./check-kubetail-access.sh

This surfaces RBAC gaps that would cause Kubetail to fail silently in specific namespaces.

Step 3: Set up HTTP monitoring in Vigilmon

With the health endpoint running, add it to Vigilmon:

1. Log in to vigilmon.online and go to Monitors → New Monitor
2. Choose HTTP / HTTPS
3. Add monitors for your endpoints:

| Monitor name | URL | Expected status | |---|---|---| | Kubetail API health | http://your-node-ip:30081/ | 200 | | Kubernetes API server livez | https://your-api-server:6443/livez | 200 | | Kubernetes API server readyz | https://your-api-server:6443/readyz | 200 |

4. Set the check interval to 1 minute
5. Under Expected response, set status code 200 and optionally match healthy in the response body
6. For the Kubernetes API endpoints, add the CA certificate under TLS settings if you use a private CA
7. Save each monitor

Step 4: Monitor the Kubetail health TCP port

Add a TCP monitor to detect networking issues that would prevent log streaming before the HTTP check catches them:

1. Go to Monitors → New Monitor
2. Choose TCP Port
3. Enter your node IP or Ingress hostname and port 30081
4. Save the monitor

A TCP failure here while the pod appears Running typically indicates a NodePort firewall rule was removed or a network policy was applied that blocks external health checks.

Step 5: Configure alert channels

When Kubetail's log access layer fails, on-call engineers lose their primary real-time debugging tool. Alert immediately so they can switch to an alternative before an incident investigation stalls.

Email alerts

1. Go to Alert Channels → Add Channel → Email
2. Enter your SRE on-call address
3. Assign the channel to all Kubetail monitors

Webhook alerts for incident routing

1. Go to Alert Channels → Add Channel → Webhook
2. Enter your incident management webhook URL (PagerDuty, Opsgenie, Slack)
3. The payload Vigilmon sends:

{
  "monitor_name": "Kubetail API health",
  "status": "down",
  "url": "http://your-node-ip:30081/",
  "started_at": "2024-01-15T10:23:00Z",
  "duration_seconds": 120
}

Route this alert to a runbook that checks API server reachability and verifies that RBAC bindings for log access are still present.

Step 6: Correlate Vigilmon alerts with Kubetail diagnostics

When you receive a Kubetail health downtime alert, run these checks:

# 1. Verify Kubernetes API server is reachable
kubectl cluster-info

# 2. Check that pod listing works
kubectl get pods --all-namespaces | head -20

# 3. Test log access directly (the operation Kubetail performs)
FIRST_POD=$(kubectl get pods --all-namespaces -o jsonpath='{.items[0].metadata.name}')
FIRST_NS=$(kubectl get pods --all-namespaces -o jsonpath='{.items[0].metadata.namespace}')
kubectl logs -n "$FIRST_NS" "$FIRST_POD" --tail=5

# 4. Check RBAC for the service account Kubetail uses
kubectl auth can-i list pods --all-namespaces
kubectl auth can-i get pods/log --all-namespaces

# 5. Inspect the health checker pod logs
kubectl logs -n observability -l app=kubetail-health -c checker --tail=20

# 6. Check the kubeconfig credential expiry (for ops workstations)
kubectl config view --minify -o jsonpath='{.users[0].user}'

# 7. Verify the health deployment is running
kubectl get deployment kubetail-health -n observability

If the health endpoint returns 503, the most common causes are an expired service account token (create a new secret or rotate the token) or a network policy that was added to the observability namespace blocking outbound API calls.

Step 7: Create a status page for log streaming infrastructure

During an active incident, Kubetail access is critical. Make its status visible to the entire engineering team:

1. Go to Status Pages → New Status Page
2. Name it: "Log Streaming Infrastructure"
3. Add your monitors: Kubetail API health, Kubernetes API server livez, Kubernetes API server readyz
4. Share the URL with SRE and platform engineering teams

This lets on-call engineers confirm log streaming availability without needing cluster access themselves.

Summary

| What you set up | What it catches | |---|---| | HTTP monitor on health endpoint | API server unreachability, RBAC permission loss, log access failures | | HTTP monitor on Kubernetes livez/readyz | API server degradation before log streaming fails | | TCP monitor on health port | Networking issues blocking Kubetail connectivity | | Email + webhook alert channels | Immediate notification when log streaming infrastructure degrades | | Status page | Team visibility into real-time log access availability |

Kubetail is a deceptively simple script that engineers rely on during their most stressful moments. External monitoring is the only way to know the infrastructure it depends on has failed before you try to use it during an incident.

Get started at vigilmon.online — free tier, no credit card required.