How to Monitor kubectx with Vigilmon

kubectx is a command-line utility that lets engineers switch between Kubernetes contexts instantly — replacing the verbose kubectl config use-context workflow with a single short command. In multi-cluster environments it is the primary mechanism developers use to direct kubectl at staging, production, or regional clusters. When the kubeconfig file kubectx reads becomes corrupted, the context list grows stale, or the cluster credentials embedded in each context expire, kubectx either silently uses the wrong cluster or exits with a cryptic error — both outcomes are dangerous when an engineer believes they are working against staging but are actually pointed at production.

In this tutorial you'll set up uptime monitoring for the Kubernetes API layer that kubectx depends on using Vigilmon — free tier, no credit card required.

Why kubectx's dependencies need external monitoring

kubectx is a thin shell script that wraps kubeconfig parsing and kubectl config use-context. Its failure modes are infrastructure-level rather than tool-level:

• Expired cluster credentials — client certificates, OIDC tokens, and service account tokens embedded in kubeconfig contexts expire silently; kubectx switches the context but the next kubectl command against that context fails with an authentication error, with no warning from kubectx itself
• API server unreachability — if a cluster's API server goes down, kubectx still lists and switches to its context successfully because it never contacts the server; engineers discover the cluster is unreachable only after switching and running a subsequent command
• Kubeconfig file corruption — a partial write to ~/.kube/config (from a concurrent kubectl config set-context) renders the YAML unparseable; kubectx exits with a YAML parse error and all context switching is blocked
• Context list drift — after a cluster is decommissioned, its stale context remains in kubeconfig; switching to it produces authentication failures that look like intermittent outages rather than a config problem
• Missing kubeconfig permissions — on shared workstations or CI runners, file permission changes on ~/.kube/config prevent kubectx from reading the context list

External monitoring from Vigilmon watches the API endpoints for each cluster context and alerts you before engineers switch to a broken context during an incident.

What you'll need

• A multi-cluster Kubernetes environment with kubectx installed
• kubectl access to your target clusters
• A node or endpoint that can expose a small HTTP health check per cluster
• A free Vigilmon account

Step 1: Expose a health endpoint per cluster context

kubectx itself exposes no HTTP interface. The practical approach is to deploy a lightweight status endpoint in each cluster that verifies its own API server is reachable and that credentials are valid.

# kubectx-health.yaml — deploy this into each cluster you want to monitor
apiVersion: v1
kind: ServiceAccount
metadata:
  name: kubectx-health
  namespace: observability
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: kubectx-health-reader
rules:
  - apiGroups: [""]
    resources: ["namespaces", "nodes"]
    verbs: ["get", "list"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: kubectx-health-binding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: kubectx-health-reader
subjects:
  - kind: ServiceAccount
    name: kubectx-health
    namespace: observability
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: kubectx-health
  namespace: observability
spec:
  replicas: 1
  selector:
    matchLabels:
      app: kubectx-health
  template:
    metadata:
      labels:
        app: kubectx-health
    spec:
      serviceAccountName: kubectx-health
      containers:
        - name: checker
          image: bitnami/kubectl:latest
          command:
            - /bin/sh
            - -c
            - |
              while true; do
                # Verify API server is reachable and credentials are valid
                if kubectl get namespaces -o name > /tmp/ns_list 2>/tmp/err; then
                  NS_COUNT=$(wc -l < /tmp/ns_list)
                  NODE_COUNT=$(kubectl get nodes --no-headers 2>/dev/null | wc -l)
                  echo "healthy context=$(kubectl config current-context 2>/dev/null || echo unknown) namespaces=$NS_COUNT nodes=$NODE_COUNT" > /tmp/status
                else
                  ERR=$(cat /tmp/err | head -1)
                  echo "unhealthy error=$ERR" > /tmp/status
                fi
                sleep 30
              done
        - name: health-server
          image: busybox:latest
          command:
            - /bin/sh
            - -c
            - |
              while true; do
                STATUS=$(cat /tmp/status 2>/dev/null || echo "starting")
                if echo "$STATUS" | grep -q "^healthy"; then
                  printf "HTTP/1.1 200 OK\r\nContent-Type: text/plain\r\nContent-Length: ${#STATUS}\r\n\r\n$STATUS" | nc -l -p 8080 -q 1
                else
                  printf "HTTP/1.1 503 Service Unavailable\r\nContent-Type: text/plain\r\nContent-Length: ${#STATUS}\r\n\r\n$STATUS" | nc -l -p 8080 -q 1
                fi
              done
          ports:
            - containerPort: 8080
              name: health
          volumeMounts:
            - name: status
              mountPath: /tmp
      volumes:
        - name: status
          emptyDir: {}
---
apiVersion: v1
kind: Service
metadata:
  name: kubectx-health
  namespace: observability
spec:
  type: NodePort
  selector:
    app: kubectx-health
  ports:
    - name: health
      port: 8080
      targetPort: 8080
      nodePort: 30082

Deploy this manifest into each cluster context you want to monitor:

# Switch to each context and deploy
for CTX in $(kubectx); do
  kubectx "$CTX"
  kubectl apply -f kubectx-health.yaml
  NODE_IP=$(kubectl get nodes -o jsonpath='{.items[0].status.addresses[?(@.type=="InternalIP")].address}')
  echo "Context $CTX: http://$NODE_IP:30082/"
done

Step 2: Validate credential health for each context

Before setting up monitoring, confirm that each context in your kubeconfig has valid credentials:

#!/bin/bash
# check-kubectx-credentials.sh
CURRENT=$(kubectx --current)

for CTX in $(kubectx); do
  kubectx "$CTX" > /dev/null 2>&1
  if kubectl get nodes --request-timeout=5s > /dev/null 2>&1; then
    echo "OK: $CTX"
  else
    echo "FAILED: $CTX"
  fi
done

# Restore original context
kubectx "$CURRENT" > /dev/null 2>&1

chmod +x check-kubectx-credentials.sh
./check-kubectx-credentials.sh

This script surfaces expired credentials and unreachable API servers per context before an engineer switches to them during an incident.

Step 3: Set up HTTP monitoring in Vigilmon

With the per-cluster health endpoints running, add each one to Vigilmon:

1. Log in to vigilmon.online and go to Monitors → New Monitor
2. Choose HTTP / HTTPS
3. Create one monitor per cluster context:

| Monitor name | URL | Expected status | |---|---|---| | kubectx — production cluster | http://prod-node-ip:30082/ | 200 | | kubectx — staging cluster | http://staging-node-ip:30082/ | 200 | | kubectx — dev cluster | http://dev-node-ip:30082/ | 200 | | Kubernetes API livez (prod) | https://prod-api-server:6443/livez | 200 | | Kubernetes API livez (staging) | https://staging-api-server:6443/readyz | 200 |

4. Set the check interval to 1 minute
5. Under Expected response, set status code 200 and optionally match healthy in the body
6. For API server endpoints using a private CA, add the CA certificate under TLS settings
7. Save each monitor

Step 4: Add TCP monitors for API server ports

Add TCP monitors to detect network-level failures before credential checks even run:

1. Go to Monitors → New Monitor
2. Choose TCP Port
3. Add the Kubernetes API server port (6443) for each cluster:

| Monitor name | Host | Port | |---|---|---| | Kubernetes API TCP (prod) | prod-api-server | 6443 | | Kubernetes API TCP (staging) | staging-api-server | 6443 |

4. Save each monitor

A TCP failure on port 6443 while the cluster's other services are reachable typically means a firewall rule or security group was changed, not a cluster-level outage.

Step 5: Configure alert channels

Context-switching failures during an incident can cause engineers to run commands against the wrong cluster. Alert immediately so the team can verify which context is broken before executing any destructive operations.

Email alerts

1. Go to Alert Channels → Add Channel → Email
2. Enter your SRE on-call address and any platform engineering addresses
3. Assign the channel to all kubectx cluster monitors

Webhook alerts

1. Go to Alert Channels → Add Channel → Webhook
2. Enter your incident management webhook URL (PagerDuty, Opsgenie, Slack)
3. The payload Vigilmon sends:

{
  "monitor_name": "kubectx — production cluster",
  "status": "down",
  "url": "http://prod-node-ip:30082/",
  "started_at": "2024-01-15T14:30:00Z",
  "duration_seconds": 90
}

Route this to a runbook that instructs on-call engineers to avoid switching to the affected context and to verify their current context before executing any cluster commands.

Step 6: Correlate Vigilmon alerts with kubectx diagnostics

When you receive a cluster health downtime alert, run these checks:

# 1. List all available contexts and their clusters
kubectx
kubectl config get-contexts

# 2. Check which context is currently active
kubectx --current

# 3. Test connectivity to the failing cluster directly
kubectl cluster-info --context=<failing-context>

# 4. Verify credential expiry for the failing context
kubectl config view --context=<failing-context> --minify \
  -o jsonpath='{.users[0].user}'

# 5. Check API server reachability
curl -k https://<cluster-api-server>:6443/livez

# 6. Inspect the health checker pod in the cluster
kubectl logs -n observability -l app=kubectx-health -c checker --tail=20

# 7. Check if the kubeconfig was recently modified
ls -la ~/.kube/config
kubectl config view --flatten > /tmp/kube-backup.yaml

If the health endpoint returns 503 with error=Unauthorized, the cluster credentials for that context have expired. Rotate the certificate or refresh the OIDC token and update the kubeconfig entry.

Step 7: Create a status page for cluster contexts

Give your entire engineering team visibility into which cluster contexts are healthy before they switch:

1. Go to Status Pages → New Status Page
2. Name it: "Kubernetes Cluster Context Health"
3. Add all your cluster monitors — one row per cluster context
4. Share the URL with engineering and SRE teams

Engineers can check this page before switching contexts during an incident, preventing operations against a broken or wrong cluster.

Summary

| What you set up | What it catches | |---|---| | HTTP monitor per cluster (health endpoint) | Expired credentials, API server unreachability, auth failures | | HTTP monitor on livez/readyz | API server degradation before context switch failures | | TCP monitor on port 6443 | Firewall and network path failures to the API server | | Email + webhook alert channels | Immediate notification when a cluster context becomes unusable | | Status page | Team-wide visibility into context health before switching |

kubectx is the gateway to every cluster operation your team runs. External monitoring ensures you know which contexts are broken before engineers switch to them — and before they run commands against the wrong cluster.

Get started at vigilmon.online — free tier, no credit card required.