tutorial

How to Monitor kubectl-whoami with Vigilmon

kubectl-whoami is a kubectl plugin that displays the authenticated Kubernetes user identity, group memberships, and extra attributes returned by the API serv...

kubectl-whoami is a kubectl plugin that displays the authenticated Kubernetes user identity, group memberships, and extra attributes returned by the API server for the currently configured kubeconfig context. SRE and DevOps teams use it to verify that service accounts, OIDC identities, and certificate-based credentials resolve to the expected user and groups before executing privileged operations — a critical step when debugging RBAC permission errors, auditing access during security incidents, or confirming that CI/CD pipelines are authenticating as the correct service account. When the Kubernetes API server's authentication and identity resolution mechanisms are degraded, kubectl-whoami returns errors or incorrect identity data that causes engineers to draw false conclusions about access configuration.

In this tutorial you'll set up uptime monitoring for the Kubernetes authentication infrastructure that kubectl-whoami depends on using Vigilmon — free tier, no credit card required.

Why kubectl-whoami's dependencies need external monitoring

kubectl-whoami calls the Kubernetes SelfSubjectAccessReview and TokenReview APIs to resolve the caller's identity. Its failure modes are tied directly to authentication and identity provider availability:

  • OIDC provider unavailability causes token validation failures — if your cluster uses OIDC for authentication (common with EKS, GKE, AKS, and on-premises setups using Dex or Keycloak), kubectl-whoami cannot resolve the user identity if the OIDC provider's token endpoint is unreachable, returning an authentication error rather than a meaningful identity
  • Stale kubeconfig credentials produce incorrect identity results — if a service account token or client certificate has expired or been revoked, kubectl-whoami returns a "forbidden" or "authentication failed" response rather than showing the expected identity, misleading engineers into thinking the API server is at fault rather than the credential
  • API server authentication module degradation returns wrong groups — Kubernetes API servers can have multiple authentication modules (client certificates, service account tokens, OIDC, webhook auth); if one module is degraded, the API may fall back to a less privileged or different identity than expected, and kubectl-whoami returns the fallback identity without indicating which authentication path was used
  • Webhook authentication backend unavailability prevents all logins — clusters using webhook-based authentication (TokenWebhook) cannot authenticate any request if the webhook backend is unreachable; kubectl-whoami returns "unable to connect to the server" rather than a clear identity resolution failure
  • SelfSubjectAccessReview RBAC restriction blocks identity checks — if the RBAC policy for the caller's identity does not include permission to create SelfSubjectAccessReview objects, kubectl-whoami fails with a permission error even though the authentication itself succeeded, creating a confusing signal during access debugging

External monitoring from Vigilmon watches the Kubernetes authentication infrastructure that kubectl-whoami depends on and alerts you when identity resolution is unreliable.

What you'll need

  • kubectl-whoami installed (kubectl whoami returns without error)
  • kubectl access to your monitored cluster
  • A node or LoadBalancer endpoint to host the health check
  • A free Vigilmon account

Step 1: Expose a health endpoint for kubectl-whoami's authentication dependencies

kubectl-whoami has no built-in HTTP interface. Deploy a health exporter that validates the Kubernetes authentication pipeline — API server reachability, token review, and identity resolution:

# kubectl-whoami-health.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: kubectl-whoami-health
  namespace: observability
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: kubectl-whoami-health-reader
rules:
  - apiGroups: ["authentication.k8s.io"]
    resources: ["selfsubjectreviews", "tokenreviews"]
    verbs: ["create"]
  - apiGroups: ["authorization.k8s.io"]
    resources: ["selfsubjectaccessreviews", "selfsubjectrulesreviews"]
    verbs: ["create"]
  - apiGroups: [""]
    resources: ["namespaces"]
    verbs: ["get", "list"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: kubectl-whoami-health-binding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: kubectl-whoami-health-reader
subjects:
  - kind: ServiceAccount
    name: kubectl-whoami-health
    namespace: observability
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: kubectl-whoami-health
  namespace: observability
spec:
  replicas: 1
  selector:
    matchLabels:
      app: kubectl-whoami-health
  template:
    metadata:
      labels:
        app: kubectl-whoami-health
    spec:
      serviceAccountName: kubectl-whoami-health
      containers:
        - name: checker
          image: bitnami/kubectl:latest
          command:
            - /bin/sh
            - -c
            - |
              while true; do
                # Test API server authentication endpoint
                if ! kubectl auth whoami --request-timeout=10s > /tmp/identity 2>&1; then
                  # Fall back to token review via API
                  TOKEN=$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)
                  REVIEW=$(kubectl create -f - --dry-run=server -o json 2>/dev/null <<EOF
    {
      "apiVersion": "authentication.k8s.io/v1",
      "kind": "TokenReview",
      "spec": {
        "token": "$TOKEN"
      }
    }
    EOF
                  )
                  if echo "$REVIEW" | grep -q '"authenticated": true'; then
                    echo "healthy auth=tokenreview identity_api=degraded" > /tmp/status
                  else
                    echo "unhealthy auth=failed api_server=authentication_error" > /tmp/status
                  fi
                  sleep 30
                  continue
                fi

                IDENTITY=$(cat /tmp/identity)
                USERNAME=$(echo "$IDENTITY" | grep -i "username" | head -1)
                echo "healthy auth=ok identity=$USERNAME" > /tmp/status
                sleep 60
              done
        - name: health-server
          image: busybox:latest
          command:
            - /bin/sh
            - -c
            - |
              while true; do
                STATUS=$(cat /tmp/status 2>/dev/null || echo "starting")
                if echo "$STATUS" | grep -q "^healthy"; then
                  printf "HTTP/1.1 200 OK\r\nContent-Type: text/plain\r\nContent-Length: ${#STATUS}\r\n\r\n$STATUS" | nc -l -p 8080 -q 1
                else
                  printf "HTTP/1.1 503 Service Unavailable\r\nContent-Type: text/plain\r\nContent-Length: ${#STATUS}\r\n\r\n$STATUS" | nc -l -p 8080 -q 1
                fi
              done
          ports:
            - containerPort: 8080
              name: health
          volumeMounts:
            - name: status
              mountPath: /tmp
      volumes:
        - name: status
          emptyDir: {}
---
apiVersion: v1
kind: Service
metadata:
  name: kubectl-whoami-health
  namespace: observability
spec:
  type: NodePort
  selector:
    app: kubectl-whoami-health
  ports:
    - name: health
      port: 8080
      targetPort: 8080
      nodePort: 30090

kubectl apply -f kubectl-whoami-health.yaml

# Verify
NODE_IP=$(kubectl get nodes -o jsonpath='{.items[0].status.addresses[?(@.type=="InternalIP")].address}')
curl http://$NODE_IP:30090/
# healthy auth=ok identity=system:serviceaccount:observability:kubectl-whoami-health

Step 2: Validate kubectl-whoami and authentication chain health

Before setting up monitoring, confirm kubectl-whoami resolves identity correctly across authentication paths:

#!/bin/bash
# check-kubectl-whoami.sh

echo "=== kubectl-whoami installation ==="
if kubectl whoami > /dev/null 2>&1 || kubectl auth whoami > /dev/null 2>&1; then
  echo "OK: kubectl-whoami installed and functional"
else
  echo "FAILED: kubectl-whoami not found — install via: kubectl krew install whoami"
  exit 1
fi

echo ""
echo "=== Identity resolution ==="
IDENTITY=$(kubectl whoami 2>/dev/null || kubectl auth whoami 2>/dev/null)
if [ -n "$IDENTITY" ]; then
  echo "OK: identity resolved:"
  echo "$IDENTITY"
else
  echo "FAILED: identity resolution returned empty — authentication may be broken"
fi

echo ""
echo "=== API server authentication endpoint ==="
if kubectl auth can-i create selfsubjectaccessreviews > /dev/null 2>&1; then
  echo "OK: SelfSubjectAccessReview API accessible"
else
  echo "WARNING: SelfSubjectAccessReview API inaccessible — kubectl-whoami may fail"
fi

echo ""
echo "=== Group membership resolution ==="
GROUPS=$(kubectl whoami --show-groups 2>/dev/null || kubectl auth whoami -o json 2>/dev/null | grep -i group)
if [ -n "$GROUPS" ]; then
  echo "OK: group memberships returned:"
  echo "$GROUPS"
else
  echo "WARNING: group memberships not returned — RBAC group policies may not apply correctly"
fi

echo ""
echo "=== OIDC provider check (if applicable) ==="
OIDC_ISSUER=$(kubectl get --raw /.well-known/openid-configuration 2>/dev/null | grep -o '"issuer":"[^"]*"' | head -1)
if [ -n "$OIDC_ISSUER" ]; then
  echo "OK: OIDC configuration endpoint accessible — $OIDC_ISSUER"
else
  echo "INFO: OIDC discovery endpoint not exposed (may be expected for non-OIDC clusters)"
fi

chmod +x check-kubectl-whoami.sh
./check-kubectl-whoami.sh

Step 3: Set up HTTP monitoring in Vigilmon

With the health endpoint running, add it to Vigilmon:

  1. Log in to vigilmon.online and go to Monitors → New Monitor
  2. Choose HTTP / HTTPS
  3. Add monitors for kubectl-whoami's authentication infrastructure:

| Monitor name | URL | Expected status | |---|---|---| | kubectl-whoami auth health | http://your-node-ip:30090/ | 200 | | Kubernetes API livez | https://your-api-server:6443/livez | 200 | | Kubernetes API readyz | https://your-api-server:6443/readyz | 200 | | OIDC provider discovery | https://your-oidc-provider/.well-known/openid-configuration | 200 |

  1. Set the check interval to 1 minute
  2. Under Expected response, match healthy in the response body for the health endpoint
  3. For clusters using OIDC, add the OIDC provider's well-known endpoint as a separate monitor — its unavailability will break all token-based identity resolution
  4. Save each monitor

Step 4: Add a TCP monitor for the Kubernetes API server

kubectl-whoami makes direct authentication calls to the Kubernetes API server. A TCP monitor catches network failures before the authentication layer:

  1. Go to Monitors → New Monitor
  2. Choose TCP Port
  3. Enter your Kubernetes API server hostname and port 6443
  4. Set the check interval to 1 minute
  5. Save the monitor

A TCP failure on port 6443 means kubectl-whoami cannot perform identity resolution for any authentication method, causing all identity checks to fail with connection timeout errors.

Step 5: Configure alert channels

kubectl-whoami is most critical during access debugging and security incident response. Identity resolution failures during an incident can severely delay root cause analysis when engineers cannot confirm whether they are operating under the correct permissions.

Email alerts

  1. Go to Alert Channels → Add Channel → Email
  2. Enter your SRE on-call and security engineering addresses
  3. Assign the channel to all kubectl-whoami monitors

Webhook alerts

  1. Go to Alert Channels → Add Channel → Webhook
  2. Enter your incident management webhook URL (PagerDuty, Opsgenie, Slack)
  3. Sample Vigilmon webhook payload:
{
  "monitor_name": "kubectl-whoami auth health",
  "status": "down",
  "url": "http://your-node-ip:30090/",
  "started_at": "2024-01-15T22:00:00Z",
  "duration_seconds": 120
}

Route this to a runbook that instructs on-call engineers to use kubectl auth can-i --list and direct inspection of ~/.kube/config credentials to determine the active identity while kubectl-whoami is unavailable.

Step 6: Correlate Vigilmon alerts with kubectl-whoami diagnostics

When you receive a kubectl-whoami authentication health downtime alert:

# 1. Check API server basic connectivity
kubectl cluster-info

# 2. Test identity resolution directly
kubectl whoami 2>&1
kubectl auth whoami 2>&1

# 3. Test the SelfSubjectAccessReview API directly
kubectl auth can-i create pods --namespace default

# 4. Check current kubeconfig context and credentials
kubectl config current-context
kubectl config view --minify

# 5. Verify token expiry for service account tokens
kubectl get secret -n observability -l kubernetes.io/service-account.name=kubectl-whoami-health -o yaml | \
  grep -A 5 "token:"

# 6. Check OIDC provider availability (if applicable)
OIDC_URL=$(kubectl config view --minify -o jsonpath='{.users[0].user.auth-provider.config.idp-issuer-url}')
curl -s "$OIDC_URL/.well-known/openid-configuration" | head -20

# 7. Check webhook authenticator health (if applicable)
kubectl get --raw /readyz/authentication

# 8. Inspect the health pod logs
kubectl logs -n observability -l app=kubectl-whoami-health -c checker --tail=20

# 9. Manual identity check via token inspection
TOKEN=$(kubectl config view --minify -o jsonpath='{.users[0].user.token}')
echo "$TOKEN" | cut -d. -f2 | base64 -d 2>/dev/null | python3 -m json.tool 2>/dev/null | grep -E '"sub"|"iss"|"aud"'

If kubectl auth whoami returns an error but kubectl get nodes succeeds, the identity resolution API (SelfSubjectReview) is degraded independently of the main API server — check the API server admission and authentication modules.

Step 7: Create a status page for Kubernetes authentication visibility

Make identity resolution infrastructure visible to the SRE and security teams:

  1. Go to Status Pages → New Status Page
  2. Name it: "Kubernetes Authentication (kubectl-whoami)"
  3. Add the kubectl-whoami health monitor, API livez/readyz monitors, and the OIDC provider monitor
  4. Share the URL with your SRE on-call and security engineering teams

During security incident response, engineers checking this page immediately know whether identity resolution is trustworthy or whether they must fall back to manual credential inspection.

Summary

| What you set up | What it catches | |---|---| | HTTP monitor on kubectl-whoami health endpoint | Authentication failures, identity API degradation | | HTTP monitor on API livez/readyz | API server degradation affecting all authentication | | HTTP monitor on OIDC provider discovery | OIDC token validation failures causing identity resolution errors | | TCP monitor on port 6443 | Network failures preventing all identity checks | | Email + webhook alert channels | Immediate notification when identity resolution is unreliable | | Status page | SRE and security team visibility into authentication infrastructure |

kubectl-whoami is most valuable in high-stakes moments — security incidents, access audits, and RBAC debugging — exactly when authentication infrastructure is most likely to be under stress. External monitoring ensures you know when identity resolution cannot be trusted before engineers make operational decisions based on incorrect or empty identity data.

Get started at vigilmon.online — free tier, no credit card required.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles