tutorial

How to Monitor kubelogin with Vigilmon

kubelogin is a kubectl plugin for Kubernetes OpenID Connect authentication, enabling OIDC-based login flows that allow engineers to authenticate to clusters ...

kubelogin is a kubectl plugin for Kubernetes OpenID Connect authentication, enabling OIDC-based login flows that allow engineers to authenticate to clusters using corporate identity providers such as Okta, Azure AD, Keycloak, and Dex. In organizations that replaced static kubeconfig certificates with OIDC-based authentication, kubelogin is the bridge between the identity provider and the cluster — and any degradation in the OIDC discovery endpoints, token issuer, or kubelogin's credential cache invalidates every engineer's cluster access simultaneously, transforming an identity provider outage into a full Kubernetes access outage.

In this tutorial you'll set up uptime monitoring for the OIDC infrastructure that kubelogin depends on using Vigilmon — free tier, no credit card required.

Why kubelogin's dependencies need external monitoring

kubelogin coordinates between three systems: the Kubernetes API server, the OIDC identity provider, and the local credential cache. Its failure modes cascade across all three:

  • OIDC discovery endpoint unavailable — kubelogin fetches /.well-known/openid-configuration from the identity provider to discover token endpoints; if this endpoint returns an error or times out, kubelogin cannot initiate the login flow and every engineer loses cluster access simultaneously
  • Token endpoint degraded — after discovery, kubelogin exchanges authorization codes or refresh tokens at the identity provider's token endpoint; if this endpoint is slow or returns 5xx errors, engineers receive authentication errors from kubectl even though their credentials are valid
  • Expired refresh tokens not detected — kubelogin caches access and refresh tokens locally; when the identity provider rotates signing keys or shortens refresh token TTLs, cached tokens become invalid and kubelogin silently fails with "token expired" errors that engineers misinterpret as a Kubernetes problem rather than an identity provider issue
  • OIDC issuer URL mismatch after IdP migration — if the organization migrates to a new identity provider or changes the issuer URL without updating kubeconfig, kubelogin fails at the token validation step because the issuer in the token does not match the configured issuer, producing confusing "iss claim" errors
  • Corporate VPN required but not enforced — many OIDC providers are accessible only on VPN; if an engineer's VPN drops mid-session, kubelogin cannot refresh expired tokens and cluster access fails without a clear connection between the VPN state and the authentication failure
  • Kubernetes API server OIDC configuration mismatch — the API server must be configured with the same OIDC issuer URL as kubelogin's kubeconfig; if these drift after a cluster upgrade, token validation fails at the API server even when the identity provider is healthy

External monitoring from Vigilmon watches the OIDC identity provider endpoints and Kubernetes API server that kubelogin depends on, alerting you before widespread authentication failures affect your engineering team.

What you'll need

  • kubelogin installed (kubectl oidc-login --version returns without error)
  • An OIDC-configured Kubernetes cluster with a known identity provider
  • kubectl access using OIDC credentials
  • A free Vigilmon account

Step 1: Expose a health endpoint for kubelogin's OIDC dependencies

kubelogin has no built-in HTTP interface. Deploy a health exporter that validates the OIDC discovery endpoint, the Kubernetes API server's OIDC configuration, and the identity provider's token endpoint are all reachable:

# kubelogin-health.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: kubelogin-health-config
  namespace: observability
data:
  OIDC_ISSUER_URL: "https://your-idp.example.com"  # Replace with your OIDC issuer URL
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: kubelogin-health
  namespace: observability
spec:
  replicas: 1
  selector:
    matchLabels:
      app: kubelogin-health
  template:
    metadata:
      labels:
        app: kubelogin-health
    spec:
      containers:
        - name: checker
          image: curlimages/curl:latest
          envFrom:
            - configMapRef:
                name: kubelogin-health-config
          command:
            - /bin/sh
            - -c
            - |
              while true; do
                OIDC_DISCOVERY_URL="${OIDC_ISSUER_URL}/.well-known/openid-configuration"
                HTTP_CODE=$(curl -sSo /tmp/oidc_discovery.json -w "%{http_code}" \
                  --connect-timeout 10 --max-time 15 \
                  "$OIDC_DISCOVERY_URL" 2>/dev/null)

                if [ "$HTTP_CODE" = "200" ]; then
                  # Extract token endpoint from discovery document
                  TOKEN_ENDPOINT=$(grep -o '"token_endpoint":"[^"]*"' /tmp/oidc_discovery.json | cut -d'"' -f4)
                  JWKS_URI=$(grep -o '"jwks_uri":"[^"]*"' /tmp/oidc_discovery.json | cut -d'"' -f4)

                  # Check token endpoint reachability
                  TOKEN_CODE=$(curl -sSo /dev/null -w "%{http_code}" \
                    --connect-timeout 10 --max-time 15 \
                    "$TOKEN_ENDPOINT" 2>/dev/null)

                  # Check JWKS endpoint (used by API server to validate tokens)
                  JWKS_CODE=$(curl -sSo /tmp/jwks.json -w "%{http_code}" \
                    --connect-timeout 10 --max-time 15 \
                    "$JWKS_URI" 2>/dev/null)

                  if [ "$JWKS_CODE" = "200" ]; then
                    KEY_COUNT=$(grep -o '"kid"' /tmp/jwks.json | wc -l | tr -d ' ')
                    echo "healthy oidc_discovery=ok token_endpoint_code=$TOKEN_CODE jwks_keys=$KEY_COUNT" > /tmp/status
                  else
                    echo "degraded oidc_discovery=ok token_endpoint_code=$TOKEN_CODE jwks_code=$JWKS_CODE" > /tmp/status
                  fi
                else
                  echo "unhealthy oidc_discovery_code=$HTTP_CODE issuer=$OIDC_ISSUER_URL" > /tmp/status
                fi

                sleep 60
              done
        - name: health-server
          image: busybox:latest
          command:
            - /bin/sh
            - -c
            - |
              while true; do
                STATUS=$(cat /tmp/status 2>/dev/null || echo "starting")
                if echo "$STATUS" | grep -q "^healthy"; then
                  printf "HTTP/1.1 200 OK\r\nContent-Type: text/plain\r\nContent-Length: ${#STATUS}\r\n\r\n$STATUS" | nc -l -p 8080 -q 1
                elif echo "$STATUS" | grep -q "^degraded"; then
                  printf "HTTP/1.1 200 OK\r\nContent-Type: text/plain\r\nContent-Length: ${#STATUS}\r\n\r\n$STATUS" | nc -l -p 8080 -q 1
                else
                  printf "HTTP/1.1 503 Service Unavailable\r\nContent-Type: text/plain\r\nContent-Length: ${#STATUS}\r\n\r\n$STATUS" | nc -l -p 8080 -q 1
                fi
              done
          ports:
            - containerPort: 8080
              name: health
          volumeMounts:
            - name: status
              mountPath: /tmp
      volumes:
        - name: status
          emptyDir: {}
---
apiVersion: v1
kind: Service
metadata:
  name: kubelogin-health
  namespace: observability
spec:
  type: NodePort
  selector:
    app: kubelogin-health
  ports:
    - name: health
      port: 8080
      targetPort: 8080
      nodePort: 30089

# Edit the ConfigMap to set your actual OIDC issuer URL before applying
kubectl apply -f kubelogin-health.yaml

# Verify
NODE_IP=$(kubectl get nodes -o jsonpath='{.items[0].status.addresses[?(@.type=="InternalIP")].address}')
curl http://$NODE_IP:30089/
# healthy oidc_discovery=ok token_endpoint_code=400 jwks_keys=3

Note: A token_endpoint_code=400 is normal — the token endpoint returns 400 when called without credentials. A 200 from the discovery endpoint and a non-5xx from the token endpoint confirm the OIDC infrastructure is reachable.

Step 2: Validate kubelogin and OIDC connectivity

Before setting up monitoring, confirm kubelogin can reach the identity provider and initiate an authentication flow:

#!/bin/bash
# check-kubelogin.sh

OIDC_ISSUER="${OIDC_ISSUER_URL:-https://your-idp.example.com}"

echo "=== kubelogin installation ==="
if kubectl oidc-login --version > /dev/null 2>&1; then
  echo "OK: $(kubectl oidc-login --version)"
else
  echo "FAILED: kubelogin not found — install via: kubectl krew install oidc-login"
  exit 1
fi

echo ""
echo "=== OIDC discovery endpoint ==="
DISCOVERY_URL="${OIDC_ISSUER}/.well-known/openid-configuration"
HTTP_CODE=$(curl -sSo /tmp/oidc.json -w "%{http_code}" "$DISCOVERY_URL" 2>/dev/null)
if [ "$HTTP_CODE" = "200" ]; then
  echo "OK: OIDC discovery endpoint reachable ($DISCOVERY_URL)"
  echo "    Token endpoint: $(grep -o '"token_endpoint":"[^"]*"' /tmp/oidc.json | cut -d'"' -f4)"
  echo "    JWKS URI: $(grep -o '"jwks_uri":"[^"]*"' /tmp/oidc.json | cut -d'"' -f4)"
else
  echo "FAILED: OIDC discovery endpoint returned HTTP $HTTP_CODE"
fi

echo ""
echo "=== JWKS endpoint (API server token validation) ==="
JWKS_URI=$(grep -o '"jwks_uri":"[^"]*"' /tmp/oidc.json | cut -d'"' -f4)
if [ -n "$JWKS_URI" ]; then
  JWKS_CODE=$(curl -sSo /tmp/jwks.json -w "%{http_code}" "$JWKS_URI" 2>/dev/null)
  if [ "$JWKS_CODE" = "200" ]; then
    KEY_COUNT=$(grep -o '"kid"' /tmp/jwks.json | wc -l)
    echo "OK: JWKS endpoint reachable — $KEY_COUNT signing keys present"
  else
    echo "FAILED: JWKS endpoint returned HTTP $JWKS_CODE — API server cannot validate tokens"
  fi
fi

echo ""
echo "=== Kubernetes API server OIDC configuration ==="
API_OIDC=$(kubectl get --raw /api/v1 2>/dev/null | head -3)
if [ -n "$API_OIDC" ]; then
  echo "OK: API server reachable"
else
  echo "WARNING: cannot reach API server to verify OIDC configuration"
fi

echo ""
echo "=== Cached token check ==="
KUBELOGIN_CACHE="${HOME}/.kube/cache/oidc-login"
if [ -d "$KUBELOGIN_CACHE" ]; then
  CACHE_FILES=$(ls "$KUBELOGIN_CACHE" | wc -l)
  echo "Cached credentials: $CACHE_FILES file(s) in $KUBELOGIN_CACHE"
else
  echo "INFO: no cached credentials found (user not yet logged in via kubelogin)"
fi

export OIDC_ISSUER_URL="https://your-idp.example.com"
chmod +x check-kubelogin.sh
./check-kubelogin.sh

Step 3: Set up HTTP monitoring in Vigilmon

With the health endpoint running, add it to Vigilmon:

  1. Log in to vigilmon.online and go to Monitors → New Monitor
  2. Choose HTTP / HTTPS
  3. Add monitors for kubelogin's OIDC infrastructure:

| Monitor name | URL | Expected status | |---|---|---| | kubelogin OIDC health endpoint | http://your-node-ip:30089/ | 200 | | OIDC discovery endpoint | https://your-idp.example.com/.well-known/openid-configuration | 200 | | OIDC JWKS endpoint | https://your-idp.example.com/oauth2/v1/keys | 200 | | Kubernetes API livez | https://your-api-server:6443/livez | 200 |

  1. Set the check interval to 1 minute for the health endpoint and OIDC discovery, 2 minutes for JWKS
  2. Under Expected response, set status code 200 and match healthy for the health endpoint; match issuer in the body for the OIDC discovery endpoint (this key is always present in valid OIDC discovery documents)
  3. For private CA identity providers, add your CA certificate under TLS settings
  4. Save each monitor

Step 4: Add TCP monitors for the identity provider and API server

kubelogin requires both the identity provider and the Kubernetes API server to be reachable. Add TCP monitors for both:

  1. Go to Monitors → New Monitor
  2. Choose TCP Port
  3. Add two monitors:

| Monitor name | Host | Port | |---|---|---| | OIDC identity provider TCP | your-idp.example.com | 443 | | Kubernetes API server TCP | your-api-server | 6443 |

  1. Set the check interval to 1 minute
  2. Save both monitors

A TCP failure to the identity provider on port 443 means kubelogin cannot initiate any login flow. A TCP failure to the API server means authenticated tokens cannot be validated even if the identity provider is healthy.

Step 5: Configure alert channels

kubelogin is the authentication layer for your entire engineering team. A failure cascades immediately — every engineer with an expired token loses cluster access simultaneously. Alert the platform team the moment any OIDC component degrades.

Email alerts

  1. Go to Alert Channels → Add Channel → Email
  2. Enter your platform engineering lead, SRE on-call, and engineering management addresses
  3. Assign the channel to all kubelogin monitors with high priority

Webhook alerts

  1. Go to Alert Channels → Add Channel → Webhook
  2. Enter your incident management webhook URL (PagerDuty, Opsgenie, Slack)
  3. Sample Vigilmon webhook payload:
{
  "monitor_name": "OIDC discovery endpoint",
  "status": "down",
  "url": "https://your-idp.example.com/.well-known/openid-configuration",
  "started_at": "2024-01-15T09:00:00Z",
  "duration_seconds": 60
}

Route this to an incident that immediately notifies engineering leadership, because a sustained OIDC outage blocks all engineers from the cluster simultaneously. Include a runbook link for emergency static kubeconfig fallback procedures.

Step 6: Correlate Vigilmon alerts with kubelogin diagnostics

When you receive a kubelogin health downtime alert:

# 1. Check OIDC discovery endpoint directly
curl -v https://your-idp.example.com/.well-known/openid-configuration 2>&1 | tail -20

# 2. Check JWKS endpoint (API server validation source)
curl -s https://your-idp.example.com/.well-known/openid-configuration | \
  python3 -c "import json,sys; d=json.load(sys.stdin); print(d.get('jwks_uri',''))" | \
  xargs curl -s | python3 -m json.tool | head -20

# 3. Test kubelogin token refresh
kubectl oidc-login get-token \
  --oidc-issuer-url=https://your-idp.example.com \
  --oidc-client-id=<client-id> \
  --oidc-extra-scope=email,groups \
  2>&1 | tail -10

# 4. Check if API server OIDC config matches
kubectl get --raw /openid/v1/jwks 2>&1 | head -5

# 5. Look for token expiry in kubeconfig
kubectl config view --minify -o jsonpath='{.users[0].user}' | python3 -m json.tool 2>/dev/null

# 6. Clear stale kubelogin credential cache
rm -rf ~/.kube/cache/oidc-login/
echo "Cleared kubelogin cache — next kubectl command will trigger re-authentication"

# 7. Check identity provider status page
# Navigate to your IdP's status page (Okta status, Azure AD health, etc.)

# 8. Emergency fallback: generate a static service account token
kubectl create serviceaccount emergency-access -n kube-system
kubectl create clusterrolebinding emergency-access \
  --clusterrole=cluster-admin \
  --serviceaccount=kube-system:emergency-access
kubectl create token emergency-access -n kube-system --duration=1h

If the OIDC discovery endpoint is confirmed down, activate the emergency static kubeconfig procedure for on-call engineers and create an incident ticket with the identity provider support team.

Step 7: Create a status page for cluster authentication health

Make kubelogin's OIDC infrastructure visible to all engineers so they understand authentication outages quickly:

  1. Go to Status Pages → New Status Page
  2. Name it: "Cluster Authentication (kubelogin / OIDC)"
  3. Add your kubelogin health monitor, OIDC discovery monitor, JWKS monitor, and API livez monitor
  4. Share the URL with all engineers who access the cluster

When this page shows degraded, engineers instantly understand that authentication failures are a known infrastructure issue, not a local misconfiguration — reducing the flood of "I can't access the cluster" tickets to the platform team during an OIDC outage.

Summary

| What you set up | What it catches | |---|---| | HTTP monitor on kubelogin health endpoint | OIDC discovery failures, JWKS unavailability, key rotation issues | | HTTP monitor on OIDC discovery endpoint | Identity provider outages blocking all login flows | | HTTP monitor on JWKS endpoint | Signing key unavailability blocking token validation at API server | | HTTP monitor on Kubernetes API livez | API server degradation preventing token acceptance | | TCP monitors on IdP and API server | Network failures blocking authentication at the connection layer | | Email + webhook alert channels | Immediate notification to platform and leadership during auth outages | | Status page | Engineer self-service visibility during OIDC outages |

kubelogin is the authentication gateway for your entire engineering team's cluster access. External monitoring ensures you detect OIDC infrastructure failures within minutes — before a token expiry wave locks everyone out of the cluster simultaneously during an incident.

Get started at vigilmon.online — free tier, no credit card required.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles