How to Monitor kubectl-view-secret with Vigilmon

kubectl-view-secret is a kubectl plugin that decodes and displays Kubernetes Secret values without requiring manual base64 decoding. Instead of running kubectl get secret <name> -o json | jq '.data.<key>' | base64 -d, engineers run kubectl view-secret <name> <key> and receive the plaintext value directly. SRE, DevOps, and security teams use it during incident response to quickly inspect TLS certificates, connection strings, API keys, and token values stored in Kubernetes Secrets. When the Kubernetes API server's secret retrieval capabilities are degraded or secret access is blocked by misconfigured RBAC, kubectl-view-secret fails in ways that can delay credential verification and incident resolution.

In this tutorial you'll set up uptime monitoring for the Kubernetes infrastructure that kubectl-view-secret depends on using Vigilmon — free tier, no credit card required.

Why kubectl-view-secret's dependencies need external monitoring

kubectl-view-secret calls the Kubernetes API server to retrieve secret resources and then decodes their base64-encoded values locally. Its failure modes come from API availability, RBAC configuration, and etcd storage health:

• API server unavailability prevents all secret retrieval — kubectl-view-secret makes standard get calls to the Kubernetes Secrets API; if the API server is unavailable or returning errors, the plugin fails with a connection error, leaving engineers unable to verify credentials during an outage when accurate secret data is most urgently needed
• RBAC misconfiguration causes silent permission failures — if the service account or user lacks get permission on secrets in a given namespace, kubectl-view-secret returns a "forbidden" error rather than the secret value; in namespaced RBAC configurations, this can appear to work in one namespace and fail in another, creating inconsistent behavior that looks like a plugin bug
• etcd performance degradation causes secret reads to time out — Kubernetes Secrets are stored in etcd; if etcd is under heavy write load or experiencing latency spikes, secret reads slow down proportionally and kubectl-view-secret can time out mid-retrieval, returning a partial or empty result
• Encrypted secret storage provider unavailability blocks decryption — clusters using envelope encryption for secrets at rest (KMS providers like AWS KMS, Google Cloud KMS, or HashiCorp Vault) cannot decrypt secret data if the KMS provider is unreachable; kubectl-view-secret receives an encrypted blob it cannot decode and returns an opaque error rather than the secret value
• Secret version conflicts after rotation leave stale values — if a secret was rotated while kubectl-view-secret was mid-retrieval, the plugin may display the pre-rotation value; in high-rotation environments (short-lived tokens, rotating TLS certs), the displayed value may already be invalid but appears valid to the engineer inspecting it

External monitoring from Vigilmon watches the Kubernetes secret storage infrastructure that kubectl-view-secret depends on and alerts you when secret retrieval cannot be trusted.

What you'll need

• kubectl-view-secret installed (kubectl view-secret --version returns without error)
• kubectl access to your monitored cluster
• A node or LoadBalancer endpoint to host the health check
• A free Vigilmon account

Step 1: Expose a health endpoint for kubectl-view-secret's dependencies

kubectl-view-secret has no built-in HTTP interface. Deploy a health exporter that validates Kubernetes secret API availability and base64 decoding consistency:

# kubectl-view-secret-health.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: kubectl-view-secret-health
  namespace: observability
---
apiVersion: v1
kind: Secret
metadata:
  name: kubectl-view-secret-canary
  namespace: observability
type: Opaque
stringData:
  health-check-key: "kubectl-view-secret-health-check-value-ok"
  timestamp: "canary"
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: kubectl-view-secret-health-reader
  namespace: observability
rules:
  - apiGroups: [""]
    resources: ["secrets"]
    verbs: ["get", "list"]
    resourceNames: ["kubectl-view-secret-canary"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: kubectl-view-secret-health-binding
  namespace: observability
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: kubectl-view-secret-health-reader
subjects:
  - kind: ServiceAccount
    name: kubectl-view-secret-health
    namespace: observability
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: kubectl-view-secret-health
  namespace: observability
spec:
  replicas: 1
  selector:
    matchLabels:
      app: kubectl-view-secret-health
  template:
    metadata:
      labels:
        app: kubectl-view-secret-health
    spec:
      serviceAccountName: kubectl-view-secret-health
      containers:
        - name: checker
          image: bitnami/kubectl:latest
          command:
            - /bin/sh
            - -c
            - |
              while true; do
                # Check Kubernetes Secrets API availability
                if ! kubectl get secret kubectl-view-secret-canary -n observability --request-timeout=10s > /dev/null 2>&1; then
                  echo "unhealthy api=unreachable secrets_api=down" > /tmp/status
                  sleep 30
                  continue
                fi

                # Retrieve and decode the canary secret value
                ENCODED=$(kubectl get secret kubectl-view-secret-canary \
                  -n observability \
                  -o jsonpath='{.data.health-check-key}' \
                  --request-timeout=10s 2>/dev/null)

                if [ -z "$ENCODED" ]; then
                  echo "degraded api=reachable secret_read=empty" > /tmp/status
                  sleep 30
                  continue
                fi

                DECODED=$(echo "$ENCODED" | base64 -d 2>/dev/null)

                if [ "$DECODED" = "kubectl-view-secret-health-check-value-ok" ]; then
                  SECRET_COUNT=$(kubectl get secrets -n observability --no-headers 2>/dev/null | wc -l | tr -d ' ')
                  echo "healthy api=reachable decode=ok secrets_in_ns=$SECRET_COUNT" > /tmp/status
                else
                  echo "degraded api=reachable decode=mismatch expected=ok got=$DECODED" > /tmp/status
                fi

                sleep 60
              done
        - name: health-server
          image: busybox:latest
          command:
            - /bin/sh
            - -c
            - |
              while true; do
                STATUS=$(cat /tmp/status 2>/dev/null || echo "starting")
                if echo "$STATUS" | grep -q "^healthy"; then
                  printf "HTTP/1.1 200 OK\r\nContent-Type: text/plain\r\nContent-Length: ${#STATUS}\r\n\r\n$STATUS" | nc -l -p 8080 -q 1
                else
                  printf "HTTP/1.1 503 Service Unavailable\r\nContent-Type: text/plain\r\nContent-Length: ${#STATUS}\r\n\r\n$STATUS" | nc -l -p 8080 -q 1
                fi
              done
          ports:
            - containerPort: 8080
              name: health
          volumeMounts:
            - name: status
              mountPath: /tmp
      volumes:
        - name: status
          emptyDir: {}
---
apiVersion: v1
kind: Service
metadata:
  name: kubectl-view-secret-health
  namespace: observability
spec:
  type: NodePort
  selector:
    app: kubectl-view-secret-health
  ports:
    - name: health
      port: 8080
      targetPort: 8080
      nodePort: 30091

kubectl apply -f kubectl-view-secret-health.yaml

# Verify
NODE_IP=$(kubectl get nodes -o jsonpath='{.items[0].status.addresses[?(@.type=="InternalIP")].address}')
curl http://$NODE_IP:30091/
# healthy api=reachable decode=ok secrets_in_ns=3

Step 2: Validate kubectl-view-secret and secrets API health

Before setting up monitoring, confirm kubectl-view-secret can retrieve and decode secrets correctly:

#!/bin/bash
# check-kubectl-view-secret.sh

echo "=== kubectl-view-secret installation ==="
if kubectl view-secret --version > /dev/null 2>&1 || kubectl view-secret --help > /dev/null 2>&1; then
  echo "OK: kubectl-view-secret installed"
else
  echo "FAILED: kubectl-view-secret not found — install via: kubectl krew install view-secret"
  exit 1
fi

echo ""
echo "=== Secrets API availability ==="
if kubectl get secrets --all-namespaces --no-headers > /dev/null 2>&1; then
  SECRET_COUNT=$(kubectl get secrets --all-namespaces --no-headers | wc -l)
  echo "OK: Secrets API reachable — $SECRET_COUNT secrets visible across all namespaces"
else
  echo "FAILED: Secrets API unreachable"
  exit 1
fi

echo ""
echo "=== RBAC permission check ==="
for ns in default kube-system observability; do
  if kubectl auth can-i get secrets -n "$ns" > /dev/null 2>&1; then
    echo "OK: can get secrets in namespace $ns"
  else
    echo "WARNING: cannot get secrets in namespace $ns"
  fi
done

echo ""
echo "=== Base64 decode round-trip test ==="
# Create a test secret, view it, verify the value
TEST_VALUE="kubectl-view-secret-test-$(date +%s)"
kubectl create secret generic view-secret-test \
  --from-literal=testkey="$TEST_VALUE" \
  --namespace=default \
  --dry-run=client -o yaml | kubectl apply -f - > /dev/null 2>&1

sleep 2
RETRIEVED=$(kubectl view-secret view-secret-test testkey --namespace=default 2>/dev/null)
kubectl delete secret view-secret-test --namespace=default > /dev/null 2>&1

if [ "$RETRIEVED" = "$TEST_VALUE" ]; then
  echo "OK: secret round-trip successful — stored and retrieved value match"
else
  echo "FAILED: secret round-trip mismatch — stored '$TEST_VALUE' but retrieved '$RETRIEVED'"
fi

echo ""
echo "=== etcd health check ==="
if kubectl get --raw /readyz/etcd > /dev/null 2>&1; then
  echo "OK: etcd readiness check passed"
else
  echo "WARNING: etcd readiness check unavailable or failed — secret reads may be slow"
fi

chmod +x check-kubectl-view-secret.sh
./check-kubectl-view-secret.sh

Step 3: Set up HTTP monitoring in Vigilmon

With the health endpoint running, add it to Vigilmon:

1. Log in to vigilmon.online and go to Monitors → New Monitor
2. Choose HTTP / HTTPS
3. Add monitors for kubectl-view-secret's secret storage infrastructure:

| Monitor name | URL | Expected status | |---|---|---| | kubectl-view-secret health | http://your-node-ip:30091/ | 200 | | Kubernetes API livez | https://your-api-server:6443/livez | 200 | | Kubernetes API readyz | https://your-api-server:6443/readyz | 200 | | etcd health | https://your-api-server:6443/readyz/etcd | 200 |

4. Set the check interval to 1 minute
5. Under Expected response, match healthy in the response body for the health endpoint and ok for the readyz/etcd endpoint
6. For clusters using KMS envelope encryption, add the KMS provider health endpoint as an additional monitor
7. Save each monitor

Step 4: Add a TCP monitor for the Kubernetes API server

kubectl-view-secret makes direct API calls to the Kubernetes API server to retrieve Secrets. A TCP monitor catches network failures before the HTTP layer:

1. Go to Monitors → New Monitor
2. Choose TCP Port
3. Enter your Kubernetes API server hostname and port 6443
4. Set the check interval to 1 minute
5. Save the monitor

A TCP failure on port 6443 means kubectl-view-secret cannot retrieve any Secret values, forcing engineers to use out-of-band methods to access credential data during incident response.

Step 5: Configure alert channels

kubectl-view-secret is used for time-sensitive credential inspection during incidents. When Secret retrieval is degraded, engineers may make incorrect decisions about credential validity, trigger unnecessary rotations, or waste time on manual base64 decoding workflows.

Email alerts

1. Go to Alert Channels → Add Channel → Email
2. Enter your SRE on-call and security engineering addresses
3. Assign the channel to all kubectl-view-secret monitors

Webhook alerts

1. Go to Alert Channels → Add Channel → Webhook
2. Enter your incident management webhook URL (PagerDuty, Opsgenie, Slack)
3. Sample Vigilmon webhook payload:

{
  "monitor_name": "kubectl-view-secret health",
  "status": "down",
  "url": "http://your-node-ip:30091/",
  "started_at": "2024-01-15T22:00:00Z",
  "duration_seconds": 120
}

Route this to a runbook that instructs on-call engineers to fall back to manual secret inspection using kubectl get secret <name> -o jsonpath='{.data.<key>}' | base64 -d when kubectl-view-secret is unavailable.

Step 6: Correlate Vigilmon alerts with kubectl-view-secret diagnostics

When you receive a kubectl-view-secret health downtime alert:

# 1. Check Kubernetes API server status
kubectl cluster-info
kubectl get nodes

# 2. Test Secrets API directly
kubectl get secrets -n observability --no-headers | wc -l

# 3. Test RBAC for secret access
kubectl auth can-i get secrets --all-namespaces

# 4. Check etcd health
kubectl get --raw /readyz/etcd

# 5. Test manual base64 decode (kubectl-view-secret equivalent)
kubectl get secret kubectl-view-secret-canary -n observability \
  -o jsonpath='{.data.health-check-key}' | base64 -d
echo ""

# 6. Check if KMS provider is healthy (if envelope encryption is enabled)
kubectl get --raw /readyz | grep kms

# 7. Check etcd latency (look for high-latency errors in API server logs)
kubectl logs -n kube-system -l component=kube-apiserver --tail=50 2>/dev/null | \
  grep -i "etcd\|timeout\|deadline"

# 8. Inspect the health pod logs
kubectl logs -n observability -l app=kubectl-view-secret-health -c checker --tail=20

# 9. Manual secret inspection as fallback
kubectl get secret <secret-name> -n <namespace> -o json | \
  python3 -c "import json,sys,base64; d=json.load(sys.stdin)['data']; [print(f'{k}: {base64.b64decode(v).decode()}') for k,v in d.items()]"

If the health endpoint returns degraded decode=mismatch, the canary Secret value was modified outside the expected workflow — check for unauthorized access or Secret mutation by admission webhooks. If it returns api=unreachable, check etcd connectivity and API server logs for storage backend errors.

Step 7: Create a status page for Kubernetes secret storage visibility

Make secret storage infrastructure health visible to the SRE and security teams:

1. Go to Status Pages → New Status Page
2. Name it: "Kubernetes Secret Storage (kubectl-view-secret)"
3. Add the kubectl-view-secret health monitor, API livez/readyz monitors, and the etcd monitor
4. Share the URL with your SRE on-call and security engineering teams

During credential-related incidents, engineers checking this page know immediately whether Secret retrieval is reliable or whether they should use alternative methods to access credential data.

Summary

| What you set up | What it catches | |---|---| | HTTP monitor on kubectl-view-secret health endpoint | Secrets API degradation, base64 decode failures, etcd issues | | HTTP monitor on API livez/readyz | API server degradation affecting all Secret operations | | HTTP monitor on etcd health endpoint | etcd failures causing secret read timeouts | | TCP monitor on port 6443 | Network failures preventing all Secret retrieval | | Email + webhook alert channels | Immediate notification when credential inspection is unreliable | | Status page | SRE and security team visibility into Secret storage health |

kubectl-view-secret is a convenience tool with an operational impact that exceeds its apparent simplicity — engineers rely on it during the most time-pressured moments of incident response. External monitoring ensures you know when the Kubernetes Secret infrastructure it depends on is degraded, before a delayed or failed credential check extends an outage.

Get started at vigilmon.online — free tier, no credit card required.