How to Monitor kubectl-tree with Vigilmon

kubectl-tree is a kubectl plugin that visualizes Kubernetes object ownership hierarchies by traversing owner references and rendering them as an indented tree. Engineers use it to understand which resources were created by which controllers — a Deployment owning a ReplicaSet owning Pods, an Argo CD Application owning a set of managed resources, or a Helm release owning every component it deployed. When the Kubernetes API server that kubectl-tree queries for owner reference graphs is degraded or unreachable, the plugin returns incomplete trees or hangs indefinitely, making it impossible to trace cascading failures or identify orphaned resources during an incident.

In this tutorial you'll set up uptime monitoring for the Kubernetes API infrastructure that kubectl-tree depends on using Vigilmon — free tier, no credit card required.

Why kubectl-tree's dependencies need external monitoring

kubectl-tree makes a series of API calls to build the owner reference graph for a target object. Its failure modes stem from API availability and RBAC configuration:

• API server unavailability breaks tree traversal mid-walk — kubectl-tree recursively queries the Kubernetes API for each discovered owner; if the API becomes unavailable mid-traversal, the tree output is silently truncated and engineers believe they have a complete picture of resource ownership when they do not
• Missing RBAC permissions return empty tree branches — if the service account or user running kubectl-tree lacks list/get permissions on a resource type, that subtree is silently omitted from the output; engineers see an incomplete tree with no indication that branches are missing
• CRD resources not visible without dynamic discovery — kubectl-tree relies on the API discovery endpoint to find all resource types; if the discovery cache is stale or the API extension server for a CRD is unhealthy, those custom resources are invisible in the tree
• Large clusters cause timeouts in deep trees — in clusters with thousands of owned objects (Helm releases with many resources, Argo CD managing a full platform), kubectl-tree's recursive API queries can time out before completing, returning a partial tree that looks complete
• Owner references point to deleted resources — if a parent resource was deleted without cleaning up its children, kubectl-tree shows orphaned resources with broken parent links; if the API cannot resolve the parent UID, the orphan may appear incorrectly placed in the tree

External monitoring from Vigilmon watches the Kubernetes API layers that kubectl-tree depends on and alerts you when tree queries will return incomplete or misleading ownership graphs.

What you'll need

• kubectl-tree installed (kubectl tree --version returns without error)
• kubectl access to your monitored cluster
• A node or LoadBalancer endpoint to host the health check
• A free Vigilmon account

Step 1: Expose a health endpoint for kubectl-tree's API dependencies

kubectl-tree has no built-in HTTP interface. Deploy a health exporter that validates the Kubernetes API server and its discovery endpoint — the two foundations kubectl-tree requires for accurate tree output:

# kubectl-tree-health.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: kubectl-tree-health
  namespace: observability
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: kubectl-tree-health-reader
rules:
  - apiGroups: [""]
    resources: ["pods", "replicasets", "deployments", "namespaces", "nodes"]
    verbs: ["get", "list"]
  - apiGroups: ["apps"]
    resources: ["deployments", "replicasets", "statefulsets", "daemonsets"]
    verbs: ["get", "list"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: kubectl-tree-health-binding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: kubectl-tree-health-reader
subjects:
  - kind: ServiceAccount
    name: kubectl-tree-health
    namespace: observability
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: kubectl-tree-health
  namespace: observability
spec:
  replicas: 1
  selector:
    matchLabels:
      app: kubectl-tree-health
  template:
    metadata:
      labels:
        app: kubectl-tree-health
    spec:
      serviceAccountName: kubectl-tree-health
      containers:
        - name: checker
          image: bitnami/kubectl:latest
          command:
            - /bin/sh
            - -c
            - |
              while true; do
                # Validate API server is reachable
                if ! kubectl get nodes --request-timeout=10s > /dev/null 2>&1; then
                  echo "unhealthy api_server=unreachable" > /tmp/status
                  sleep 30
                  continue
                fi

                # Validate API discovery endpoint (required for CRD traversal)
                if ! kubectl api-resources --request-timeout=10s > /tmp/api_resources 2>&1; then
                  echo "degraded api_discovery=failed" > /tmp/status
                  sleep 30
                  continue
                fi

                RESOURCE_COUNT=$(wc -l < /tmp/api_resources)
                NODE_COUNT=$(kubectl get nodes --no-headers 2>/dev/null | wc -l | tr -d ' ')

                # Validate owner reference resolution works by checking a replicaset
                RS_COUNT=$(kubectl get replicasets --all-namespaces --no-headers 2>/dev/null | wc -l | tr -d ' ')

                echo "healthy api=reachable resources=$RESOURCE_COUNT nodes=$NODE_COUNT replicasets=$RS_COUNT" > /tmp/status
                sleep 60
              done
        - name: health-server
          image: busybox:latest
          command:
            - /bin/sh
            - -c
            - |
              while true; do
                STATUS=$(cat /tmp/status 2>/dev/null || echo "starting")
                if echo "$STATUS" | grep -q "^healthy"; then
                  printf "HTTP/1.1 200 OK\r\nContent-Type: text/plain\r\nContent-Length: ${#STATUS}\r\n\r\n$STATUS" | nc -l -p 8080 -q 1
                elif echo "$STATUS" | grep -q "^degraded"; then
                  printf "HTTP/1.1 200 OK\r\nContent-Type: text/plain\r\nContent-Length: ${#STATUS}\r\n\r\n$STATUS" | nc -l -p 8080 -q 1
                else
                  printf "HTTP/1.1 503 Service Unavailable\r\nContent-Type: text/plain\r\nContent-Length: ${#STATUS}\r\n\r\n$STATUS" | nc -l -p 8080 -q 1
                fi
              done
          ports:
            - containerPort: 8080
              name: health
          volumeMounts:
            - name: status
              mountPath: /tmp
      volumes:
        - name: status
          emptyDir: {}
---
apiVersion: v1
kind: Service
metadata:
  name: kubectl-tree-health
  namespace: observability
spec:
  type: NodePort
  selector:
    app: kubectl-tree-health
  ports:
    - name: health
      port: 8080
      targetPort: 8080
      nodePort: 30088

kubectl apply -f kubectl-tree-health.yaml

# Verify
NODE_IP=$(kubectl get nodes -o jsonpath='{.items[0].status.addresses[?(@.type=="InternalIP")].address}')
curl http://$NODE_IP:30088/
# healthy api=reachable resources=142 nodes=3 replicasets=18

Step 2: Validate kubectl-tree and owner reference resolution

Before setting up monitoring, confirm kubectl-tree can build a complete tree from the cluster:

#!/bin/bash
# check-kubectl-tree.sh

echo "=== kubectl-tree installation ==="
if kubectl tree version > /dev/null 2>&1; then
  echo "OK: kubectl-tree installed"
else
  echo "FAILED: kubectl-tree not found — install via: kubectl krew install tree"
  exit 1
fi

echo ""
echo "=== API discovery check ==="
RESOURCE_COUNT=$(kubectl api-resources 2>/dev/null | wc -l)
echo "API resource types discovered: $RESOURCE_COUNT"
[ "$RESOURCE_COUNT" -lt 50 ] && echo "WARNING: unusually low resource count — API discovery may be degraded"

echo ""
echo "=== Owner reference traversal test ==="
# Find a deployment to use as a tree root
DEPLOYMENT=$(kubectl get deployments --all-namespaces --no-headers 2>/dev/null | head -1 | awk '{print $1 "/" $2}')
if [ -n "$DEPLOYMENT" ]; then
  NS=$(echo "$DEPLOYMENT" | cut -d/ -f1)
  NAME=$(echo "$DEPLOYMENT" | cut -d/ -f2)
  echo "Testing tree traversal on deployment: $NAME in namespace: $NS"
  if kubectl tree deployment "$NAME" -n "$NS" 2>&1 | head -10; then
    echo "OK: tree traversal successful"
  else
    echo "WARNING: tree traversal failed — check RBAC and API availability"
  fi
else
  echo "WARNING: no deployments found — cannot test tree traversal"
fi

echo ""
echo "=== RBAC permission check ==="
for resource in pods replicasets deployments statefulsets daemonsets; do
  if kubectl auth can-i list "$resource" --all-namespaces > /dev/null 2>&1; then
    echo "OK: can list $resource"
  else
    echo "WARNING: cannot list $resource — tree will have missing branches"
  fi
done

chmod +x check-kubectl-tree.sh
./check-kubectl-tree.sh

Step 3: Set up HTTP monitoring in Vigilmon

With the health endpoint running, add it to Vigilmon:

1. Log in to vigilmon.online and go to Monitors → New Monitor
2. Choose HTTP / HTTPS
3. Add monitors for kubectl-tree's API infrastructure:

| Monitor name | URL | Expected status | |---|---|---| | kubectl-tree API health | http://your-node-ip:30088/ | 200 | | Kubernetes API livez | https://your-api-server:6443/livez | 200 | | Kubernetes API readyz | https://your-api-server:6443/readyz | 200 | | Kubernetes API discovery | https://your-api-server:6443/api | 200 |

4. Set the check interval to 1 minute
5. Under Expected response, set status code 200 and match healthy in the response body for the health endpoint
6. For private CA clusters, add your CA certificate under TLS settings
7. Save each monitor

Step 4: Add a TCP monitor for the Kubernetes API server

kubectl-tree makes direct API connections to the Kubernetes API server. A TCP monitor catches network failures that would cause tree traversal to hang or return empty results:

1. Go to Monitors → New Monitor
2. Choose TCP Port
3. Enter your Kubernetes API server hostname and port 6443
4. Set the check interval to 1 minute
5. Save the monitor

A TCP failure on port 6443 means kubectl-tree will time out on every query, returning empty or truncated trees that look like genuine "no owned resources" results.

Step 5: Configure alert channels

kubectl-tree is used during incidents to trace cascading failures through resource ownership chains. When its API dependencies are degraded, engineers investigate ownership hierarchies with incomplete data and may miss the root cause.

Email alerts

1. Go to Alert Channels → Add Channel → Email
2. Enter your SRE on-call and platform engineering addresses
3. Assign the channel to all kubectl-tree monitors

Webhook alerts

1. Go to Alert Channels → Add Channel → Webhook
2. Enter your incident management webhook URL (PagerDuty, Opsgenie, Slack)
3. Sample Vigilmon webhook payload:

{
  "monitor_name": "kubectl-tree API health",
  "status": "down",
  "url": "http://your-node-ip:30088/",
  "started_at": "2024-01-15T22:00:00Z",
  "duration_seconds": 120
}

Route this to a runbook that instructs on-call engineers to use kubectl get <resource> -o yaml to manually inspect ownerReferences fields when kubectl-tree is unavailable during an incident.

Step 6: Correlate Vigilmon alerts with kubectl-tree diagnostics

When you receive a kubectl-tree health downtime alert:

# 1. Check Kubernetes API server status
kubectl cluster-info
kubectl get nodes

# 2. Test API discovery endpoint
kubectl api-resources | wc -l

# 3. Try a basic tree traversal
kubectl tree deployment <name> -n <namespace>

# 4. Check if owner references resolve manually
kubectl get replicaset <name> -n <namespace> -o jsonpath='{.metadata.ownerReferences}'

# 5. Verify RBAC for tree traversal
kubectl auth can-i list pods --all-namespaces
kubectl auth can-i list replicasets --all-namespaces
kubectl auth can-i list deployments --all-namespaces

# 6. Check API extension server health (for CRD resources)
kubectl get apiservices | grep -v True

# 7. Inspect the health pod
kubectl logs -n observability -l app=kubectl-tree-health -c checker --tail=20

# 8. Manual owner reference inspection as fallback
kubectl get <resource> <name> -n <namespace> -o jsonpath='{.metadata.ownerReferences[*].name}'

If kubectl api-resources returns an incomplete list or hangs, the API extension server for one or more CRDs is unhealthy. Use kubectl get apiservices | grep -v True to identify the degraded API group and contact the team responsible for that CRD.

Step 7: Create a status page for Kubernetes resource ownership visibility

Make kubectl-tree's API dependencies visible to the SRE and platform teams:

1. Go to Status Pages → New Status Page
2. Name it: "Kubernetes Resource Ownership (kubectl-tree)"
3. Add your kubectl-tree health monitor, API livez, API readyz, and API discovery monitors
4. Share the URL with the SRE on-call team

During an incident, engineers checking this page instantly know whether kubectl-tree output is trustworthy or whether they need to fall back to manual ownerReferences inspection.

Summary

| What you set up | What it catches | |---|---| | HTTP monitor on kubectl-tree health endpoint | API unreachability, discovery failures, missing replicasets | | HTTP monitor on API livez/readyz | API server degradation affecting tree traversal accuracy | | HTTP monitor on API discovery endpoint | CRD visibility failures causing missing tree branches | | TCP monitor on port 6443 | Network failures causing tree traversal to hang or return empty | | Email + webhook alert channels | Immediate notification when tree accuracy is compromised | | Status page | SRE visibility into ownership graph reliability during incidents |

kubectl-tree is most valuable during incidents when engineers need to understand resource ownership cascades quickly. External monitoring ensures you know when the API infrastructure it depends on is degraded — before an incident forces engineers to rely on misleading or incomplete tree output.

Get started at vigilmon.online — free tier, no credit card required.