How to Monitor kube-state-metrics with Vigilmon

kube-state-metrics is a Prometheus metrics exporter that exposes the state of Kubernetes objects — pods, deployments, nodes, services, PersistentVolumeClaims, and more — as Prometheus-compatible metrics. Unlike the Metrics Server, which focuses on resource usage (CPU and memory), kube-state-metrics exposes the logical state of the Kubernetes API: whether a deployment has the desired number of replicas, whether a pod is in a CrashLoopBackOff, whether a PVC is bound. Most Kubernetes alerting rules for Grafana and PagerDuty depend on kube-state-metrics being available.

But kube-state-metrics itself is a deployment that can fail. If it goes down, every alert rule that depends on kube_pod_status_phase, kube_deployment_status_replicas_unavailable, and dozens of other metrics silently stops firing. In this tutorial you'll set up uptime and response-time monitoring for kube-state-metrics using Vigilmon — free tier, no credit card required.

Why kube-state-metrics needs external monitoring

kube-state-metrics failures create blind spots in your alerting stack:

• kube-state-metrics pod crash — if the exporter crashes, Prometheus still scrapes the endpoint but receives connection refused; alert rules that evaluate to absent(kube_pod_status_phase) only fire if you have explicit absence alerts configured
• Prometheus scrape config drift — a change to the Prometheus ServiceMonitor or scrape config that removes kube-state-metrics from scrape targets silently drops all Kubernetes state metrics without any indication in the kube-state-metrics pod itself
• Watch cache corruption — kube-state-metrics maintains a watch cache of cluster state; on large clusters or during API server disruptions this cache can fall behind, producing stale metrics while returning HTTP 200
• Port mismatch after upgrade — kube-state-metrics changed its default telemetry port between versions; upgrades can silently break scrape configurations pointing to the old port
• RBAC permission gaps — kube-state-metrics needs ClusterRole read access to dozens of resource types; a permission gap causes it to skip those resource types and produce incomplete metrics with no error surfaced in the HTTP response

External monitoring from Vigilmon watches the kube-state-metrics HTTP endpoint from outside the cluster, giving you an independent signal that the exporter is reachable and producing metrics.

What you'll need

• kube-state-metrics deployed in your Kubernetes cluster
• The kube-state-metrics Service reachable externally or via Ingress
• A free Vigilmon account

Step 1: Deploy kube-state-metrics and expose its endpoints

Install kube-state-metrics via the official Helm chart:

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update
helm install kube-state-metrics prometheus-community/kube-state-metrics \
  --namespace monitoring \
  --create-namespace \
  --set prometheusScrape=true

By default, kube-state-metrics exposes two ports:

• Port 8080 — Kubernetes object state metrics (the primary metrics endpoint)
• Port 8081 — self-telemetry metrics (scrape duration, cache size, etc.)

Expose via Ingress:

# kube-state-metrics-ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: kube-state-metrics-ingress
  namespace: monitoring
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /$2
spec:
  ingressClassName: nginx
  rules:
    - host: metrics.yourdomain.com
      http:
        paths:
          - path: /ksm(/|$)(.*)
            pathType: Prefix
            backend:
              service:
                name: kube-state-metrics
                port:
                  number: 8080

kubectl apply -f kube-state-metrics-ingress.yaml

# Verify the metrics endpoint
curl https://metrics.yourdomain.com/ksm/metrics | grep kube_pod_status_phase | head -5
# kube_pod_status_phase{exported_namespace="default",phase="Running",...} 3

# Check the healthz endpoint
curl https://metrics.yourdomain.com/ksm/healthz
# OK

Step 2: Understand the kube-state-metrics health indicators

kube-state-metrics exposes several endpoints for health checking:

• /healthz — returns OK when the exporter is running; failure means it has crashed or is still initializing
• /livez — liveness check; returns OK
• /readyz — readiness check; returns OK only after the initial watch cache has populated; failures here indicate that no metrics are yet available
• /metrics — the primary metrics endpoint; a missing or truncated response indicates cache corruption or watch errors

Key self-telemetry metrics to watch:

kube_state_metrics_list_total           # total list operations
kube_state_metrics_watch_total          # total watch operations
kube_state_metrics_decode_errors_total  # errors decoding API objects
kube_state_metrics_shard_ordinal        # shard identity (for HA deployments)

Step 3: Set up HTTP monitoring in Vigilmon

With kube-state-metrics endpoints accessible, add them to Vigilmon:

1. Log in to vigilmon.online and go to Monitors → New Monitor
2. Choose HTTP / HTTPS
3. Add a monitor for each endpoint:

| Monitor name | URL | Expected status | |---|---|---| | kube-state-metrics health | https://metrics.yourdomain.com/ksm/healthz | 200 | | kube-state-metrics ready | https://metrics.yourdomain.com/ksm/readyz | 200 | | kube-state-metrics metrics | https://metrics.yourdomain.com/ksm/metrics | 200 |

4. Set the check interval to 1 minute
5. For the health monitor, under Expected response, match OK in the response body
6. For the metrics endpoint, match kube_pod_status_phase in the response body to confirm Kubernetes object metrics are present
7. Save each monitor

Vigilmon probes from multiple geographic regions. A probe failure on the metrics endpoint while the pod appears healthy typically indicates a network policy or Ingress configuration issue.

Step 4: Monitor the kube-state-metrics TCP port

Add a TCP-layer monitor to catch port-level failures:

1. Go to Monitors → New Monitor
2. Choose TCP Port
3. Enter your kube-state-metrics hostname and port 8080
4. Save the monitor

A TCP failure at the port level when the pod is Running indicates a Service selector drift (common after Helm upgrades that rename labels), a NetworkPolicy blocking inbound connections, or a pod that is Running but not listening (crash in the HTTP server goroutine).

Step 5: Configure alert channels

kube-state-metrics downtime silently disables your entire Kubernetes state alerting stack. Configure alerts with high urgency.

Email alerts

1. Go to Alert Channels → Add Channel → Email
2. Enter your observability team or on-call SRE address
3. Assign the channel to all kube-state-metrics monitors

Webhook alerts for incident routing

1. Go to Alert Channels → Add Channel → Webhook
2. Enter your incident management webhook URL (PagerDuty, Opsgenie, Slack)
3. The payload Vigilmon sends:

{
  "monitor_name": "kube-state-metrics metrics",
  "status": "down",
  "url": "https://metrics.yourdomain.com/ksm/metrics",
  "started_at": "2024-01-15T10:23:00Z",
  "duration_seconds": 180
}

Wire this alert to a runbook that notes the time of failure, checks Prometheus for gaps in kube_* metrics matching that window, and restarts the kube-state-metrics deployment.

Step 6: Correlate Vigilmon alerts with kube-state-metrics diagnostics

When you receive a kube-state-metrics downtime alert, run these checks:

# 1. Check pod status
kubectl get pods -n monitoring -l app.kubernetes.io/name=kube-state-metrics

# 2. Check pod logs for API server or RBAC errors
kubectl logs -n monitoring -l app.kubernetes.io/name=kube-state-metrics --tail=100

# 3. Verify RBAC binding
kubectl get clusterrolebinding | grep kube-state-metrics
kubectl describe clusterrole kube-state-metrics | grep "resources:" -A 3

# 4. Check readiness probe status
kubectl describe pod -n monitoring -l app.kubernetes.io/name=kube-state-metrics | grep -A 10 "Readiness\|Liveness"

# 5. Query Prometheus for scrape gaps
# In Prometheus UI: up{job="kube-state-metrics"}
# If this returns 0 or is absent, the scrape target is down

# 6. Check self-telemetry for decode errors
kubectl port-forward svc/kube-state-metrics 8081:8081 -n monitoring &
curl http://localhost:8081/metrics | grep decode_errors
# kube_state_metrics_decode_errors_total 0

# 7. Check for API server rate limiting
kubectl logs -n monitoring -l app.kubernetes.io/name=kube-state-metrics | grep "rate limit\|429\|Too Many"

# 8. Verify watch cache is populated
curl http://localhost:8080/readyz
# OK

If Vigilmon shows the metrics endpoint returning slowly (high response time) rather than failing, kube-state-metrics may be struggling with a large cluster or frequent watch reconnections — check the kube_state_metrics_watch_total metric for abnormal reconnection rates.

Step 7: Create a status page for Kubernetes observability infrastructure

kube-state-metrics is foundational observability infrastructure. Make its status visible to engineering leadership:

1. Go to Status Pages → New Status Page
2. Name it: "Kubernetes Observability Infrastructure"
3. Add your monitors: kube-state-metrics health, readiness, metrics
4. Share the URL with your observability and SRE teams

Summary

| What you set up | What it catches | |---|---| | HTTP monitor on /healthz | kube-state-metrics crash, pod failure | | HTTP monitor on /readyz | Watch cache not initialized, initial startup failure | | HTTP monitor on /metrics | Metrics unavailable, RBAC gaps causing empty responses | | TCP monitor on port 8080 | NetworkPolicy blocks, Service selector drift | | Email + webhook alert channels | Immediate notification when Kubernetes state metrics drop | | Status page | Visibility into observability infrastructure health |

kube-state-metrics is the backbone of Kubernetes state alerting. External monitoring ensures that the exporter itself is always healthy so your Prometheus alert rules never silently evaluate against stale or missing data.

Get started at vigilmon.online — free tier, no credit card required.