tutorial

Monitoring Kubernetes Node Problem Detector: A Guide for Platform Engineers

Kubernetes Node Problem Detector (NPD) is a cluster-level daemon that surfaces node-level failures — kernel panics, disk pressure, container runtime hangs, O...

Kubernetes Node Problem Detector (NPD) is a cluster-level daemon that surfaces node-level failures — kernel panics, disk pressure, container runtime hangs, OOM kills — as Kubernetes conditions and events. Without NPD, these low-level failures are invisible to the control plane; pods get rescheduled onto degraded nodes, and reliability suffers. But NPD itself is a monitoring component that needs to be monitored: if the daemon crashes, goes unresponsive, or stops reporting conditions, your cluster becomes blind to entire categories of node failure.

What Node Problem Detector Does

NPD runs as a DaemonSet — one pod per node — and continuously monitors:

  • System logs (kernel log, docker log, containerd log) for known error patterns
  • System metrics via custom plugins (disk, memory, CPU, network)
  • Kernel conditions such as KernelDeadlock, ReadonlyFilesystem, MemoryPressure

When NPD detects a problem, it creates a NodeCondition or a Kubernetes Event. The scheduler and other controllers can then act on these conditions to cordon degraded nodes or trigger remediation.

Why Monitoring NPD Is Critical

NPD is a reliability primitive. When it fails:

  • Node conditions go unreported — a node with a kernel deadlock looks healthy to the scheduler
  • Events are not generated — operators lose the signal that drives cluster remediation
  • System log scraping stops — transient kernel issues accumulate silently
  • Auto-remediation tools (like Node Auto Repair on GKE) lose their input signal

A single node with a crashed NPD pod is a blind spot. An NPD DaemonSet update gone wrong can create blind spots across your entire cluster.

Key Components to Monitor

1. DaemonSet Pod Health

The most fundamental check: all NPD pods must be running and ready.

# Verify all DaemonSet pods are ready
kubectl get daemonset node-problem-detector -n kube-system

# Expected: DESIRED == READY
# NAME                    DESIRED   CURRENT   READY   ...
# node-problem-detector   5         5         5       ...

A DESIRED vs READY mismatch means one or more nodes have a failed NPD pod. Find the failing pod:

kubectl get pods -n kube-system -l app=node-problem-detector --field-selector='status.phase!=Running'

And inspect its logs:

kubectl logs -n kube-system -l app=node-problem-detector --since=10m | grep -i error

2. System Log Scraping Health

NPD monitors system logs by tailing files like /var/log/kern.log, /var/log/docker.log, or /run/log/journal. If these files rotate unexpectedly, grow faster than NPD can consume, or disappear, NPD will stop detecting problems.

Check that log monitoring is active by reviewing NPD's own output:

# Check for log tailing errors in NPD pod logs
kubectl logs -n kube-system \
  $(kubectl get pods -n kube-system -l app=node-problem-detector -o name | head -1) \
  | grep -E "(error|failed|cannot open)"

You can also validate that NPD is actively processing logs by checking the timestamp of its most recent condition update:

kubectl get node <node-name> -o json | \
  jq '.status.conditions[] | select(.type | startswith("Kernel")) | {type, lastHeartbeatTime, status}'

The lastHeartbeatTime should be recent (within the last few minutes). A stale heartbeat — even with status: "False" — indicates NPD has stopped reporting.

3. Kernel Issue Detection Rate Monitoring

Expose NPD's Prometheus metrics (port 20257 by default) to track problem detection rates:

# Rate of problem events being generated
rate(problem_counter[5m])

# Gauge for currently active problems
problem_gauge{type="KernelDeadlock"}
problem_gauge{type="ReadonlyFilesystem"}

# NPD plugin scrape success rate
node_problem_detector_plugin_active

Alert when problem_gauge for critical conditions is non-zero:

- alert: NodeKernelDeadlock
  expr: problem_gauge{type="KernelDeadlock"} > 0
  for: 1m
  labels:
    severity: critical
  annotations:
    summary: "Kernel deadlock detected on node {{ $labels.node }}"
    description: "Node Problem Detector has detected a kernel deadlock. The node may need to be drained and rebooted."

4. Condition Reporting Health

Each NPD pod reports NodeConditions to the API server. If a pod loses API server connectivity, conditions become stale — the node appears healthy even if NPD has detected problems.

Monitor this by checking condition heartbeat age:

# Check all NPD-managed conditions across the cluster for staleness
kubectl get nodes -o json | jq -r '
  .items[] | .metadata.name as $node |
  .status.conditions[] |
  select(.type | test("Kernel|Memory|Disk|PID|NetworkUnavailable")) |
  [$node, .type, .lastHeartbeatTime, .status] | @tsv
' | sort -k3

Look for lastHeartbeatTime values older than 5-10 minutes on a running cluster.

5. Event Generation Verification

NPD should generate Kubernetes Events when problems are detected. Verify events are flowing:

# Check for recent NPD-generated events
kubectl get events --all-namespaces --field-selector reason=NodeConditionMismatch \
  --sort-by='.lastTimestamp' | tail -20

# Or watch events from the node-problem-detector source
kubectl get events -n kube-system --field-selector source=node-problem-detector \
  --sort-by='.lastTimestamp'

Prometheus Alerting Rules

Deploy a full alerting suite for NPD health:

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: node-problem-detector-alerts
  namespace: monitoring
spec:
  groups:
  - name: node-problem-detector
    rules:
    - alert: NPDDaemonSetNotFullyRolledOut
      expr: |
        kube_daemonset_status_desired_number_scheduled{daemonset="node-problem-detector"}
        != kube_daemonset_status_number_ready{daemonset="node-problem-detector"}
      for: 10m
      labels:
        severity: warning
      annotations:
        summary: "Node Problem Detector DaemonSet is not fully ready"
        description: "{{ $value }} NPD pod(s) are not ready. Node observability is degraded."

    - alert: NPDPodNotRunning
      expr: |
        kube_pod_status_phase{namespace="kube-system",
          pod=~"node-problem-detector.*",
          phase!="Running"} > 0
      for: 5m
      labels:
        severity: critical
      annotations:
        summary: "Node Problem Detector pod is not running on {{ $labels.pod }}"

    - alert: NPDConditionStaleness
      expr: |
        (time() - kube_node_status_condition_last_transition_time{
          condition=~"KernelDeadlock|ReadonlyFilesystem"}) > 600
      for: 5m
      labels:
        severity: warning
      annotations:
        summary: "NPD condition heartbeat is stale on node {{ $labels.node }}"
        description: "The {{ $labels.condition }} condition has not been updated in over 10 minutes."

    - alert: NodeKernelDeadlockDetected
      expr: problem_gauge{type="KernelDeadlock"} > 0
      for: 1m
      labels:
        severity: critical
      annotations:
        summary: "Kernel deadlock on {{ $labels.node }}"

Custom Problem Detection Rules

NPD supports custom system-log monitors. Here's an example config that detects containerd failures — often missed by default NPD configurations:

{
  "plugin": "filelog",
  "pluginConfig": {
    "timestamp": "^time=\"(\\S*)\"",
    "message": "msg=\"([^\"]*)\"",
    "timestampFormat": "2006-01-02T15:04:05.999999999Z07:00"
  },
  "logPath": "/var/log/containers/containerd.log",
  "lookback": "5m",
  "bufferSize": 10,
  "source": "containerd-monitor",
  "conditions": [
    {
      "type": "ContainerdHang",
      "reason": "ContainerdHangDetected",
      "message": "Containerd is not responding"
    }
  ],
  "rules": [
    {
      "type": "permanent",
      "condition": "ContainerdHang",
      "reason": "ContainerdHangDetected",
      "pattern": "containerd is not responding"
    }
  ]
}

Integrating Vigilmon for External NPD Coverage

NPD's Prometheus metrics endpoint (:20257/metrics) can be exposed via a Service and monitored externally with Vigilmon. This catches cases where internal Prometheus monitoring itself fails:

  1. Create a NodePort or Ingress for NPD's metrics endpoint (restrict access with network policies).
  2. Add a Vigilmon HTTP monitor targeting https://your-cluster-metrics-proxy.yourdomain.com/node-problem-detector/metrics.
  3. Set a keyword assertion for problem_gauge — this string appears in every healthy NPD metrics response and confirms the scrape succeeded.
  4. 1-minute check interval — NPD failures compound quickly as scheduling decisions are made on stale conditions.
  5. Alert to your on-call rotation — NPD failures often indicate node-level problems that require immediate triage.

For simpler setups, expose a lightweight health endpoint from an aggregator that checks DaemonSet readiness and reports HTTP 200 (all ready) or 503 (degraded), then monitor that with Vigilmon.

Monitoring Checklist

| Signal | Check Method | Alert When | |--------|-------------|------------| | DaemonSet readiness | DESIRED vs READY count | Any pod not ready >10m | | Pod phase | kube_pod_status_phase | Non-Running >5m | | Condition heartbeat age | lastHeartbeatTime | Stale >10m | | Log scraping errors | NPD pod logs | Any "cannot open" errors | | Active kernel conditions | problem_gauge | KernelDeadlock > 0 | | Event generation | kubectl get events | No events for unusual duration |

Conclusion

Node Problem Detector is a critical layer in your cluster's self-healing stack — but it's only effective when it's running correctly on every node. Monitoring NPD's DaemonSet health, condition reporting freshness, and log scraping activity ensures you don't have hidden blind spots. Combine Prometheus alerting for internal failures with Vigilmon's external monitoring for the metrics endpoint, and you get complete visibility into your cluster's node observability layer — so node problems get detected and acted on, rather than silently accumulating.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles