Monitoring Node Feature Discovery (NFD) with Vigilmon: Controller Health, Node Labeling Alerts & Feature Detection Status

Node Feature Discovery (NFD) is the Kubernetes add-on that automatically detects hardware features and system configuration on each node — CPU capabilities, memory topology, PCIe devices, kernel parameters, and more — then labels nodes with this metadata so workloads can schedule to exactly the right hardware. When NFD is healthy, GPU workloads land on GPU nodes, AVX-512 optimized jobs schedule on capable CPUs, and NUMA-aware applications find nodes with matching memory topology. When it fails, one of two things happens: either nodes lose their feature labels and workloads fail to schedule because no node matches their nodeSelector (visible, but confusing), or NFD stops updating labels while nodes are added, removed, or reconfigured — creating stale labels that route workloads to wrong hardware. Vigilmon monitors NFD's health endpoints from outside the cluster, alerting your platform team before either failure mode disrupts production workloads.

What You'll Build

• HTTP monitors for NFD master controller health
• HTTP monitors for NFD worker DaemonSet health
• A heartbeat monitor to verify node feature labels are being updated
• Tiered alert routing: master failures page on-call, worker degradation notifies the platform team

Prerequisites

• Node Feature Discovery installed in a Kubernetes cluster (version 0.14+)
• NFD deployed with both master and worker components
• An Ingress controller or NodePort exposing NFD health endpoints
• A free account at vigilmon.online

Why NFD Monitoring Is Not Optional

NFD's failure modes have cascading effects across the cluster:

Scheduling failures when labels go stale. NFD labels are the source of truth for hardware-aware scheduling. If NFD workers stop running, nodes that gain new hardware (a new GPU installed, a NIC added) never get those labels. Workloads targeting that hardware sit in Pending indefinitely — your GPU cluster effectively becomes unusable for new jobs without any clear error pointing to NFD.

Wrong-node scheduling from stale labels. When a node is reconfigured or hardware is removed, NFD must update labels to remove capabilities that no longer exist. If the worker DaemonSet is down, those labels persist. A job requiring NVIDIA GPU schedules onto a node where the GPU was removed — it starts, then immediately fails with a device error. This is extremely hard to diagnose without knowing to check NFD health.

NFD master failure blocks feature discovery entirely. The NFD master aggregates feature data from workers and writes NodeFeature objects to the API server. If the master crashes, even healthy workers continue collecting data locally but can't propagate it. No new nodes get labeled. Existing labels stop being refreshed.

Topology Manager and NUMA placement breaks. If your cluster uses Kubernetes Topology Manager for NUMA-aware workloads, it depends on NFD's topology.node.k8s.io labels. A stale or missing topology label causes the Topology Manager to make suboptimal placement decisions — degrading workload performance without any error.

Step 1: Verify NFD Health Endpoints

NFD exposes health endpoints on both master and worker components:

# Check NFD master health
kubectl exec -n node-feature-discovery deploy/nfd-master -- \
  wget -qO- http://localhost:8081/healthz

# Check NFD worker DaemonSet — pick any worker pod
NFD_WORKER=$(kubectl get pods -n node-feature-discovery \
  -l app=nfd-worker -o jsonpath='{.items[0].metadata.name}')
kubectl exec -n node-feature-discovery $NFD_WORKER -- \
  wget -qO- http://localhost:8081/healthz

# Verify node labels are present
kubectl get nodes -o json | jq '.items[] | {name: .metadata.name, nfd_labels: [.metadata.labels | to_entries[] | select(.key | startswith("feature.node.kubernetes.io"))]}'

# Check NFD NodeFeature objects
kubectl get nodefeatures -A

NFD master and worker both expose /healthz on port 8081 by default.

Step 2: Expose Health Endpoints via Ingress

# nfd-health-services.yaml
---
apiVersion: v1
kind: Service
metadata:
  name: nfd-master-health
  namespace: node-feature-discovery
spec:
  selector:
    app: nfd-master
  ports:
    - name: health
      port: 8081
      targetPort: 8081
---
apiVersion: v1
kind: Service
metadata:
  name: nfd-worker-health
  namespace: node-feature-discovery
spec:
  selector:
    app: nfd-worker
  ports:
    - name: health
      port: 8081
      targetPort: 8081
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: nfd-health
  namespace: node-feature-discovery
spec:
  rules:
    - host: nfd-health.internal.yourdomain.com
      http:
        paths:
          - path: /master/healthz
            pathType: Prefix
            backend:
              service:
                name: nfd-master-health
                port:
                  number: 8081
          - path: /worker/healthz
            pathType: Prefix
            backend:
              service:
                name: nfd-worker-health
                port:
                  number: 8081

Apply and verify:

kubectl apply -f nfd-health-services.yaml
curl https://nfd-health.internal.yourdomain.com/master/healthz
# Expected: 200 OK

Step 3: Add Vigilmon HTTP Monitors

In the Vigilmon dashboard:

1. Go to Add Monitor → HTTP
2. Configure the following monitors:

| Monitor Name | URL | Priority | |---|---|---| | nfd master health | https://nfd-health.yourdomain.com/master/healthz | Critical | | nfd worker health | https://nfd-health.yourdomain.com/worker/healthz | High |

Settings for all monitors:

• Check interval: 2 minutes
• Timeout: 10 seconds
• Expected status code: 200
• Alert after: 2 consecutive failures

NFD master failure is your highest priority — without it, feature discovery stops cluster-wide. Worker failures are high-severity because they affect the freshness of hardware labels on every node.

Step 4: Monitor Node Label Freshness and Coverage

Track that feature labels exist and are up to date across your node fleet:

# Count nodes with NFD labels (should equal your total node count)
kubectl get nodes -o json | jq '[.items[] | .metadata.labels | to_entries[] | select(.key | startswith("feature.node.kubernetes.io"))] | length'

# Find nodes with no NFD labels (these are uncovered by NFD worker)
kubectl get nodes -o json | jq '[.items[] | select(.metadata.labels | keys | map(startswith("feature.node.kubernetes.io")) | any | not) | .metadata.name]'

# Check NFD NodeFeatureRule objects that might affect scheduling
kubectl get nodefeaturerules -A

If your GPU cluster uses NVIDIA-specific NFD labels, verify they exist on GPU nodes:

kubectl get nodes -l "feature.node.kubernetes.io/pci-10de.present=true" --no-headers | wc -l
# Should match your GPU node count

Step 5: Heartbeat Monitor for Feature Label Updates

Create a CronJob that verifies NFD is actively updating node labels:

# nfd-label-heartbeat.yaml
apiVersion: batch/v1
kind: CronJob
metadata:
  name: nfd-label-heartbeat
  namespace: node-feature-discovery
spec:
  schedule: "*/10 * * * *"
  jobTemplate:
    spec:
      template:
        spec:
          restartPolicy: OnFailure
          serviceAccountName: nfd-heartbeat
          containers:
            - name: heartbeat
              image: bitnami/kubectl:latest
              env:
                - name: HEARTBEAT_URL
                  valueFrom:
                    secretKeyRef:
                      name: vigilmon-secrets
                      key: nfd-heartbeat-url
              command:
                - /bin/sh
                - -c
                - |
                  set -e
                  # Verify NFD master pod is running
                  MASTER_RUNNING=$(kubectl get pods -n node-feature-discovery \
                    -l app=nfd-master \
                    --field-selector=status.phase=Running \
                    --no-headers | wc -l)
                  if [ "$MASTER_RUNNING" -eq 0 ]; then
                    echo "FAIL: NFD master has no running pods"
                    exit 1
                  fi
                  # Verify NFD workers are running on all schedulable nodes
                  SCHEDULABLE_NODES=$(kubectl get nodes \
                    --field-selector=spec.unschedulable!=true \
                    --no-headers | wc -l)
                  WORKER_PODS=$(kubectl get pods -n node-feature-discovery \
                    -l app=nfd-worker \
                    --field-selector=status.phase=Running \
                    --no-headers | wc -l)
                  if [ "$WORKER_PODS" -lt "$SCHEDULABLE_NODES" ]; then
                    echo "WARN: NFD workers ($WORKER_PODS) fewer than schedulable nodes ($SCHEDULABLE_NODES)"
                    exit 1
                  fi
                  # Verify NodeFeature objects exist (master is writing discoveries)
                  NF_COUNT=$(kubectl get nodefeatures -A --no-headers 2>/dev/null | wc -l)
                  if [ "$NF_COUNT" -eq 0 ]; then
                    echo "FAIL: No NodeFeature objects found — NFD master may be stalled"
                    exit 1
                  fi
                  wget -qO- "$HEARTBEAT_URL"
                  echo "NFD label heartbeat sent"

Create RBAC and secret:

kubectl create serviceaccount nfd-heartbeat -n node-feature-discovery
kubectl create clusterrolebinding nfd-heartbeat-reader \
  --clusterrole=view \
  --serviceaccount=node-feature-discovery:nfd-heartbeat

kubectl create secret generic vigilmon-secrets \
  --from-literal=nfd-heartbeat-url='https://vigilmon.online/heartbeat/YOUR_TOKEN' \
  -n node-feature-discovery

In Vigilmon, create a Heartbeat monitor:

• Name: nfd feature label updates
• Expected interval: 10 minutes
• Grace period: 5 minutes

Step 6: Monitor NodeFeatureRule and Custom Detectors

If you use custom NodeFeatureRule resources or the NFD topology updater, monitor those too:

# Check topology updater status
kubectl get pods -n node-feature-discovery -l app=nfd-topology-updater

# Verify topology labels are present on nodes
kubectl get nodes -o json | jq '.items[] | {name: .metadata.name, topology: .metadata.labels | to_entries[] | select(.key | startswith("topology.node.kubernetes.io"))}'

For clusters using the NFD topology updater, expose its health similarly to the master and add a Vigilmon monitor with the same settings as the master.

Step 7: Configure Alert Routing

Route alerts by impact on workload scheduling:

Critical — NFD master down (page on-call):

1. Alert Channels → Add Channel → PagerDuty
2. Assign to: nfd master health, nfd feature label updates heartbeat

High — NFD workers degraded (notify platform team Slack):

1. Alert Channels → Add Channel → Slack Webhook
2. Paste your #platform-alerts webhook URL
3. Assign to: nfd worker health

Master failure warrants immediate response because it stops all feature discovery. Worker degradation is serious but degrades gradually — the platform team has time to investigate without a full page.

What You're Now Monitoring

| Component | Monitor Type | Failure Detected | |---|---|---| | NFD master | HTTP health | Feature discovery stopped, no new labels applied | | NFD workers (DaemonSet) | HTTP health | Per-node feature detection stalled | | Feature label updates | Heartbeat | Labels not refreshing, stale hardware metadata | | NodeFeature objects | Heartbeat | Master not writing discovery results to API | | Worker-to-node coverage | Heartbeat | Worker missing from schedulable nodes |

Conclusion

Node Feature Discovery is the hardware intelligence layer of Kubernetes — when it works, the scheduler makes smart placement decisions automatically. When it fails, GPU jobs hang in Pending, NUMA-sensitive workloads degrade silently, and your operations team spends hours debugging scheduler behavior before realizing the feature labels are stale. Vigilmon gives your platform team the external vantage point to catch NFD failures the moment they happen.

Start monitoring Node Feature Discovery for free at vigilmon.online. Your first monitor is running in under two minutes.