tutorial

Monitoring NVIDIA GPU Operator with Vigilmon: Driver Health, Device Plugin Alerts & GPU Workload Status

How to monitor the NVIDIA GPU Operator's driver daemonset, container toolkit, and device plugin with Vigilmon — catching GPU infrastructure failures before they silently break ML training jobs, inference workloads, and GPU-accelerated services.

The NVIDIA GPU Operator automates the full GPU software stack on Kubernetes — deploying GPU drivers, the container toolkit (nvidia-container-runtime), device plugins, DCGM exporters, MIG configurators, and validation workloads across your GPU nodes. Every ML training job, inference server, and GPU-accelerated service depends on this operator being healthy. When the GPU Operator works correctly, GPU resources appear in Kubernetes and workloads schedule onto them transparently. When it fails, one of two things happens: either GPU resources disappear from nodes and all GPU workloads are rejected (visible immediately), or a component partially fails — drivers load but the device plugin stops running, leaving GPU nodes appearing schedulable while all GPU pods fail at runtime. Vigilmon monitors the GPU Operator's health endpoints from outside the cluster, alerting your ML infrastructure team the moment either failure mode appears.

What You'll Build

  • HTTP monitors for GPU Operator controller health
  • HTTP monitors for the DCGM exporter metrics endpoint (proving GPU telemetry is flowing)
  • A heartbeat monitor to verify GPU device plugin DaemonSet coverage
  • Tiered alert routing: device plugin failures page on-call, operator degradation notifies the ML infrastructure team

Prerequisites

  • NVIDIA GPU Operator installed in a Kubernetes cluster (version 23.x or 24.x)
  • At least one GPU node in the cluster
  • An Ingress controller exposing GPU Operator health endpoints over HTTPS
  • A free account at vigilmon.online

Why GPU Operator Monitoring Is Not Optional

The GPU Operator manages a complex, multi-component software stack where failures compound:

Device plugin failure makes GPUs invisible to Kubernetes. The nvidia-device-plugin DaemonSet is what exposes nvidia.com/gpu as a schedulable resource. If this DaemonSet has a pod crash on even one GPU node, that node's GPUs disappear from the Kubernetes resource model. Any job scheduled there fails with 0/1 nodes are available: 1 Insufficient nvidia.com/gpu. But the node still appears Ready — your infra looks healthy while your ML jobs sit in Pending.

Driver DaemonSet failure on one node crashes all GPU pods on that node. The GPU driver DaemonSet loads kernel modules. A failed driver pod means the GPU is inaccessible to all containers on that node. Running training jobs crash immediately with CUDA errors. This looks like a workload bug, not an infrastructure failure, and can take hours to diagnose.

MIG misconfiguration silently fragments GPU capacity. For Multi-Instance GPU (MIG) configurations, the operator manages MIG partitioning. A MIG reconfiguration failure leaves GPUs in an inconsistent partition state — some slices visible, others not — causing erratic scheduling behavior that's nearly impossible to debug without external monitoring.

Operator controller failure stops automated remediation. The GPU Operator controller watches for node changes and ensures the full software stack is running. If the controller crashes, it stops reacting to node additions or failures. New GPU nodes added to the cluster won't get drivers or device plugins. Failed components won't be restarted. The cluster silently loses GPU capacity over time.

Step 1: Verify GPU Operator Health Endpoints

# Check GPU Operator controller health
kubectl exec -n gpu-operator deploy/gpu-operator -- \
  wget -qO- http://localhost:8080/healthz

# List all GPU Operator DaemonSets
kubectl get daemonsets -n gpu-operator-resources

# Verify device plugin is running on all GPU nodes
kubectl get pods -n gpu-operator-resources \
  -l app=nvidia-device-plugin-daemonset --no-headers

# Check DCGM exporter is running
kubectl get pods -n gpu-operator-resources \
  -l app=nvidia-dcgm-exporter --no-headers

# Verify GPUs are visible to Kubernetes
kubectl get nodes -o json | jq '.items[] | select(.status.capacity["nvidia.com/gpu"] != null) | {name: .metadata.name, gpus: .status.capacity["nvidia.com/gpu"]}'

DCGM exporter exposes Prometheus metrics on port 9400:

DCGM_POD=$(kubectl get pods -n gpu-operator-resources \
  -l app=nvidia-dcgm-exporter -o jsonpath='{.items[0].metadata.name}')
kubectl exec -n gpu-operator-resources $DCGM_POD -- \
  wget -qO- http://localhost:9400/metrics | head -20

Step 2: Expose Health Endpoints via Ingress

# gpu-operator-health-services.yaml
---
apiVersion: v1
kind: Service
metadata:
  name: gpu-operator-health
  namespace: gpu-operator
spec:
  selector:
    app.kubernetes.io/name: gpu-operator
  ports:
    - name: health
      port: 8080
      targetPort: 8080
---
apiVersion: v1
kind: Service
metadata:
  name: dcgm-exporter-metrics
  namespace: gpu-operator-resources
spec:
  selector:
    app: nvidia-dcgm-exporter
  ports:
    - name: metrics
      port: 9400
      targetPort: 9400
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: gpu-operator-health
  namespace: gpu-operator
spec:
  rules:
    - host: gpu-ops-health.internal.yourdomain.com
      http:
        paths:
          - path: /operator/healthz
            pathType: Prefix
            backend:
              service:
                name: gpu-operator-health
                port:
                  number: 8080
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: dcgm-metrics
  namespace: gpu-operator-resources
spec:
  rules:
    - host: gpu-ops-health.internal.yourdomain.com
      http:
        paths:
          - path: /dcgm/metrics
            pathType: Prefix
            backend:
              service:
                name: dcgm-exporter-metrics
                port:
                  number: 9400

Apply and verify:

kubectl apply -f gpu-operator-health-services.yaml
curl https://gpu-ops-health.internal.yourdomain.com/operator/healthz
# Expected: 200 OK
curl https://gpu-ops-health.internal.yourdomain.com/dcgm/metrics | grep DCGM_FI_DEV_GPU_UTIL
# Expected: DCGM metrics output with GPU utilization

Step 3: Add Vigilmon HTTP Monitors

In the Vigilmon dashboard:

  1. Go to Add Monitor → HTTP
  2. Configure the following monitors:

| Monitor Name | URL | Priority | |---|---|---| | gpu operator controller | https://gpu-ops-health.yourdomain.com/operator/healthz | Critical | | dcgm exporter metrics | https://gpu-ops-health.yourdomain.com/dcgm/metrics | Critical |

Settings for all monitors:

  • Check interval: 1 minute
  • Timeout: 15 seconds
  • Expected status code: 200
  • Alert after: 2 consecutive failures

For the DCGM exporter monitor, add a keyword check for DCGM_FI_DEV_GPU_UTIL — this confirms the exporter is not only running but actively collecting GPU telemetry. An empty metrics response (exporter running but DCGM daemon failed) would pass a status-code-only check.

Step 4: Monitor GPU Resource Availability

Track that GPU resources remain visible to Kubernetes:

# Check allocatable GPU count across the cluster
kubectl get nodes -o json | jq '[.items[] | .status.allocatable["nvidia.com/gpu"] // "0" | tonumber] | add'

# Find GPU nodes where device plugin is missing
kubectl get nodes -l nvidia.com/gpu.present=true -o name | while read node; do
  PODS=$(kubectl get pods -n gpu-operator-resources \
    --field-selector spec.nodeName=${node#node/} \
    -l app=nvidia-device-plugin-daemonset --no-headers 2>/dev/null | wc -l)
  [ "$PODS" -eq 0 ] && echo "MISSING device plugin on $node"
done

For clusters using NVIDIA MIG, verify MIG partitions are in the expected state:

# Check MIG configuration status
kubectl get clusterpolicies -o jsonpath='{.items[0].status.state}'
# Expected: "ready"

Step 5: Heartbeat Monitor for GPU Stack Completeness

Create a CronJob that verifies every GPU node has the complete software stack running:

# gpu-operator-heartbeat.yaml
apiVersion: batch/v1
kind: CronJob
metadata:
  name: gpu-operator-heartbeat
  namespace: gpu-operator
spec:
  schedule: "*/5 * * * *"
  jobTemplate:
    spec:
      template:
        spec:
          restartPolicy: OnFailure
          serviceAccountName: gpu-operator-heartbeat
          containers:
            - name: heartbeat
              image: bitnami/kubectl:latest
              env:
                - name: HEARTBEAT_URL
                  valueFrom:
                    secretKeyRef:
                      name: vigilmon-secrets
                      key: gpu-operator-heartbeat-url
              command:
                - /bin/sh
                - -c
                - |
                  set -e
                  # Check GPU Operator ClusterPolicy is ready
                  POLICY_STATE=$(kubectl get clusterpolicies gpu-cluster-policy \
                    -o jsonpath='{.status.state}' 2>/dev/null)
                  if [ "$POLICY_STATE" != "ready" ]; then
                    echo "FAIL: ClusterPolicy state is '$POLICY_STATE' (expected 'ready')"
                    exit 1
                  fi
                  # Verify device plugin is running on all GPU nodes
                  GPU_NODES=$(kubectl get nodes \
                    -l nvidia.com/gpu.present=true \
                    --no-headers | wc -l)
                  DEVICE_PLUGIN_PODS=$(kubectl get pods -n gpu-operator-resources \
                    -l app=nvidia-device-plugin-daemonset \
                    --field-selector=status.phase=Running \
                    --no-headers | wc -l)
                  if [ "$DEVICE_PLUGIN_PODS" -lt "$GPU_NODES" ]; then
                    echo "FAIL: Device plugin running on $DEVICE_PLUGIN_PODS of $GPU_NODES GPU nodes"
                    exit 1
                  fi
                  # Verify DCGM exporter is running
                  DCGM_PODS=$(kubectl get pods -n gpu-operator-resources \
                    -l app=nvidia-dcgm-exporter \
                    --field-selector=status.phase=Running \
                    --no-headers | wc -l)
                  if [ "$DCGM_PODS" -eq 0 ]; then
                    echo "FAIL: No DCGM exporter pods running"
                    exit 1
                  fi
                  # Verify GPU resources are allocatable
                  GPU_CAPACITY=$(kubectl get nodes -o json | \
                    python3 -c "import sys,json; data=json.load(sys.stdin); print(sum(int(n['status'].get('allocatable',{}).get('nvidia.com/gpu','0')) for n in data['items']))")
                  if [ "$GPU_CAPACITY" -eq 0 ]; then
                    echo "FAIL: No allocatable GPUs found in cluster"
                    exit 1
                  fi
                  wget -qO- "$HEARTBEAT_URL"
                  echo "GPU Operator heartbeat sent — $GPU_CAPACITY GPUs allocatable"

Create RBAC and secret:

kubectl create serviceaccount gpu-operator-heartbeat -n gpu-operator
kubectl create clusterrolebinding gpu-operator-heartbeat-reader \
  --clusterrole=view \
  --serviceaccount=gpu-operator:gpu-operator-heartbeat

kubectl create secret generic vigilmon-secrets \
  --from-literal=gpu-operator-heartbeat-url='https://vigilmon.online/heartbeat/YOUR_TOKEN' \
  -n gpu-operator

In Vigilmon, create a Heartbeat monitor:

  • Name: gpu operator stack health
  • Expected interval: 5 minutes
  • Grace period: 3 minutes

Step 6: Monitor GPU Driver and Validator Jobs

The GPU Operator runs a validation job after installing drivers to confirm CUDA works:

# Check validator status
kubectl get pods -n gpu-operator-resources -l app=nvidia-operator-validator

# Check driver daemonset
kubectl get pods -n gpu-operator-resources -l app=nvidia-driver-daemonset

# Look for crash loops
kubectl get pods -n gpu-operator-resources --no-headers | grep -v "Running\|Completed"

Add a monitoring step to your heartbeat CronJob:

# Add to heartbeat command:
# Verify no critical GPU Operator pods are crash-looping
CRASHLOOP_PODS=$(kubectl get pods -n gpu-operator-resources --no-headers | \
  grep CrashLoopBackOff | wc -l)
if [ "$CRASHLOOP_PODS" -gt 0 ]; then
  echo "WARN: $CRASHLOOP_PODS GPU Operator pods in CrashLoopBackOff"
  exit 1
fi

Step 7: Configure Alert Routing

Route alerts by operational impact:

Critical — Device plugin missing or GPUs not allocatable (page on-call):

  1. Alert Channels → Add Channel → PagerDuty
  2. Assign to: gpu operator stack health heartbeat, dcgm exporter metrics

High — GPU Operator controller degraded (notify ML infra Slack):

  1. Alert Channels → Add Channel → Slack Webhook
  2. Paste your #ml-infra-alerts webhook URL
  3. Assign to: gpu operator controller

GPU workloads represent your most expensive infrastructure — an undetected device plugin failure can leave an entire fleet of GPU nodes idle while training jobs queue up. Page immediately; don't wait for someone to notice slow job throughput.

What You're Now Monitoring

| Component | Monitor Type | Failure Detected | |---|---|---| | GPU Operator controller | HTTP health | Controller crash stops stack management | | DCGM exporter | HTTP + keyword | GPU telemetry pipeline broken | | Device plugin coverage | Heartbeat | GPUs invisible on one or more nodes | | ClusterPolicy state | Heartbeat | Operator in error or reconciling state | | Driver daemonset | Heartbeat | Driver missing from GPU nodes | | Crash-looping components | Heartbeat | Any GPU Operator pod in CrashLoopBackOff | | Allocatable GPU count | Heartbeat | Total GPU capacity drop from expected |

Conclusion

The NVIDIA GPU Operator manages your most expensive and specialized infrastructure — GPU nodes represent thousands of dollars per hour in compute capacity. A silent device plugin failure means that capacity sits idle while ML jobs queue. A driver crash means running training jobs terminate abruptly, wasting hours of compute. Vigilmon gives your ML infrastructure team the external monitoring layer to detect GPU Operator failures within minutes, long before job failures, slow throughput, or irate data scientists surface the problem.

Start monitoring the NVIDIA GPU Operator for free at vigilmon.online. Your first monitor is running in under two minutes.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles