How to Monitor SR-IOV Network Operator with Vigilmon (HTTP + TCP + Alerts)

SR-IOV Network Operator is a Kubernetes operator that automates the provisioning and lifecycle management of SR-IOV (Single Root I/O Virtualization) network devices and virtual functions (VFs) across every node in the cluster. It configures SR-IOV device plugins, network policies, and VF drivers so that latency-sensitive pods — NFV workloads, financial data planes, HPC applications — can bypass the kernel network stack and communicate at near-native NIC speeds. If the operator, its webhook server, or the device plugin DaemonSet becomes unavailable, VF allocation stops and all SR-IOV-dependent pods queue indefinitely with Insufficient resource errors. Vigilmon adds the external perspective that SR-IOV Network Operator cannot provide for itself: continuous uptime and latency checks from outside your infrastructure that keep watching even when the operator is the component that has failed. You can add your first monitor for free, no credit card required.

Why SR-IOV Network Operator needs external monitoring

Operator webhook failure blocks all SR-IOV pod admissions. SR-IOV Network Operator registers an admission webhook that validates and mutates pod specs that request SR-IOV resources via k8s.v1.cni.cncf.io/networks annotations. If the webhook server crashes or becomes unresponsive, the Kubernetes API server cannot complete the admission review for any pod that references an SR-IOV network, causing pod creation to fail with webhook timeout errors. Unlike most operator failures which degrade existing workloads gradually, a webhook failure immediately blocks all new SR-IOV pod scheduling across the entire cluster. An external HTTP monitor targeting the webhook health endpoint detects this failure within one check interval.
Device plugin DaemonSet crash removes VF visibility from the kubelet. The SR-IOV device plugin runs as a DaemonSet on every SR-IOV-capable node and advertises available VF capacity to the kubelet via the Kubernetes device plugin API. If the device plugin pod crashes on a node — due to an OOM kill, a socket error, or a missing /var/lib/kubelet/device-plugins mount — the kubelet on that node loses all awareness of available VFs. Pods requesting SR-IOV resources cannot be scheduled on the affected node, and no Kubernetes cluster-level event indicates why. An external monitor on the device plugin health port surfaces this failure mode faster than cluster-level scheduling failure rates reveal it.
SriovNetworkNodeState Failed sync halts VF configuration on a node. The operator continuously reconciles SriovNetworkNodeState resources that represent the desired VF configuration on each node. If the reconciler encounters an invalid VF count (requesting more VFs than the physical NIC supports), an unsupported driver binding, or a kernel module loading failure, the node's SriovNetworkNodeState enters a Failed sync status. The node continues running existing VF workloads, but no new VF configuration can be applied and the operator cannot automatically recover without manual intervention. An external monitor on the operator's health endpoint detects this state because the operator's health check reflects reconciliation errors for its managed nodes.
VF pool exhaustion causes latency-sensitive pod scheduling to stall silently. Each SR-IOV-capable NIC has a fixed number of virtual functions — typically 16 to 64 per physical function. Under high pod churn, during cluster-wide rolling updates, or after ungraceful node drains that leave VFs in an allocated state, the pool can become exhausted. New pods requesting VFs are queued by the Kubernetes scheduler with Insufficient intel.com/sriov_net events, but there is no cluster-level alert and no automatic recovery mechanism. The SR-IOV device plugin metrics endpoint exposes per-resource VF capacity, and an external Vigilmon monitor on this endpoint surfaces pool exhaustion before deployment pipelines begin stalling.
Driver binding regression after kernel upgrade silently bypasses SR-IOV. After a node kernel upgrade, VF driver bindings may revert to the default kernel driver (e.g. ixgbevf or mlx5_core) instead of the DPDK-compatible vfio-pci driver required for DPDK-based NFV workloads. The VF is allocated to the pod, and the pod starts successfully, but it uses a slower kernel-mode driver path instead of the DPDK data plane. The performance degradation — typically an order of magnitude in throughput and latency — does not trigger any Kubernetes health check failure. An external HTTP monitor on the operator's webhook and health endpoints, combined with node condition monitoring, can detect the driver mismatch because the operator reports driver binding errors in its status API.

What you'll need

A running SR-IOV Network Operator installation with its webhook and health endpoints accessible
The public hostname or IP for your SR-IOV Network Operator deployment, for example https://sriov-operator.example.com
A free Vigilmon account — monitors start running in under a minute with no credit card required

Step 1: Expose SR-IOV Network Operator's health endpoint

Verify the operator's health and webhook endpoints are reachable before configuring Vigilmon monitors.

# Check SR-IOV Network Operator pod status
kubectl get pods -n sriov-network-operator

# Sample output:
# NAME                                    READY   STATUS    RESTARTS   AGE
# sriov-network-operator-7d8f9c-xkp2l    1/1     Running   0          2d
# sriov-device-plugin-ds-h2x4n           1/1     Running   0          2d
# sriov-device-plugin-ds-wq7rl           1/1     Running   0          2d

# Check operator health endpoint
curl -k https://sriov-operator.example.com/healthz

# Sample output: {"status": "ok"}

# Check webhook server health
curl -k https://sriov-operator.example.com:9443/healthz

# Check the SriovNetworkNodeState for each node
kubectl get sriovnetworknodestates -n sriov-network-operator -o wide

# Sample output:
# NAME    SYNC STATUS   DESIRED SYNC STATE
# node1   Succeeded     Idle
# node2   Succeeded     Idle

Verify the SR-IOV device plugin and operator are running on the expected nodes:

# Check device plugin DaemonSet rollout
kubectl get daemonset -n sriov-network-operator sriov-device-plugin-ds -o wide

# Verify VF capacity is advertised on SR-IOV nodes
kubectl describe node <sriov-node-name> | grep -A 5 "intel.com/sriov"

# Sample output:
# Capacity:
#   intel.com/intel_sriov_netdevice: 8
# Allocatable:
#   intel.com/intel_sriov_netdevice: 6

# Check operator admission webhook registration
kubectl get validatingwebhookconfigurations | grep sriov
kubectl get mutatingwebhookconfigurations | grep sriov

Step 2: Set up HTTP monitoring in Vigilmon

Add an HTTP monitor for the SR-IOV Network Operator health endpoint:

Log in to vigilmon.online and click Add Monitor.
Set Monitor type to HTTP.
Enter the URL: https://sriov-operator.example.com/healthz
Set Check interval to 60 seconds and Expected status code to 200.
Click Save Monitor — Vigilmon will run the first check within a minute.

Add a monitor for the webhook server health endpoint

The admission webhook is the most critical component — its failure immediately blocks all SR-IOV pod admissions:

Click Add Monitor again.
Set Monitor type to HTTP.
Enter the URL: https://sriov-operator.example.com:9443/healthz
Set Expected status code to 200 and Timeout to 10 seconds.
Set a Response time threshold of 5000ms to detect webhook processing slowdowns before the Kubernetes API server begins timing out admission reviews.
Click Save Monitor.

Why external monitoring catches what internal checks miss

SR-IOV Network Operator sits at the intersection of hardware-level NIC configuration and Kubernetes pod scheduling. When the operator fails, the symptoms appear in pod scheduling — Insufficient resource errors, pods stuck in Pending, deployment pipelines stalling — not as a monitoring dashboard alert. Kubernetes liveness probes confirm whether the operator container is running, but they cannot detect whether the admission webhook is responding within the Kubernetes timeout, whether node reconciliation has entered a failed state, or whether VF pool exhaustion is approaching. Vigilmon operates entirely outside your infrastructure and surfaces operator health issues before hardware resource shortages cascade into scheduling failures for your highest-priority workloads.

Step 3: Monitor SR-IOV Network Operator's TCP port

A TCP monitor confirms that the SR-IOV Network Operator webhook server is accepting connections at the network layer before any TLS or HTTP processing begins.

In Vigilmon, click Add Monitor.
Set Monitor type to TCP.
Enter Host: sriov-operator.example.com and Port: 9443.
Set Check interval to 60 seconds.
Click Save Monitor.

If the operator exposes additional service ports, verify them first:

# Check all ports exposed by the SR-IOV operator
kubectl get service -n sriov-network-operator -o wide

# Find operator container ports
kubectl get deployment -n sriov-network-operator sriov-network-operator \
  -o jsonpath='{.spec.template.spec.containers[0].ports}' | jq .

# Sample output:
# [
#   {"containerPort": 9443, "name": "webhook",  "protocol": "TCP"},
#   {"containerPort": 8080, "name": "metrics",  "protocol": "TCP"},
#   {"containerPort": 8081, "name": "health",   "protocol": "TCP"}
# ]

Add a TCP monitor for port 8080 to independently confirm the metrics port is open, which Prometheus scrapes for VF allocation capacity metrics.

Step 4: Configure alert channels

Email alerts

In Vigilmon, go to Alert Channels and click Add Channel.
Choose Email and enter the address that should receive notifications (for example, platform-team@example.com or your network engineering on-call rotation).
Assign this channel to all SR-IOV Network Operator monitors created in Steps 2 and 3.
Click Save — Vigilmon will send a test email to confirm delivery.

Webhook alerts

For Slack, PagerDuty, or an internal incident management system, add a webhook channel:

In Alert Channels, click Add Channel and choose Webhook.
Paste your endpoint URL (a Slack incoming webhook, PagerDuty Events API v2 endpoint, or custom URL).
Vigilmon will POST the following JSON payload when a monitor transitions to down:

{
  "monitor_id": "mon_sriov_operator_healthz",
  "monitor_name": "SR-IOV Network Operator /healthz",
  "status": "down",
  "previous_status": "up",
  "checked_at": "2026-07-12T09:14:22Z",
  "response_time_ms": null,
  "status_code": null,
  "error": "Connection refused",
  "url": "https://sriov-operator.example.com/healthz"
}

Assign all SR-IOV Network Operator monitors to this channel and click Save.

Runbook: diagnostics to run when an alert fires

#!/bin/bash
# SR-IOV Network Operator alert runbook — run these in order when Vigilmon fires

echo "=== Operator pod status ==="
kubectl get pods -n sriov-network-operator -o wide

echo "=== Device plugin DaemonSet status ==="
kubectl get daemonset -n sriov-network-operator -o wide

echo "=== Recent events ==="
kubectl get events -n sriov-network-operator --sort-by='.lastTimestamp' | tail -20

echo "=== Health endpoint response ==="
curl -kv https://sriov-operator.example.com/healthz 2>&1 | tail -20

echo "=== Webhook health ==="
curl -kv https://sriov-operator.example.com:9443/healthz 2>&1 | tail -10

echo "=== SriovNetworkNodeState sync status ==="
kubectl get sriovnetworknodestates -n sriov-network-operator -o wide

echo "=== VF capacity per node ==="
kubectl get nodes -o custom-columns='NAME:.metadata.name,SRIOV:.status.allocatable.intel\.com/intel_sriov_netdevice'

echo "=== Pods pending with insufficient SR-IOV resources ==="
kubectl get pods -A | grep Pending
kubectl get events -A | grep -i "insufficient.*sriov\|sriov.*insufficient" | tail -10

echo "=== Port connectivity ==="
nc -zv sriov-operator.example.com 9443
nc -zv sriov-operator.example.com 8080

echo "=== TLS certificate expiry ==="
echo | openssl s_client -connect sriov-operator.example.com:9443 \
  -servername sriov-operator.example.com 2>/dev/null \
  | openssl x509 -noout -enddate

echo "=== Operator logs (last 100 lines) ==="
kubectl logs -n sriov-network-operator \
  -l app=sriov-network-operator --tail=100

echo "=== Webhook configuration ==="
kubectl get mutatingwebhookconfigurations | grep sriov
kubectl describe mutatingwebhookconfiguration sriov-operator-webhook-config 2>/dev/null \
  | grep -A5 "Failure Policy"

Step 5: Create a public status page

A Vigilmon status page gives your platform engineering team and NFV/HPC workload teams a single URL to check SR-IOV Network Operator health during incidents — no cluster access required.

In Vigilmon, navigate to Status Pages and click New Status Page.
Give it a name such as SR-IOV Network Operator and choose a subdomain like sriov-status.vigilmon.page.
Add all monitors created in this tutorial: the operator /healthz HTTP monitor, the webhook /healthz HTTP monitor, and the TCP port monitor.
Optionally add a description explaining that the SR-IOV Network Operator manages hardware VF allocation for latency-sensitive workloads and that downtime affects all SR-IOV pod scheduling.
Click Publish — the page is immediately accessible and updates in real time.

Share this URL in your team's incident response runbook and link it from your NFV workload documentation. When data plane pods fail to schedule with Insufficient intel.com/sriov_net errors, the status page is the first place to check whether the operator and device plugin are healthy.

Putting it all together

| Monitor | Type | What it catches | |---|---|---| | https://sriov-operator.example.com/healthz | HTTP | Operator crash, reconciliation failure, SriovNetworkNodeState errors | | https://sriov-operator.example.com:9443/healthz | HTTP | Webhook failure blocking SR-IOV pod admissions, TLS expiry | | sriov-operator.example.com:9443 | TCP | Network-level webhook port connectivity loss, TLS listener crash |

# Quick-reference diagnostic one-liners when a Vigilmon alert fires

# 1. Are SR-IOV operator pods running?
kubectl get pods -n sriov-network-operator -o wide

# 2. Is the operator health endpoint responding?
curl -kf https://sriov-operator.example.com/healthz || echo "Health check FAILED"

# 3. Is the webhook endpoint responding?
curl -kf https://sriov-operator.example.com:9443/healthz || echo "Webhook FAILED"

# 4. Are any SriovNetworkNodeStates in Failed status?
kubectl get sriovnetworknodestates -n sriov-network-operator \
  -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.status.syncStatus}{"\n"}{end}'

# 5. How many VFs are available per node?
kubectl get nodes -o custom-columns='NODE:.metadata.name,VF_ALLOC:.status.allocatable.intel\.com/intel_sriov_netdevice'

# 6. Are any pods pending with insufficient SR-IOV resources?
kubectl get events -A | grep -i "insufficient.*sriov"

# 7. Is port 9443 open?
nc -zv sriov-operator.example.com 9443 && echo "Port open" || echo "Port CLOSED"

# 8. How long until the webhook TLS cert expires?
echo | openssl s_client -connect sriov-operator.example.com:9443 \
  -servername sriov-operator.example.com 2>/dev/null \
  | openssl x509 -noout -enddate

What's next

SSL certificate monitoring — Vigilmon can alert you days before the TLS certificate on sriov-operator.example.com:9443 expires. A webhook TLS certificate expiry causes the Kubernetes API server to reject all SR-IOV admission reviews, immediately blocking all new SR-IOV pod creation. Add an SSL monitor pointing at the webhook hostname to get advance warning before this happens.
Heartbeat monitoring for VF reconciliation — Configure a Kubernetes CronJob that creates a test SR-IOV pod with a single VF request, verifies it receives a VF allocation, then deletes it and verifies the VF is released. Have it POST a heartbeat to a Vigilmon dead-man's-switch endpoint. If the VF allocation pipeline stops processing, the heartbeat stops and Vigilmon fires an alert before your latency-sensitive workloads begin failing to schedule.
Latency trend monitoring — Use Vigilmon's response time history to establish a latency baseline for the SR-IOV webhook health endpoint. Sudden latency increases on the webhook port often indicate that the webhook server is under high load from concurrent pod admission reviews during a large rolling deployment, giving you time to scale the operator before webhook timeouts start rejecting pod creations.

Get started free at vigilmon.online — no credit card, monitors start running in under a minute.