tutorial

Monitoring NMState with Vigilmon

NMState provides declarative network state management for Linux — learn how to monitor applied network configurations, interface health, and connectivity verification with Vigilmon so drift and failures get caught immediately.

NMState is a declarative API for managing network configurations on Linux systems. You define the desired network state in YAML, and NMState applies it via NetworkManager or other backends — bonds, VLANs, bridges, routes, DNS, and more. It's the foundation for OpenShift and RHEL node network configuration. But declarative configuration management has a subtle failure mode: the desired state gets applied once, and then drift or hardware failures can silently move the actual state away from it with no automatic alerts. Vigilmon adds that alerting layer, catching interface failures, connectivity loss, and configuration drift before they cascade into application outages.

What You'll Set Up

  • HTTP monitors for services depending on NMState-managed interfaces
  • TCP connectivity checks for network segments defined by NMState
  • Cron heartbeat monitors for configuration drift detection scripts
  • Custom health endpoint exposing NMState interface status
  • SSL certificate monitoring for management endpoints

Prerequisites

  • NMState 2.0+ installed (dnf install nmstate on RHEL/Fedora, or pip install nmstate)
  • NetworkManager running and managing at least one interface
  • A free Vigilmon account

Verify NMState is operational:

nmstatectl show
# Should output current network state as YAML

Step 1: Monitor Services Depending on NMState Interfaces

NMState manages the network interfaces that your services use to communicate. The most direct signal of NMState health is whether the services relying on those interfaces are reachable.

For each critical service bound to a NMState-managed interface:

  1. Log in to vigilmon.online and click Add Monitor.
  2. Set Type to HTTP / HTTPS.
  3. Enter the service URL: https://api.example.com/health.
  4. Set Check interval to 1 minute.
  5. Click Save.

If the interface goes down — due to a failed nmstatectl apply, a bad configuration rollback, or hardware failure — the service becomes unreachable and Vigilmon alerts within one check interval.

For multi-homed servers with NMState managing several interfaces (management, data plane, storage), add monitors for services on each interface independently. A bond failure on the data plane interface should fire a different alert than a management interface failure.

Step 2: TCP Monitors for Network Segment Reachability

NMState often manages complex configurations: VLANs, OVS bridges, and bonded interfaces that route traffic between network segments. Add TCP monitors to verify that key network paths remain reachable after configuration changes.

  1. Click Add Monitor → TCP Port.
  2. Enter the target host IP in the segment reachable via the NMState-managed interface.
  3. Set Port to 22 (SSH), 443, or the relevant service port.
  4. Set Check interval to 1 minute.
  5. Click Save.

This is particularly valuable when NMState manages VLAN interfaces:

# Example NMState VLAN configuration
interfaces:
  - name: eth0.100
    type: vlan
    state: up
    vlan:
      base-iface: eth0
      id: 100
    ipv4:
      enabled: true
      address:
        - ip: 192.168.100.10
          prefix-length: 24

After applying this config with nmstatectl apply vlan.yaml, add a Vigilmon TCP monitor for a host on 192.168.100.0/24. If the VLAN interface disappears after a NetworkManager restart, the monitor catches it.

Step 3: Heartbeat Monitoring for Configuration Drift Detection

NMState's declarative model assumes the applied state stays applied — but hardware events, NetworkManager upgrades, and kernel driver updates can cause drift. A drift detection script that compares desired vs. actual state and pings Vigilmon when they match gives you continuous conformance monitoring.

  1. In Vigilmon, click Add Monitor → Cron Heartbeat.
  2. Set the expected ping interval to 15 minutes.
  3. Copy the heartbeat URL: https://vigilmon.online/heartbeat/YOUR_TOKEN.

Create the drift detection script:

#!/bin/bash
# /usr/local/bin/nmstate-drift-check.sh

DESIRED_STATE="/etc/nmstate/desired-state.yaml"
VIGILMON_URL="https://vigilmon.online/heartbeat/YOUR_TOKEN"
LOG="/var/log/nmstate-drift.log"

# Get current state
CURRENT=$(nmstatectl show --json 2>/dev/null)
if [ $? -ne 0 ]; then
  echo "$(date): nmstatectl show failed" >> "$LOG"
  exit 1
fi

# Verify key interfaces are UP
BOND0_STATE=$(echo "$CURRENT" | python3 -c "
import json, sys
data = json.load(sys.stdin)
ifaces = {i['name']: i for i in data.get('interfaces', [])}
bond = ifaces.get('bond0', {})
print(bond.get('state', 'unknown'))
")

if [ "$BOND0_STATE" = "up" ]; then
  curl -sf "$VIGILMON_URL" > /dev/null
else
  echo "$(date): bond0 state is '$BOND0_STATE', expected 'up'" >> "$LOG"
fi

Schedule every 15 minutes:

*/15 * * * * /usr/local/bin/nmstate-drift-check.sh

Customize the interface name and state checks to match your desired NMState configuration. If the bond goes down or enters a degraded state, the script stops pinging and Vigilmon alerts after the 15-minute window.

Step 4: Custom Health Endpoint for NMState Interface Status

Build a lightweight HTTP service that exposes NMState interface health, giving Vigilmon a rich endpoint to poll:

#!/usr/bin/env python3
# nmstate_health.py
from http.server import HTTPServer, BaseHTTPRequestHandler
import subprocess
import json

CRITICAL_INTERFACES = ['bond0', 'eth0', 'br0']  # adjust to your setup

class Handler(BaseHTTPRequestHandler):
    def do_GET(self):
        if self.path != '/health':
            self.send_response(404)
            self.end_headers()
            return

        try:
            result = subprocess.run(
                ['nmstatectl', 'show', '--json'],
                capture_output=True, text=True, timeout=15
            )
            state = json.loads(result.stdout)
            ifaces = {i['name']: i for i in state.get('interfaces', [])}

            issues = []
            for name in CRITICAL_INTERFACES:
                iface = ifaces.get(name)
                if not iface:
                    issues.append(f'{name}: missing')
                elif iface.get('state') != 'up':
                    issues.append(f"{name}: {iface.get('state', 'unknown')}")

            if issues:
                body = json.dumps({'status': 'degraded', 'issues': issues}).encode()
                self.send_response(503)
            else:
                body = json.dumps({
                    'status': 'ok',
                    'interfaces_up': len(CRITICAL_INTERFACES)
                }).encode()
                self.send_response(200)
        except Exception as e:
            body = json.dumps({'status': 'error', 'message': str(e)}).encode()
            self.send_response(503)

        self.send_header('Content-Type', 'application/json')
        self.end_headers()
        self.wfile.write(body)

    def log_message(self, format, *args):
        pass

if __name__ == '__main__':
    HTTPServer(('127.0.0.1', 9102), Handler).serve_forever()

Run as a systemd unit:

[Unit]
Description=NMState Health Endpoint
After=NetworkManager.service

[Service]
ExecStart=/usr/bin/python3 /usr/local/bin/nmstate_health.py
Restart=always
RestartSec=5

[Install]
WantedBy=multi-user.target

Enable and start:

systemctl enable --now nmstate-health

Add a Vigilmon HTTP monitor for http://localhost:9102/health. When any critical interface drops from up, the endpoint returns 503 and Vigilmon fires an alert within 60 seconds.

Step 5: Monitor the NMState API for Cluster Environments

In OpenShift and Kubernetes environments, NMState runs as the Node Network Configuration Operator, exposing Kubernetes CRD status. You can expose cluster-level NMState health through the Kubernetes API server:

# Check NodeNetworkState across all nodes
kubectl get nodenetworkstate -o json | python3 -c "
import json, sys
data = json.load(sys.stdin)
for item in data['items']:
    name = item['metadata']['name']
    conditions = item.get('status', {}).get('conditions', [])
    for c in conditions:
        if c['type'] == 'Available' and c['status'] != 'True':
            print(f'DEGRADED: {name} - {c[\"message\"]}')
"

Add a Vigilmon cron heartbeat driven by this check to surface node-level network configuration failures in your cluster.

Step 6: Configure Alert Channels

  1. Go to Alert Channels in Vigilmon and connect Slack, email, PagerDuty, or a webhook.
  2. Set Consecutive failures before alert to 1 for the NMState health endpoint — interface failures are binary and don't benefit from wait-and-see logic.
  3. For drift detection heartbeats, set the tolerance to match your drift check interval plus one period (e.g., 30 minutes for a 15-minute check schedule).

Tag monitors with the managed interface name (bond0, br0, eth0.100) so alert messages immediately identify which interface failed and which services are at risk.

Summary

| Monitor | Target | What It Catches | |---|---|---| | HTTP | Service on NMState-managed interface | Interface down, routing failure | | TCP | Key hosts in each managed network segment | VLAN/bond failure | | Cron heartbeat | Drift detection script | Configuration drift from desired state | | HTTP sidecar | NMState interface status endpoint | Critical interface state change |

NMState's declarative model is powerful for provisioning and auditing network configuration — but it doesn't ship with runtime alerting. With Vigilmon monitoring the services bound to each interface, the interfaces themselves via a health sidecar, and configuration drift via heartbeats, you get continuous conformance verification on top of NMState's apply-time guarantees.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles