tutorial

Monitoring OVN-Kubernetes with Vigilmon: CNI Overlay Health, OVN Northbound DB, Gateway Router Connectivity & TLS Certificate Alerts

How to monitor OVN-Kubernetes with Vigilmon — CNI overlay health checks, OVN Northbound and Southbound database availability, gateway router connectivity, pod network reachability, and TLS certificate alerting.

OVN-Kubernetes is a CNI plugin that implements Kubernetes pod networking using Open Virtual Network (OVN), the control plane layer built on top of Open vSwitch (OVS). Unlike overlay-based CNIs that encapsulate packets with VXLAN or GENEVE, OVN-Kubernetes programs OpenFlow rules directly into OVS, enabling hardware-accelerated packet forwarding, native load balancing, and fine-grained network policy enforcement at the hypervisor level. OVN-Kubernetes is the default CNI for OpenShift and is increasingly adopted in bare-metal Kubernetes deployments that require predictable network performance. When the OVN Northbound database becomes unavailable, no new network policies or pod network assignments can be applied — existing pods continue to communicate, but new pod deployments get no network connectivity. When the OVN controller loses sync with the Southbound database, the OpenFlow rules on a node become stale and packet forwarding silently breaks. Vigilmon gives you external visibility into the OVN-Kubernetes stack: application reachability through the OVN data plane, TLS certificate monitoring for API endpoints, and TCP-level probes that catch OVN controller failures before they cascade to application outages.

What You'll Build

  • HTTP monitors on applications running across OVN-Kubernetes nodes to validate overlay routing
  • TCP monitors on OVN database ports to detect control plane failures
  • SSL certificate monitors for services exposed through OVN-Kubernetes load balancers
  • A failure mode table covering OVN controller sync loss, database unavailability, and network policy misconfigurations
  • An alerting runbook mapping Vigilmon alerts to OVN diagnostic commands

Prerequisites

  • A Kubernetes cluster with OVN-Kubernetes installed as the CNI plugin (standalone or via OpenShift)
  • Applications deployed across multiple nodes
  • External access to at least one service (LoadBalancer or NodePort)
  • A free account at vigilmon.online

Step 1: Understand OVN-Kubernetes's Health Surface

OVN-Kubernetes runs several critical components: ovnkube-master (manages the Northbound DB and generates OVN logical resources), ovnkube-node (runs on each node and programs OVS), and the OVN Northbound and Southbound databases. Check their status before configuring external monitors:

# Check OVN-Kubernetes pods
kubectl get pods -n ovn-kubernetes -o wide
# Key pods: ovnkube-master, ovnkube-node (one per node), ovn-northd

# Check OVN Northbound DB leader (in clustered deployments)
kubectl exec -n ovn-kubernetes <ovnkube-master-pod> -- ovn-nbctl get-connection

# Verify OVN controller sync status on a node
kubectl exec -n ovn-kubernetes <ovnkube-node-pod> -- ovn-controller show-trace

# Check OVS bridge configuration
kubectl exec -n ovn-kubernetes <ovnkube-node-pod> -- ovs-vsctl show

The key external health signals are:

  1. Application HTTP responses through the OVN data plane (validates OpenFlow rules are correct)
  2. OVN database port availability (indicates control plane health)
  3. TLS certificates for services exposed through OVN load balancers
  4. TCP connectivity to services to validate OVN load balancer programming

Step 2: Monitor Application HTTP Connectivity Through the OVN Data Plane

The primary external test of OVN-Kubernetes health is an HTTP check on an application that exercises cross-node OVN routing. When OVN-Kubernetes programs OpenFlow rules correctly, pod traffic routes through the OVN logical router with minimal latency. A failed HTTP check indicates either a misconfigured logical router, a broken OVS flow table, or a failed node controller:

# Verify your application is reachable through the OVN data plane
kubectl get svc my-application
curl https://myapp.example.com/health
  1. Log in to Vigilmon → Add Monitor → HTTP.
  2. URL: https://myapp.example.com/health (your application's health endpoint).
  3. Check interval: 60 seconds.
  4. Response timeout: 10 seconds.
  5. Expected status: 200.
  6. Keyword: your health response field (e.g., ok, healthy).
  7. Label: myapp.example.com (OVN-Kubernetes data plane).
  8. Click Save.

Add one HTTP monitor for each service that routes through the OVN-Kubernetes overlay. Because OVN programs per-node OpenFlow rules, a failure on one node's ovnkube-node pod breaks traffic only for pods scheduled on that node — monitoring multiple services helps isolate node-specific failures from cluster-wide control plane failures.

Step 3: Monitor OVN Northbound Database Availability

The OVN Northbound database stores the logical network topology — logical switches, routers, load balancers, and ACLs derived from Kubernetes NetworkPolicy objects. If the Northbound DB becomes unavailable, no new pods receive network configuration and no new NetworkPolicies are enforced. The OVN Northbound database listens on TCP port 6641 by default:

# Verify Northbound DB is listening
kubectl exec -n ovn-kubernetes <ovnkube-master-pod> -- \
  ovn-nbctl --timeout=5 show | head -5

If your OVN DB endpoint is accessible externally (via NodePort or LoadBalancer):

  1. Add Monitor → TCP.
  2. Host: ovn-db.example.com (or node IP hosting the OVN DB).
  3. Port: 6641.
  4. Check interval: 60 seconds.
  5. Label: OVN Northbound DB TCP.
  6. Click Save.

Similarly for the OVN Southbound database (port 6642):

  1. Add Monitor → TCP.
  2. Host: ovn-db.example.com.
  3. Port: 6642.
  4. Check interval: 60 seconds.
  5. Label: OVN Southbound DB TCP.
  6. Click Save.

Access note: In most Kubernetes deployments, OVN DB ports are not exposed externally for security reasons. If that is the case in your cluster, focus on application HTTP monitors as your primary health signal and use kubectl-based checks for OVN DB monitoring.

Step 4: Monitor the OVN-Kubernetes API Health Endpoint

The ovnkube-master component exposes a health endpoint that reflects its synchronization state with the OVN databases. If this endpoint returns errors, new Kubernetes objects (pods, services, NetworkPolicies) are not being translated into OVN logical resources:

# Port-forward to the ovnkube-master health endpoint
kubectl port-forward -n ovn-kubernetes <ovnkube-master-pod> 9409:9409 &
curl http://localhost:9409/healthz

If this endpoint is exposed externally:

  1. Add Monitor → HTTP.
  2. URL: http://ovnkube-master.example.com/healthz.
  3. Check interval: 60 seconds.
  4. Expected status: 200.
  5. Label: ovnkube-master health.
  6. Click Save.

Step 5: Monitor TLS Certificates for OVN-Kubernetes Exposed Services

Services with LoadBalancer or NodePort types that are exposed through OVN-Kubernetes's built-in load balancer need TLS certificate monitoring. OVN-Kubernetes programs Kubernetes Service load balancers directly into OVS using OVN load balancer objects — when the associated TLS certificate expires, HTTPS traffic fails immediately:

For each HTTPS service in your cluster:

  1. Add Monitor → SSL Certificate.
  2. Domain: myapp.example.com.
  3. Alert when expiry is within: 30 days.
  4. Alert again: 14 days, 7 days, 3 days, 1 day.
  5. Click Save.

If OVN-Kubernetes is deployed on OpenShift, also monitor OpenShift's API server certificate:

  1. Add Monitor → SSL Certificate.
  2. Domain: api.mycluster.example.com.
  3. Alert when expiry is within: 30 days.
  4. Click Save.

Step 6: Monitor OVN-Kubernetes Load Balancer TCP Connectivity

OVN-Kubernetes programs load balancers directly in OVS for Kubernetes Services. A TCP connectivity check on a NodePort or LoadBalancer service validates that OVN's load balancer DNAT rules are correctly programmed in OVS — separate from the application-level HTTP check:

# Get LoadBalancer port for a service
kubectl get svc my-application -o jsonpath='{.spec.ports[0].port}'
  1. Add Monitor → TCP.
  2. Host: myapp.example.com.
  3. Port: 443 (or your service port).
  4. Check interval: 60 seconds.
  5. Label: myapp LoadBalancer TCP (OVN LB).
  6. Click Save.

A TCP monitor that passes while an HTTP monitor fails indicates the OVN load balancer is routing packets correctly but the application itself is returning errors. A TCP monitor that fails while the node is healthy in Kubernetes indicates an OVN OpenFlow rule failure.

Step 7: Configure Alerting

In Vigilmon under Settings → Notifications, configure alert channels and response runbooks:

| Monitor | Trigger | Immediate action | |---|---|---| | Application HTTP endpoint | Non-200 or timeout | Check kubectl get pods -n ovn-kubernetes -o wide; inspect ovnkube-node logs on the affected node | | OVN Northbound DB TCP | Connection refused | OVN control plane failure; check ovnkube-master pod; run ovn-nbctl show | | OVN Southbound DB TCP | Connection refused | ovn-northd may be failing to sync; check ovn-northd pod logs | | Service LoadBalancer TCP | Connection refused | OVN LB rules may be stale; check ovs-ofctl dump-flows br-int on the node | | SSL certificate | < 30 days | Check cert-manager or manual renewal; inspect ingress/route TLS config | | ovnkube-master health | Non-200 | Northbound DB sync failure; new objects not being translated to OVN resources |

Alert grouping: Create an ovn-kubernetes monitor group. When the OVN Northbound DB fails, the ovnkube-master health monitor fires simultaneously — the group view distinguishes a control plane failure (DB + master health fire together) from a data plane failure (only application HTTP monitors fire).

Common OVN-Kubernetes Failure Modes and What Vigilmon Catches

| Scenario | Vigilmon signal | |---|---| | OVN Northbound DB leader election fails | Application HTTP monitors start failing for new pods; TCP DB monitor fires | | ovnkube-node crash on a node | HTTP monitors for pods on that node timeout; TCP LB monitor for affected services fires | | OVS daemon crash (kills OpenFlow rules) | All application monitors on that node fail simultaneously; TCP LB monitor fires | | NetworkPolicy ACL misconfiguration | HTTP monitors return 403 or timeout depending on drop vs reject ACL | | TLS certificate expiry on service | SSL monitor fires at 30-day threshold; HTTP monitor fires on expiry | | OVN load balancer entry stale after service deletion | TCP monitors for the deleted service may still pass briefly, then time out | | ovnkube-master pod OOM killed | Application HTTP monitors pass (existing pods work); new pod deployments get no networking | | Node firewall blocks OVN Geneve encapsulation (UDP 6081) | Cross-node HTTP monitors fail intermittently; same-node requests pass | | Kubernetes API server unreachable from ovnkube-master | Existing pods continue working; new objects don't get network config | | RAFT quorum loss in clustered OVN DB | OVN DB TCP monitors fire; application HTTP passes briefly then fails as rules go stale |

OVN-Kubernetes delivers high-performance, feature-rich networking for Kubernetes by programming OpenFlow rules directly into Open vSwitch — but its layered architecture (Kubernetes → OVN Northbound DB → OVN Southbound DB → OVS) means failures can occur silently at any layer. Vigilmon gives you external visibility across all of these: application HTTP reachability that validates the full OVN data path, TCP probes on OVN database ports that detect control plane failures before they cascade, and TLS certificate expiry alerts for every HTTPS service in your cluster. When an OVS flow table becomes stale after a node controller crash, Vigilmon catches it within 60 seconds.

Start monitoring your OVN-Kubernetes cluster in under 5 minutes — register free at vigilmon.online.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles