tutorial

Monitoring WeaveNet with Vigilmon: CNI Network Health, Pod Connectivity, Mesh Encryption & Weave Scope Availability

How to monitor WeaveNet with Vigilmon — CNI network health checks, pod-to-pod connectivity, mesh encryption status, Weave Scope availability, and network partition alerting.

WeaveNet is a popular Kubernetes CNI (Container Network Interface) plugin from Weaveworks that creates a virtual overlay network across cluster nodes, enabling pod-to-pod communication without requiring external routers or NAT configuration. WeaveNet supports optional encryption for all inter-node traffic using NaCl's Salsa20 cipher, and integrates with Weave Scope for live network topology visualization. When a WeaveNet router fails to establish a mesh connection between two nodes, pods on those nodes lose connectivity silently — existing TCP connections may stay open while new connections fail. When the WeaveNet daemon crashes on a node, all pods on that node lose network connectivity and become unreachable, often before Kubernetes itself marks them as unhealthy. Vigilmon gives you external visibility into the WeaveNet mesh: endpoint reachability across nodes, application HTTP health through the overlay, TLS certificate monitoring for Weave Scope, and TCP-level connectivity probes that catch network partitions before your users do.

What You'll Build

  • HTTP monitors on applications running across different nodes to validate cross-node WeaveNet routing
  • TCP monitors on WeaveNet's control plane port to detect router failures
  • SSL certificate monitors for Weave Scope HTTPS endpoints
  • A failure mode table covering the most common WeaveNet network partition and daemon crash scenarios
  • An alerting runbook that maps Vigilmon alerts to WeaveNet diagnostic commands

Prerequisites

  • A Kubernetes cluster with WeaveNet installed as the CNI plugin
  • Applications deployed across multiple nodes (to exercise cross-node routing)
  • Weave Scope installed and accessible (optional but recommended)
  • External access to at least one service in your cluster (LoadBalancer or NodePort)
  • A free account at vigilmon.online

Step 1: Understand WeaveNet's Health Surface

WeaveNet exposes health information through its weave CLI, the weave-net DaemonSet status, and a metrics endpoint. Before setting up external monitors, verify the mesh is fully operational:

# Check WeaveNet DaemonSet pods on all nodes
kubectl get pods -n kube-system -l name=weave-net -o wide

# Inspect WeaveNet router status on a specific node
kubectl exec -n kube-system <weave-net-pod> -c weave -- /home/weave/weave --local status

# Check peer connections — each node should appear
kubectl exec -n kube-system <weave-net-pod> -c weave -- /home/weave/weave --local status peers

# Verify WeaveNet IPAM allocation
kubectl exec -n kube-system <weave-net-pod> -c weave -- /home/weave/weave --local status ipam

The most important signals for external monitoring are:

  1. Application HTTP responses across the WeaveNet overlay (end-to-end cross-node connectivity)
  2. TCP connectivity to WeaveNet's control port (6783) on each node
  3. Weave Scope availability (if deployed)
  4. TLS certificates for any HTTPS endpoints served through the WeaveNet overlay

Step 2: Monitor Application HTTP Connectivity Across the Overlay

The most reliable external test of WeaveNet health is an HTTP check on an application that exercises cross-node pod communication. If pods on separate nodes communicate through the WeaveNet overlay to serve the request, a successful response proves the overlay is routing correctly between those nodes:

# Find which node each pod replica is running on
kubectl get pods -o wide -l app=my-application

# Verify the LoadBalancer endpoint routes to pods across nodes
curl https://myapp.example.com/health
  1. Log in to Vigilmon → Add Monitor → HTTP.
  2. URL: https://myapp.example.com/health (your application's health endpoint).
  3. Check interval: 60 seconds.
  4. Response timeout: 10 seconds.
  5. Expected status: 200.
  6. Keyword: your health response identifier (e.g., ok, healthy, status).
  7. Label: myapp.example.com (WeaveNet overlay).
  8. Click Save.

Add a monitor for each distinct service that routes through the WeaveNet overlay, particularly services that span multiple nodes. If a WeaveNet node router crashes, cross-node requests to pods on that node will start failing while same-node requests continue to succeed — multiple per-service monitors help you localize the failure.

Step 3: Monitor WeaveNet Control Plane TCP Connectivity

WeaveNet routers communicate with each other over TCP port 6783 (and UDP 6783/6784 for fast datapath). A TCP connectivity check on this port from Vigilmon's external probes tells you whether the WeaveNet router process is alive and accepting connections — even before application-level failures surface:

  1. Add Monitor → TCP.
  2. Host: your node's external IP or hostname (e.g., node1.example.com).
  3. Port: 6783.
  4. Check interval: 60 seconds.
  5. Label: WeaveNet router TCP (node1).
  6. Click Save.

Repeat for each node in your cluster. This is especially useful in bare-metal or on-premises clusters where Kubernetes node conditions may lag behind actual network failures.

Note on firewall rules: TCP port 6783 must be open between nodes for WeaveNet to function. If your cluster uses a firewall that blocks external access to port 6783, this TCP monitor will always fail — restrict TCP monitoring to port 6783 only on nodes that have external network access, and use HTTP application monitors as the primary health signal instead.

Step 4: Monitor Weave Scope Availability

Weave Scope provides real-time visualization of your WeaveNet topology, container relationships, and network traffic. If Scope becomes unavailable, your team loses visibility into live container communication — often exactly when you need it most during an incident:

# Weave Scope default service port
kubectl get svc -n weave weave-scope-app
# Returns: Type=LoadBalancer or NodePort, Port=4040
  1. Add Monitor → HTTP.
  2. URL: https://scope.example.com (your Scope endpoint, or http://node-ip:4040 if NodePort).
  3. Check interval: 2 minutes.
  4. Response timeout: 15 seconds.
  5. Expected status: 200.
  6. Label: Weave Scope UI.
  7. Click Save.

If Scope is exposed over HTTPS:

  1. Add Monitor → SSL Certificate.
  2. Domain: scope.example.com.
  3. Alert when expiry is within: 30 days.
  4. Alert again: 14 days, 7 days, 3 days, 1 day.
  5. Click Save.

Step 5: Monitor TLS Certificates for Services on the WeaveNet Overlay

Applications served through the WeaveNet overlay that terminate TLS at the ingress or application layer need certificate monitoring. An expired certificate breaks all HTTPS traffic to that service and is often undetected until a user reports a browser certificate warning:

For each HTTPS service in your cluster:

  1. Add Monitor → SSL Certificate.
  2. Domain: myapp.example.com.
  3. Alert when expiry is within: 30 days.
  4. Alert again: 14 days, 7 days, 3 days, 1 day.
  5. Click Save.

Step 6: Monitor WeaveNet Metrics Endpoint

WeaveNet exposes a Prometheus-compatible metrics endpoint at port 6782. If you expose this endpoint externally (or through a monitoring proxy), Vigilmon can check its availability:

# Access WeaveNet metrics from within the cluster
kubectl exec -n kube-system <weave-net-pod> -c weave -- \
  curl -s http://localhost:6782/metrics | head -20

If exposed externally:

  1. Add Monitor → HTTP.
  2. URL: http://weave-metrics.example.com/metrics (if proxied externally).
  3. Check interval: 2 minutes.
  4. Expected status: 200.
  5. Keyword: weave_ (WeaveNet metric prefix).
  6. Label: WeaveNet metrics endpoint.
  7. Click Save.

Step 7: Configure Alerting

In Vigilmon under Settings → Notifications, configure alert channels and response runbooks:

| Monitor | Trigger | Immediate action | |---|---|---| | Application HTTP endpoint | Non-200 or timeout | Check WeaveNet DaemonSet: kubectl get pods -n kube-system -l name=weave-net -o wide; run weave status peers on affected nodes | | WeaveNet TCP port 6783 | Connection refused | WeaveNet router crashed; check: kubectl logs -n kube-system <weave-net-pod> -c weave; restart DaemonSet pod if needed | | Weave Scope UI | Non-200 | Check Scope deployment: kubectl get pods -n weave; verify service is routing correctly | | SSL certificate | < 30 days | Review cert-manager or manual certificate renewal; check expiry with openssl s_client -connect myapp.example.com:443 | | WeaveNet metrics | Non-200 | WeaveNet router may be failing to expose metrics; check router health on that node |

Alert grouping: Create a weavenet monitor group containing all WeaveNet-related monitors. When a full node failure occurs, both the TCP monitor for that node and the application monitors for pods scheduled on that node fire together — the group view makes the node-level scope immediately visible.

Common WeaveNet Failure Modes and What Vigilmon Catches

| Scenario | Vigilmon signal | |---|---| | WeaveNet daemon crash on a node | TCP port 6783 monitor fires; HTTP monitors for pods on that node start failing | | Network partition splits nodes into two groups | Cross-partition HTTP monitors timeout; same-partition requests continue passing | | WeaveNet IPAM conflict (duplicate IP allocation) | Application monitors return intermittent errors; connections succeed then fail | | Weave Scope pod OOM killed | Scope HTTP monitor fires; application connectivity unaffected | | TLS certificate expiry on ingress | SSL monitor fires at 30-day threshold; HTTP monitor fires with TLS error on expiry day | | WeaveNet encryption key mismatch after upgrade | Cross-node HTTP monitors start timing out; TCP port 6783 may stay green | | Node firewall rule blocks WeaveNet UDP | Cross-node HTTP monitors fail intermittently (TCP fallback slower); TCP 6783 may still pass | | Kubernetes node NotReady (WeaveNet not initialized) | HTTP monitors for pods on that node timeout; TCP monitor also fires | | WeaveNet fast datapath disabled fallback to sleeve | Application HTTP monitors show increased latency but continue passing | | Cluster DNS fails (CoreDNS loses WeaveNet connectivity) | HTTP monitors using hostnames fail; monitors using IPs continue passing |

WeaveNet's overlay mesh gives Kubernetes clusters flexible pod-to-pod networking across heterogeneous infrastructure — but its distributed mesh architecture means failures can occur at the router level, the encryption layer, the IPAM system, or the data path independently. Vigilmon gives you external visibility across all of these: cross-node HTTP reachability that proves the overlay is routing, TCP connectivity probes on the WeaveNet control port, Weave Scope availability monitoring, and TLS certificate expiry alerts for every HTTPS service in your cluster. When a node router crash silently isolates pods from the rest of the mesh, Vigilmon alerts you in under a minute.

Start monitoring your WeaveNet cluster in under 5 minutes — register free at vigilmon.online.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles