tutorial

How to Monitor Nebula Overlay Networks with Vigilmon

Nebula, developed by Slack and open-sourced through Defined Networking, is a scalable overlay mesh networking tool that lets you build high-performance, encr...

Nebula, developed by Slack and open-sourced through Defined Networking, is a scalable overlay mesh networking tool that lets you build high-performance, encrypted private networks across thousands of nodes spanning multiple cloud providers, data centers, and edge locations. Its lighthouse-based NAT traversal, certificate-based identity, and pure Go implementation make it a favorite for platform teams who need WireGuard-level performance without the manual peer management overhead.

But Nebula's strengths — decentralized peer-to-peer tunnels, lighthouse coordination, certificate lifecycle — introduce monitoring challenges that generic uptime tools can't address. This tutorial shows you how to build complete observability for your Nebula overlay network using Vigilmon.

Why Nebula needs dedicated monitoring

Nebula's architecture creates several failure modes that are invisible without proactive checks:

  • Lighthouse failures — nodes use lighthouses to discover each other's public endpoints. A down lighthouse doesn't break existing tunnels immediately, but when nodes roam or reconnect, they can't find peers. Silent degradation is the failure mode.
  • Certificate expiry — every Nebula host has a certificate signed by your CA. Expired certs drop nodes from the overlay silently; the affected node sees connection refused rather than an expiry error.
  • Firewall rule drift — Nebula's built-in firewall controls traffic between nodes. A bad config push can silently block service traffic without appearing in system logs.
  • Relay node saturation — for nodes that can't punch through NAT directly, Nebula uses relay nodes. A saturated relay degrades latency mesh-wide without a clean error surface.
  • Peer connectivity gaps — nebula-cert print and status endpoints reveal current state, but nothing alerts you when a critical peer drops and stays down.

Monitoring Nebula means checking the lighthouse layer, certificate validity, relay health, and the services running over the overlay.

What you'll need

  • A running Nebula overlay network with at least one lighthouse
  • One or more hosts joined to the overlay
  • A free Vigilmon account

Step 1: Expose health endpoints on overlay services

Every service bound to a Nebula interface should expose a health route. Bind to the Nebula IP (typically in the 10.x.x.x range you configured in your ca.crt subnets):

Go (net/http):

package main

import (
    "encoding/json"
    "net/http"
)

func main() {
    http.HandleFunc("/health", func(w http.ResponseWriter, r *http.Request) {
        w.Header().Set("Content-Type", "application/json")
        json.NewEncoder(w).Encode(map[string]string{"status": "ok"})
    })
    // Bind to Nebula interface IP
    http.ListenAndServe("10.42.0.1:8080", nil)
}

Python / FastAPI:

from fastapi import FastAPI
import uvicorn

app = FastAPI()

@app.get("/health")
async def health():
    return {"status": "ok"}

if __name__ == "__main__":
    uvicorn.run(app, host="10.42.0.1", port=8080)

Nginx (wraps any backend):

server {
    listen 10.42.0.1:80;

    location /health {
        access_log off;
        return 200 '{"status":"ok"}';
        add_header Content-Type application/json;
    }

    location / {
        proxy_pass http://127.0.0.1:3000;
    }
}

Services bound to the Nebula IP are only reachable from other Nebula nodes — correctly isolated from the public internet by design.

Step 2: Monitor the lighthouse with a Vigilmon heartbeat probe

The lighthouse is the most critical component of your Nebula network. Run a probe on the lighthouse host that validates it's actively handling peer queries:

#!/bin/bash
# nebula-lighthouse-probe.sh
# Run on the lighthouse host

HEARTBEAT_URL="https://vigilmon.online/api/heartbeat/YOUR_LIGHTHOUSE_HEARTBEAT"

# Check the Nebula process is running
if ! pgrep -x nebula > /dev/null; then
  echo "$(date): Nebula process not running" >> /var/log/nebula-probe.log
  exit 1
fi

# Check the lighthouse UDP port is open (default: 4242)
if ! ss -ulnp | grep -q ':4242'; then
  echo "$(date): Nebula UDP port 4242 not listening" >> /var/log/nebula-probe.log
  exit 1
fi

# Check overlay IP is reachable (ping self on Nebula interface)
if ! ping -c 1 -W 3 -I nebula1 10.42.0.1 > /dev/null 2>&1; then
  echo "$(date): Nebula interface not responding" >> /var/log/nebula-probe.log
  exit 1
fi

# All checks passed — ping Vigilmon
curl -s "$HEARTBEAT_URL" --max-time 10 --retry 2

Schedule it:

* * * * * /opt/scripts/nebula-lighthouse-probe.sh

In Vigilmon: Monitors → New Monitor → Heartbeat, interval 2 minutes, name Nebula Lighthouse Health. If the lighthouse crashes or its port stops listening, the heartbeat stops and Vigilmon alerts.

Step 3: Monitor certificate expiry

Nebula certificates expire. An expired host cert silently drops that node from the overlay. Run a certificate expiry check on every critical node:

#!/bin/bash
# nebula-cert-check.sh
# Alert Vigilmon if any cert expires within 30 days

HEARTBEAT_URL="https://vigilmon.online/api/heartbeat/YOUR_CERT_HEARTBEAT"
WARN_DAYS=30
CERT_FILE="/etc/nebula/host.crt"

# Extract expiry timestamp from the Nebula cert
expiry_unix=$(nebula-cert print -path "$CERT_FILE" -json 2>/dev/null | \
  python3 -c "import json,sys; d=json.load(sys.stdin); print(d['details']['notAfter'])" 2>/dev/null)

if [ -z "$expiry_unix" ]; then
  echo "$(date): Could not read certificate expiry" >> /var/log/nebula-cert-check.log
  exit 1
fi

now_unix=$(date +%s)
days_left=$(( (expiry_unix - now_unix) / 86400 ))

if [ "$days_left" -lt "$WARN_DAYS" ]; then
  echo "$(date): Certificate expires in ${days_left} days — alert suppressed" >> /var/log/nebula-cert-check.log
  exit 1
fi

curl -s "$HEARTBEAT_URL" --max-time 10

Create a Heartbeat monitor with a 2-day interval — if the cert check stops pinging (cert expired or script failed), Vigilmon fires. Run the script daily via cron:

0 9 * * * /opt/scripts/nebula-cert-check.sh

Step 4: Monitor connectivity between critical node pairs

Nebula's overlay should allow specific services to talk. Use a probe on a non-lighthouse node to verify end-to-end connectivity across the overlay:

#!/bin/bash
# nebula-connectivity-probe.sh
# Run on a leaf node — checks connectivity to critical peers

HEARTBEAT_URL="https://vigilmon.online/api/heartbeat/YOUR_CONNECTIVITY_HEARTBEAT"

# Peer addresses to verify
PEERS=(
  "10.42.0.1"   # lighthouse
  "10.42.0.10"  # database node
  "10.42.0.20"  # API node
)

all_ok=true

for peer in "${PEERS[@]}"; do
  if ! ping -c 2 -W 3 -I nebula1 "$peer" > /dev/null 2>&1; then
    echo "$(date): Cannot reach Nebula peer $peer" >> /var/log/nebula-connectivity.log
    all_ok=false
  fi
done

# Also check service health endpoints
SERVICES=(
  "http://10.42.0.10:5432"
  "http://10.42.0.20:8080/health"
)

for svc in "${SERVICES[@]}"; do
  code=$(curl -s -o /dev/null -w "%{http_code}" --max-time 5 "$svc")
  if [ "$code" != "200" ]; then
    echo "$(date): Service $svc returned $code" >> /var/log/nebula-connectivity.log
    all_ok=false
  fi
done

$all_ok && curl -s "$HEARTBEAT_URL" --max-time 10

Add to crontab on each critical leaf node every 2 minutes.

Step 5: Monitor the public-facing lighthouse port from outside

The lighthouse's UDP port needs to be reachable from the public internet for NAT traversal to work. While Vigilmon doesn't do UDP checks directly, you can expose a lightweight HTTP probe on the lighthouse machine to verify it's reachable:

# Add a tiny HTTP responder alongside Nebula on the lighthouse
python3 -m http.server 8888 &

# Or use nginx:
server {
    listen 8888;
    location /ping {
        return 200 'ok';
        add_header Content-Type text/plain;
    }
}

In Vigilmon: Monitors → New Monitor → HTTP, URL http://YOUR_LIGHTHOUSE_IP:8888/ping, expected status 200, name Nebula Lighthouse Reachability. If the machine goes down or its public IP changes, this monitor fires before your nodes notice.

Step 6: Configure alerting and escalation

Route Nebula alerts to your networking on-call:

  1. Alert Channels → Add Channel → Slack — #nebula-ops or #infra-alerts
  2. Add Channel → PagerDuty (or email) for the lighthouse heartbeat
  3. Lighthouse Reachability — immediate (0-minute) alert delay; a down lighthouse blocks new peer connections
  4. Certificate expiry — 30-minute delay acceptable; this is a warning, not an outage

Step 7: Build an overlay network status page

Consolidate Nebula health into a single status page for your infrastructure team:

  1. Status Pages → New Status Page
  2. Name: Nebula Overlay Network
  3. Add: Lighthouse Health, Lighthouse Reachability, Cert Validity, Connectivity Probe
  4. Share the URL in #infra

Complete monitoring reference

| Monitor | Type | What it catches | |---------|------|-----------------| | Lighthouse Health | Heartbeat | Nebula process down, UDP port closed | | Lighthouse Reachability | HTTP | Machine down, public IP changed | | Certificate Validity | Heartbeat | Cert expiry within 30 days | | Overlay Connectivity | Heartbeat | Peer-to-peer tunnel failure | | Service endpoints | HTTP (per service) | Application layer failures |

Troubleshooting common Nebula failures

  • Node can't find peers — check lighthouse logs: journalctl -u nebula -n 100. If the lighthouse is healthy, the peer's cert may have expired or its config has the wrong lighthouse IP.
  • Connectivity probe fails for one peer — run nebula-cert print -path /etc/nebula/host.crt on the unreachable node; compare the subnet range and groups against your firewall rules in config.yaml.
  • Certificate expiry heartbeat stops — run nebula-cert print -path /etc/nebula/host.crt -json manually to confirm expiry date; re-sign with nebula-cert sign using your CA key and distribute the new cert.
  • Lighthouse UDP port probe fails — run ss -ulnp on the lighthouse; if port 4242 isn't listed, Nebula crashed. Check systemctl status nebula and restart.

Next steps

  • Automate cert rotation — use a cron job to re-sign certs 60 days before expiry and wrap it with a Vigilmon heartbeat
  • Relay node monitoring — if you use Nebula relay nodes, duplicate the connectivity probe to run through the relay path and add a dedicated heartbeat

Start monitoring your Nebula overlay today at vigilmon.online — free, no credit card required.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles