tutorial

How to Monitor Innernet Private Networks with Vigilmon

Innernet is a WireGuard-based private networking system built by Tonari. Unlike raw WireGuard, Innernet adds a structured management layer: a central server ...

Innernet is a WireGuard-based private networking system built by Tonari. Unlike raw WireGuard, Innernet adds a structured management layer: a central server issues peer invitations, manages CIDR allocations, and handles peer-to-peer association automatically. It's designed for teams who want the cryptographic guarantees of WireGuard without managing a mesh of static peer configs by hand.

This approach simplifies operations enormously — but it also creates infrastructure dependencies that need monitoring. The Innernet server is now a critical control-plane component. Its health, API availability, and the peers it manages all need observability. This tutorial shows you how to monitor Innernet deployments end-to-end with Vigilmon.

Why Innernet needs dedicated monitoring

Innernet's architecture introduces failure modes specific to its managed-WireGuard model:

  • Server availability — the innernet-server process manages peer associations. If it goes down, existing tunnels keep working (WireGuard persists in the kernel), but new peers can't join and peer refreshes fail.
  • CIDR exhaustion — Innernet allocates CIDRs from the pool you configure at setup. Exhausted pools silently block new peer invitations without alerting administrators.
  • Peer fetch failures — Innernet peers periodically fetch updates from the server (innernet fetch). If the fetch silently fails (network partition, TLS error, server down), peers drift out of sync. New peers added server-side become unreachable.
  • Interface drift — Innernet manages the WireGuard interface through its own daemon. OS updates, kernel module issues, or daemon crashes can leave the interface in a stale state without a visible alert.
  • Services over the tunnel — even with healthy peer management, application-layer failures on the tunnel are invisible to Innernet itself.

Complete monitoring requires covering the server control plane, peer sync health, tunnel connectivity, and application services.

What you'll need

  • A running innernet-server instance
  • At least one Innernet peer joined to the network
  • A free Vigilmon account

Step 1: Expose health endpoints on Innernet-hosted services

Services reachable over your Innernet tunnel should expose /health endpoints bound to their Innernet interface IP (typically in the subnet you configured, e.g., 10.66.0.0/16):

Go:

package main

import (
    "encoding/json"
    "net/http"
)

func main() {
    http.HandleFunc("/health", func(w http.ResponseWriter, r *http.Request) {
        w.Header().Set("Content-Type", "application/json")
        json.NewEncoder(w).Encode(map[string]string{"status": "ok"})
    })
    // Bind to Innernet interface address
    http.ListenAndServe("10.66.0.1:8080", nil)
}

Node.js / Express:

const express = require('express');
const app = express();

app.get('/health', (req, res) => {
  res.json({ status: 'ok', timestamp: new Date().toISOString() });
});

// Bind to Innernet interface address
app.listen(8080, '10.66.0.1');

systemd socket activation (any backend):

[Socket]
ListenStream=10.66.0.1:8080

Step 2: Monitor the Innernet server health with a heartbeat

The innernet-server process exposes an API on a configurable port (default: 8080 on the server). Probe it from the server host:

#!/bin/bash
# innernet-server-probe.sh
# Run on the innernet-server host

HEARTBEAT_URL="https://vigilmon.online/api/heartbeat/YOUR_SERVER_HEARTBEAT"
INNERNET_API="http://localhost:8080"

# Check that the innernet-server process is running
if ! pgrep -x innernet-server > /dev/null; then
  echo "$(date): innernet-server not running" >> /var/log/innernet-probe.log
  exit 1
fi

# Verify the server API is responding
http_code=$(curl -s -o /dev/null -w "%{http_code}" --max-time 5 "$INNERNET_API/v1/admin/peers")
if [ "$http_code" != "200" ] && [ "$http_code" != "401" ]; then
  # 401 = auth required = server is up; anything else = problem
  echo "$(date): innernet-server API returned $http_code" >> /var/log/innernet-probe.log
  exit 1
fi

# Verify the WireGuard interface managed by innernet is up
if ! ip link show innernet > /dev/null 2>&1; then
  echo "$(date): innernet WireGuard interface not found" >> /var/log/innernet-probe.log
  exit 1
fi

curl -s "$HEARTBEAT_URL" --max-time 10 --retry 2

Schedule every minute:

* * * * * /opt/scripts/innernet-server-probe.sh

In Vigilmon: Monitors → New Monitor → Heartbeat, interval 2 minutes, name Innernet Server Health.

Step 3: Monitor peer sync health

Innernet peers run innernet fetch (or the daemon does it automatically) to pull updated peer lists from the server. Stale peers mean newly added nodes are unreachable. Monitor sync freshness from each peer:

#!/bin/bash
# innernet-sync-probe.sh
# Run on an Innernet peer host

HEARTBEAT_URL="https://vigilmon.online/api/heartbeat/YOUR_SYNC_HEARTBEAT"
MAX_SYNC_AGE_MINUTES=10
LOG_FILE="/var/log/innernet-sync.log"

# Get the last sync time from innernet status
# innernet status --json outputs peer and server info
last_sync=$(innernet status --json 2>/dev/null | \
  python3 -c "
import json, sys, time
try:
    d = json.load(sys.stdin)
    # Extract last_modified from the first interface
    ts = d[0].get('last_modified', 0) if d else 0
    print(ts)
except:
    print(0)
" 2>/dev/null)

if [ -z "$last_sync" ] || [ "$last_sync" = "0" ]; then
  echo "$(date): Could not determine last sync time" >> "$LOG_FILE"
  exit 1
fi

now=$(date +%s)
age_minutes=$(( (now - last_sync) / 60 ))

if [ "$age_minutes" -gt "$MAX_SYNC_AGE_MINUTES" ]; then
  echo "$(date): Innernet last sync was ${age_minutes} minutes ago — stale" >> "$LOG_FILE"
  exit 1
fi

curl -s "$HEARTBEAT_URL" --max-time 10

Create a Heartbeat monitor with 15-minute interval, name Innernet Peer Sync. Run the script every 5 minutes via cron.

Step 4: Monitor tunnel connectivity between peers

Verify that peer-to-peer connectivity actually works across the Innernet overlay — not just that the daemon thinks it's up:

#!/bin/bash
# innernet-connectivity-probe.sh

HEARTBEAT_URL="https://vigilmon.online/api/heartbeat/YOUR_CONNECTIVITY_HEARTBEAT"

# Critical peers to verify connectivity to
PEERS=(
  "10.66.0.1"   # innernet-server node
  "10.66.0.10"  # database peer
  "10.66.0.20"  # API peer
)

all_ok=true

for peer in "${PEERS[@]}"; do
  if ! ping -c 2 -W 4 "$peer" > /dev/null 2>&1; then
    echo "$(date): Cannot reach peer $peer over innernet" >> /var/log/innernet-connectivity.log
    all_ok=false
  fi
done

# Also check application health endpoints
SERVICES=(
  "http://10.66.0.10:8080/health"
  "http://10.66.0.20:3000/health"
)

for svc in "${SERVICES[@]}"; do
  code=$(curl -s -o /dev/null -w "%{http_code}" --max-time 5 "$svc")
  if [ "$code" != "200" ]; then
    echo "$(date): Service $svc returned $code" >> /var/log/innernet-connectivity.log
    all_ok=false
  fi
done

$all_ok && curl -s "$HEARTBEAT_URL" --max-time 10

Step 5: Monitor the Innernet server from outside

The server's public-facing port needs to be reachable for peers to fetch updates. Monitor it externally from Vigilmon's global probes:

If your Innernet server has a public hostname, add a TCP Port monitor in Vigilmon:

  • Host: innernet.yourdomain.com
  • Port: 8080 (or your configured port)
  • Name: Innernet Server Port

For HTTP-exposed management APIs, use an HTTP monitor:

  • URL: https://innernet.yourdomain.com/v1/admin/peers
  • Expected status: 401 (auth required = server up)
  • Name: Innernet Server API

This catches public IP changes, firewall rule drift, and server outages before your peer sync jobs start failing.

Step 6: Monitor scheduled peer management jobs

If you script peer invitations, CIDR allocation, or peer cleanup, wrap those jobs with Vigilmon heartbeats:

#!/bin/bash
# innernet-peer-audit.sh — run nightly

HEARTBEAT_URL="https://vigilmon.online/api/heartbeat/YOUR_AUDIT_HEARTBEAT"

# Count active peers — alert if count drops unexpectedly
peer_count=$(innernet-server list-peers --network my-network 2>/dev/null | wc -l)

if [ "$peer_count" -lt 2 ]; then
  echo "$(date): Unusually low peer count: $peer_count" >> /var/log/innernet-audit.log
  exit 1
fi

echo "$(date): $peer_count peers active" >> /var/log/innernet-audit.log
curl -s "$HEARTBEAT_URL" --max-time 10

Create a Heartbeat monitor with 25-hour interval for the nightly audit job.

Step 7: Configure alerting

Route alerts by severity:

  1. Innernet Server Health — immediate alert; server down means no new peer connections
  2. Innernet Server Port — immediate; external unreachability blocks peer fetches
  3. Peer Sync — 5-minute delay; transient fetch failures are common, but prolonged sync failure is a problem
  4. Connectivity Probe — immediate for database/API peers; delayed for less critical nodes

In Vigilmon: Alert Channels → Add Channel → Slack (#innernet-ops), assign monitors by severity.

Complete monitoring reference

| Monitor | Type | What it catches | |---------|------|-----------------| | Innernet Server Health | Heartbeat | Server process down, interface missing | | Innernet Server Port | TCP | Server unreachable from public internet | | Innernet Server API | HTTP | API layer failure, TLS issues | | Peer Sync Health | Heartbeat | Stale peer list, fetch failure | | Overlay Connectivity | Heartbeat | End-to-end tunnel failure | | Peer Audit Job | Heartbeat | Missed nightly audit, low peer count |

Troubleshooting common failures

  • Server heartbeat stops, process is running — check innernet-server logs: journalctl -u innernet-server -n 50; look for database errors or port binding conflicts.
  • Peer sync heartbeat stops — run innernet fetch manually on the peer and inspect the output; common causes are TLS cert mismatch, server unreachability, or corrupted local peer database.
  • Connectivity probe fails for one peer — verify that peer's status with innernet status on the server; if the peer is listed but unreachable, check whether it last fetched recently and whether its public endpoint changed.
  • Server API returns 500 — Innernet server uses SQLite; check for disk space issues (df -h) and verify the database file isn't corrupted.

Next steps

  • Automate peer lifecycle — script peer invitations and removals, wrapping each with a heartbeat to confirm completion
  • Multi-network monitoring — if you run multiple Innernet networks, create a Vigilmon status page per network for fast operational triage

Start monitoring your Innernet private network today at vigilmon.online — free, no credit card required.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles