Monitoring FRRouting with Vigilmon

FRRouting (FRR) is the open-source routing protocol suite that runs BGP, OSPF, IS-IS, RIP, and other routing protocols on Linux. It's the engine behind many enterprise routers, data center leaf/spine fabrics, and cloud network appliances. When FRR daemons crash or BGP sessions drop, traffic can black-hole silently — routers keep forwarding on stale routes until the RIB eventually flushes, masking the outage until applications start timing out. Vigilmon gives you proactive alerting on FRR daemon health, BGP session state, and routing connectivity so you catch failures before traffic is affected.

What You'll Set Up

• HTTP uptime monitors for FRR's built-in HTTP API (if enabled)
• TCP connectivity checks for BGP peer sessions
• Cron heartbeat monitors for BGP session state verification scripts
• Custom health endpoint that checks FRR daemon and BGP peer status
• Alert channels for routing failures with appropriate severity routing

Prerequisites

• FRRouting 8.0+ installed (apt install frr or dnf install frr)
• At least one routing daemon enabled (BGP, OSPF, or similar)
• vtysh CLI accessible for status queries
• A free Vigilmon account

Verify FRR is running:

systemctl status frr
vtysh -c "show version"

Step 1: Monitor FRR Daemon Processes

FRR runs as a collection of daemons: zebra (RIB manager), bgpd, ospfd, isisd, etc. If any critical daemon crashes, the protocols it manages stop working while the host remains reachable. Monitoring process health is your first line of defense.

Create a lightweight health endpoint that checks all enabled FRR daemons:

#!/usr/bin/env python3
# frr_health.py
from http.server import HTTPServer, BaseHTTPRequestHandler
import subprocess
import json

# Match your /etc/frr/daemons configuration
ENABLED_DAEMONS = ['zebra', 'bgpd']  # add ospfd, isisd, etc. as needed

class Handler(BaseHTTPRequestHandler):
    def do_GET(self):
        if self.path != '/health':
            self.send_response(404)
            self.end_headers()
            return

        dead = []
        for daemon in ENABLED_DAEMONS:
            result = subprocess.run(
                ['systemctl', 'is-active', f'frr'],
                capture_output=True, text=True
            )
            # Check individual daemon via vtysh
            check = subprocess.run(
                ['vtysh', '-c', f'show daemon {daemon}'],
                capture_output=True, text=True, timeout=5
            )
            if check.returncode != 0 or 'disabled' in check.stdout:
                dead.append(daemon)

        if dead:
            body = json.dumps({'status': 'degraded', 'dead_daemons': dead}).encode()
            self.send_response(503)
        else:
            body = json.dumps({
                'status': 'ok',
                'daemons': ENABLED_DAEMONS
            }).encode()
            self.send_response(200)

        self.send_header('Content-Type', 'application/json')
        self.end_headers()
        self.wfile.write(body)

    def log_message(self, format, *args):
        pass

if __name__ == '__main__':
    HTTPServer(('0.0.0.0', 9103), Handler).serve_forever()

Run as a systemd service and add a Vigilmon HTTP monitor for http://your-router-host:9103/health.

Step 2: TCP Monitor for BGP Peer Connectivity

BGP runs over TCP port 179. A TCP monitor verifying that your router accepts connections on port 179 confirms that bgpd is listening and the firewall is not blocking BGP traffic — prerequisites for any BGP session to establish.

1. Log in to vigilmon.online and click Add Monitor.
2. Set Type to TCP Port.
3. Enter your router's management IP.
4. Set Port to 179.
5. Set Check interval to 1 minute.
6. Click Save.

For each router in your BGP topology, add a dedicated TCP monitor. This catches bgpd crashes (port closes) and ACL/firewall changes that block BGP establishment.

If your FRR routers peer over a dedicated peering VLAN, run the TCP check from a host on that VLAN rather than the management network to verify the actual peering path.

Step 3: Heartbeat Monitoring for BGP Session State

A router accepting TCP connections on port 179 does not mean BGP sessions are established. Sessions can be in Active, Connect, or OpenSent states — not Established — and traffic forwarding will be affected without any obvious external signal.

Add a BGP session state monitor:

1. In Vigilmon, click Add Monitor → Cron Heartbeat.
2. Set the expected ping interval to 5 minutes.
3. Copy the heartbeat URL: https://vigilmon.online/heartbeat/YOUR_TOKEN.

Create the BGP session check script:

#!/bin/bash
# /usr/local/bin/frr-bgp-check.sh

VIGILMON_URL="https://vigilmon.online/heartbeat/YOUR_TOKEN"
LOG="/var/log/frr-bgp-check.log"
REQUIRED_PEERS=2  # minimum established BGP sessions

# Get BGP summary in JSON
BGP_JSON=$(vtysh -c "show bgp summary json" 2>/dev/null)

if [ -z "$BGP_JSON" ]; then
  echo "$(date): vtysh returned empty output" >> "$LOG"
  exit 1
fi

ESTABLISHED=$(echo "$BGP_JSON" | python3 -c "
import json, sys
data = json.load(sys.stdin)
count = 0
for vrf_data in data.values():
    peers = vrf_data.get('peers', {})
    for peer, info in peers.items():
        if info.get('state') == 'Established':
            count += 1
print(count)
" 2>/dev/null)

if [ -z "$ESTABLISHED" ]; then
  echo "$(date): Failed to parse BGP JSON" >> "$LOG"
  exit 1
fi

if [ "$ESTABLISHED" -ge "$REQUIRED_PEERS" ]; then
  curl -sf "$VIGILMON_URL" > /dev/null
else
  echo "$(date): Only $ESTABLISHED/$REQUIRED_PEERS BGP sessions established" >> "$LOG"
fi

Schedule every 5 minutes:

*/5 * * * * /usr/local/bin/frr-bgp-check.sh

When BGP sessions drop below the required count — due to link failure, authentication mismatch, or bgpd restart — the script stops sending heartbeats and Vigilmon alerts within 5–10 minutes.

Step 4: OSPF Adjacency Monitoring

For OSPF deployments, adjacency state is the equivalent of BGP's Established state. An OSPF router in 2-Way or ExStart state is not exchanging routes:

#!/bin/bash
# /usr/local/bin/frr-ospf-check.sh

VIGILMON_URL="https://vigilmon.online/heartbeat/YOUR_OSPF_TOKEN"
LOG="/var/log/frr-ospf-check.log"

# Check OSPF neighbor states
FULL_COUNT=$(vtysh -c "show ip ospf neighbor json" 2>/dev/null | python3 -c "
import json, sys
try:
    data = json.load(sys.stdin)
    neighbors = data.get('neighbors', {})
    full = sum(
        1 for iface in neighbors.values()
        for n in (iface if isinstance(iface, list) else [iface])
        if n.get('state', '').startswith('Full')
    )
    print(full)
except:
    print(0)
")

if [ "$FULL_COUNT" -gt 0 ]; then
  curl -sf "$VIGILMON_URL" > /dev/null
else
  echo "$(date): No OSPF Full adjacencies found" >> "$LOG"
fi

Add a separate Vigilmon cron heartbeat for the OSPF check, independent from the BGP check, so protocol-specific failures trigger targeted alerts.

Step 5: Route Table Verification Heartbeat

Routing protocol session state does not guarantee that the correct routes are in the forwarding table. Route maps, prefix lists, or redistribution misconfigurations can leave sessions up while blocking specific prefixes.

#!/bin/bash
# /usr/local/bin/frr-route-check.sh

VIGILMON_URL="https://vigilmon.online/heartbeat/YOUR_ROUTE_TOKEN"
EXPECTED_PREFIX="10.0.0.0/8"   # a prefix that must be present

# Check for expected prefix in FIB
if vtysh -c "show ip route $EXPECTED_PREFIX" 2>/dev/null | grep -q "^\*>"; then
  curl -sf "$VIGILMON_URL" > /dev/null
else
  echo "$(date): Expected prefix $EXPECTED_PREFIX not found in RIB" >> /var/log/frr-route-check.log
fi

This check verifies that a critical prefix (e.g., a default route or a datacenter supernet) is present in FRR's RIB. It catches route withdrawal events that don't take down the session itself.

Step 6: Configure Alert Channels

1. Go to Alert Channels in Vigilmon and connect Slack, PagerDuty, email, or a webhook.
2. For BGP heartbeat monitors, set the alert threshold to 1 missed interval — a single missing heartbeat represents 5 minutes of session loss.
3. For the TCP port monitor on 179, set Consecutive failures to 2 to avoid false alarms from brief TCP resets during BGP hold-timer negotiation.
4. Route alerts directly to your network operations team, separate from application alerts.

For multi-router deployments, use Vigilmon's monitor groups to organize by:

• Site (datacenter A, datacenter B)
• Protocol (BGP, OSPF)
• Role (edge, spine, leaf)

This makes it easy to correlate alerts when a single link failure causes multiple monitors to fire simultaneously.

Summary

| Monitor | Target | What It Catches | |---|---|---| | HTTP sidecar | FRR daemon health endpoint | Daemon crash (bgpd, zebra, ospfd) | | TCP | Port 179 on each BGP router | bgpd down, firewall blocking BGP | | Cron heartbeat (BGP) | BGP session state script | Session not Established | | Cron heartbeat (OSPF) | OSPF adjacency state script | Adjacency not Full | | Cron heartbeat (routes) | RIB prefix presence check | Critical route withdrawn |

FRR's routing protocol failures are notoriously silent from the application layer — traffic routes around failures on stale FIB entries until TTLs expire, creating delayed outages that are hard to diagnose after the fact. With Vigilmon monitoring daemon health, session state, and route table correctness, you get early warning at every layer of the routing stack, giving your network operations team time to act before users are impacted.