Monitoring Oxidized with Vigilmon: Web API Health, Configuration Backup Status, SSL Certificates & Uptime Checks

Oxidized is the open-source network configuration backup tool that polls routers, switches, and firewalls on a schedule — fetching running configurations via SSH or Telnet and committing them to a Git repository for history, diff tracking, and compliance auditing. Network operations teams depend on Oxidized to maintain a continuous configuration history: when a change causes an outage, Oxidized's Git history is the forensic record engineers reach for first. But Oxidized itself can fail: the Ruby process can crash, the polling loop can stall, network changes can prevent Oxidized from reaching devices, and backup jobs can fail silently for individual nodes. Vigilmon gives you an external, independent monitoring layer for Oxidized: HTTP API availability, backup job health, node fetch status, and SSL certificate expiry.

What You'll Build

• An HTTP API availability monitor confirming the Oxidized web API is running
• A node status check verifying Oxidized is actively fetching configurations
• A fetch failure rate monitor to detect when devices start failing to back up
• SSL certificate monitoring for your Oxidized web API endpoint
• Alerting configured for a configuration-backup-critical system

Prerequisites

• Oxidized running with the REST API enabled (rest: 0.0.0.0:8888 in your config file)
• Oxidized accessible via HTTPS (behind Nginx, Caddy, or another TLS-terminating reverse proxy)
• A free account at vigilmon.online

Step 1: Enable the Oxidized REST API

Before adding monitors, confirm Oxidized's REST API is enabled. In your Oxidized configuration file (~/.config/oxidized/config):

rest: 0.0.0.0:8888

Verify the API is reachable:

curl http://localhost:8888/nodes

A healthy response returns a JSON array of node objects. Place this behind a reverse proxy with TLS for external access.

Step 2: Monitor the Oxidized HTTP API

Oxidized's REST API is the primary interface for querying node status, triggering manual backups, and checking fetch results. A Ruby process crash, an exception in the Oxidized event loop, or a misconfigured reverse proxy takes the API offline:

curl -I https://your-oxidized.example.com/nodes

A healthy response returns 200 OK with a JSON array. Add the monitor:

1. Log in to Vigilmon → Add Monitor → HTTP.
2. URL: https://your-oxidized.example.com/nodes.
3. Check interval: 5 minutes.
4. Expected status: 200.
5. Keyword: name (present in every valid node object in the JSON array).
6. Label: Oxidized - API / Node List.
7. Click Save.

Authentication: If you have configured HTTP basic authentication for Oxidized's REST API (recommended), add the Authorization: Basic <base64> header to this monitor in Vigilmon's request headers field.

Step 3: Monitor Node Fetch Activity

Oxidized tracks the last time each node was fetched. The /nodes endpoint includes last timestamps for each node. A more targeted check is the /node/fetch/:name endpoint for a critical device — this confirms Oxidized can reach and fetch a specific device:

curl https://your-oxidized.example.com/node/fetch/your-core-router

A healthy response triggers an immediate fetch and returns the node's configuration. For a non-invasive status check without triggering a fetch, query the node's status:

curl https://your-oxidized.example.com/node/show/your-core-router

A healthy response returns the node object with recent last fetch timestamp and status: success. Add the monitor:

1. Add Monitor → HTTP.
2. URL: https://your-oxidized.example.com/node/show/your-core-router.
3. Check interval: 15 minutes.
4. Expected status: 200.
5. Keyword: success (present in status field when last fetch succeeded).
6. Label: Oxidized - Core Router Fetch Status.
7. Click Save.

Repeat this monitor for your most critical network devices — core routers, distribution switches, and firewall clusters.

Staleness vs failure: A status: success in the node show response confirms the last fetch succeeded, but it does not confirm the fetch was recent. For freshness checks, combine this monitor with your Git repository's commit age — a separate alert if the last commit is older than 2× the polling interval.

Step 4: Monitor Failed Nodes via the Stats Endpoint

Oxidized exposes aggregate statistics that include counts of nodes in each state. Use the stats endpoint to detect when the failure rate across all nodes rises — which can indicate a network change that blocked Oxidized's access to a device segment, or a credential rotation that invalidated SSH keys:

curl https://your-oxidized.example.com/statistics

A healthy response includes node counts by state:

{
  "nodes": {
    "total": 42,
    "up": 41,
    "down": 1,
    ...
  }
}

Add the monitor:

1. Add Monitor → HTTP.
2. URL: https://your-oxidized.example.com/statistics.
3. Check interval: 15 minutes.
4. Expected status: 200.
5. Keyword: nodes (present in any valid statistics response).
6. Label: Oxidized - Fetch Statistics.
7. Click Save.

Threshold alerting: The keyword check confirms the statistics endpoint is returning data. For alert-on-high-failure-count logic, route the Oxidized stats to a metrics system (Prometheus via the Oxidized metrics exporter) and alert there. Vigilmon's check here is a belt-and-suspenders signal that Oxidized is alive and reporting.

Step 5: Monitor SSL Certificates

If Oxidized's REST API is exposed over HTTPS (which it should be in any multi-team environment), an expired certificate breaks API access for every client and integration:

curl -v https://your-oxidized.example.com/nodes 2>&1 | grep -i "expire"

Add SSL monitoring:

1. Add Monitor → SSL Certificate.
2. Domain: your-oxidized.example.com.
3. Alert when expiry is within: 30 days.
4. Alert again: 14 days, 7 days, 3 days, 1 day.
5. Click Save.

Step 6: Monitor the Git Repository Endpoint (If Exposed)

If your Oxidized deployment pushes to a self-hosted Gitea or Gogs instance (common in air-gapped network environments), and if that Git frontend is accessible via HTTPS, add a monitor to confirm the configuration history repository is accessible:

curl -I https://your-gitea.example.com/oxidized/network-configs

Add the monitor:

1. Add Monitor → HTTP.
2. URL: https://your-gitea.example.com/oxidized/network-configs.
3. Check interval: 15 minutes.
4. Expected status: 200.
5. Label: Oxidized - Config Git Repository.
6. Click Save.

This monitor catches the scenario where Oxidized is running and fetching configurations successfully but the Git push is failing — meaning the history is not being persisted even as fetch health looks fine.

Step 7: Configure Alerting

In Vigilmon under Settings → Notifications, configure alert channels for your network operations team:

| Monitor | Trigger | Action | |---|---|---| | Oxidized - API / Node List | Non-200 or name keyword missing | Oxidized process down; check Ruby process and system logs | | Oxidized - Core Router Fetch Status | Non-200 or success keyword missing | Critical device not backing up; check SSH connectivity and credentials | | Oxidized - Fetch Statistics | Non-200 or nodes keyword missing | Statistics endpoint broken; broader Oxidized API issue | | Oxidized - Config Git Repository | Non-200 | Git push failing; configurations fetched but not persisted | | SSL Certificate | < 30 days to expiry | Renew certificate; Oxidized API clients will fail on expiry |

Alert sensitivity: Set Oxidized monitors to alert after 3 consecutive failures. Oxidized's polling cycle means individual check bounces are more common than in request-response services. For the critical device fetch status monitors, tighten to 2 consecutive failures — a core router that has not backed up for two check intervals (30 minutes) is worth investigating immediately.

Off-hours alerting: Configuration backup failures are not always emergency-level incidents — a single device that fails to back up during a maintenance window is low priority. Use Vigilmon's alert scheduling to send Oxidized node-fetch failures as a daily digest email during business hours, while routing Oxidized process death (API unavailable) to your on-call channel 24/7.

Common Oxidized Failure Modes and What Vigilmon Catches

| Scenario | Vigilmon monitor | |---|---| | Oxidized Ruby process crash (OOM or unhandled exception) | API / Node List fires | | SSH key rotation on monitored devices without updating Oxidized | Core device Fetch Status fires with auth failure | | ACL change blocking Oxidized's management IP | Core device Fetch Status fires; other devices may still succeed | | Oxidized unable to reach Git remote (network or auth issue) | Git Repository monitor fires; Fetch monitors stay green | | Reverse proxy misconfiguration after server update | All HTTP monitors fire | | Oxidized config file corrupted or unreadable on restart | API fires on next process start | | TLS certificate expires | SSL monitor alerts at 30-day threshold | | New device added with wrong credentials | That device's fetch status fires; other nodes unaffected | | Oxidized polling loop stalled (rare Ruby event loop deadlock) | API may stay green while no fetches are happening; monitor with Git commit age externally | | Disk full on Oxidized host | Git commits start failing; Git Repository monitor fires |

Why Monitor Oxidized with an External Tool

Oxidized is often deployed as a background daemon with no human watching its output. The Ruby process can crash, SSH connectivity to devices can break, or Git push can fail — and because Oxidized runs silently in the background, no one notices until an engineer needs to compare a configuration before and after an outage and discovers weeks of history are missing. Vigilmon surfaces Oxidized failures the moment they happen, not at the next incident post-mortem. An alert that fires when a critical router stops backing up gives your team time to investigate and restore backup coverage before the gap matters.

The external monitoring perspective is also important for Oxidized specifically: because Oxidized itself monitors your network devices, an Oxidized failure creates a blind spot in your network change history. Vigilmon closes that blind spot.

Oxidized keeps your network configuration history intact — but only when it is running and fetching successfully. Vigilmon provides the external monitoring layer: API availability, individual device fetch health, aggregate backup statistics, Git repository accessibility, and SSL certificate expiry — all monitored independently from outside your infrastructure.

Start monitoring Oxidized in under 5 minutes — register free at vigilmon.online.