Monitoring Cobbler with Vigilmon

Cobbler is the Linux network install server that makes automated bare metal provisioning possible: it manages DHCP, DNS, TFTP, and HTTP to PXE-boot machines and push OS installations without ever touching a USB drive. When Cobbler is healthy, new servers appear on the network and boot themselves into configured systems. When Cobbler's services go down, your provisioning pipeline silently breaks — machines that should auto-install just sit at the PXE prompt. Vigilmon monitors Cobbler's API and supporting services so you catch failures before the next server goes dark.

What You'll Set Up

• HTTP uptime monitor for the Cobbler web interface and API
• TCP port monitors for Cobbler's DHCP and TFTP services
• Cron heartbeat for Cobbler sync jobs
• SSL certificate alerts for the Cobbler web interface
• Alert channels for on-call notification

Prerequisites

• Cobbler 3.x installed on a Linux server
• Cobbler web interface accessible (default port 443 or 80)
• A free Vigilmon account

Step 1: Monitor the Cobbler Web Interface

Cobbler ships with a web interface (cobbler-web) served by Apache. Add an HTTP monitor to verify it's accessible:

1. Log in to vigilmon.online and click Add Monitor.
2. Set Type to HTTP / HTTPS.
3. Enter the Cobbler web URL:

   https://your-cobbler-host/cobbler_web/
   

   Or if using HTTP:

   http://your-cobbler-host/cobbler_web/
   

4. Set Check interval to 2 minutes.
5. Set Expected HTTP status to 200 or 302 (Cobbler redirects to a login page).
6. Click Save.

Step 2: Monitor the Cobbler API via XMLRPC

Cobbler exposes an XMLRPC API on port 25151 (or proxied through Apache). You can probe this port directly to confirm the Cobbler daemon (cobblerd) is running:

1. Click Add Monitor → TCP Port.
2. Set Host to your Cobbler server IP or hostname.
3. Set Port to 25151.
4. Set Check interval to 1 minute.
5. Click Save.

The cobblerd process handles all API calls, sync operations, and configuration writes. If this port goes dark while Apache is still up, the web interface will appear accessible but all actions will fail.

For an HTTP-level API check, Cobbler also responds on the XMLRPC path through Apache:

1. Click Add Monitor → HTTP / HTTPS.
2. Enter:

   http://your-cobbler-host/cobbler_api
   

3. Set Expected HTTP status to 200.
4. Enable Keyword check with keyword cobbler.
5. Click Save.

Step 3: Monitor TFTP (PXE Boot Service)

TFTP serves the PXE bootloader to machines attempting a network boot. Without TFTP, no machine can start the installation process. Monitor the TFTP port:

1. Click Add Monitor → TCP Port.
2. Set Host to your Cobbler server IP.
3. Set Port to 69.
4. Set Check interval to 1 minute.
5. Click Save.

Note: Port 69 uses UDP, not TCP. Some monitoring systems can't probe UDP ports. If your Vigilmon plan supports UDP probing, use it; otherwise use a heartbeat script (Step 5) to verify TFTP responds correctly:

# Test TFTP from a remote host
tftp your-cobbler-host -c get pxelinux.0 /tmp/pxelinux.0 && echo "TFTP OK" || echo "TFTP FAILED"

Step 4: Monitor the Cobbler HTTP Boot Server

Cobbler serves OS installation files (kickstart files, packages, stage2 images) via HTTP on port 80. Add a monitor for the distribution tree:

1. Click Add Monitor → HTTP / HTTPS.
2. Enter:

   http://your-cobbler-host/cobbler/
   

3. Set Check interval to 2 minutes.
4. Set Expected HTTP status to 200 or 301.
5. Click Save.

This directory listing serves the actual OS installation content. If it returns a 404 or 503, machines may PXE boot successfully but stall during the OS installation phase when they can't fetch packages or the kickstart file.

Step 5: Heartbeat for Cobbler Sync

cobbler sync regenerates DHCP/DNS configurations, rebuilds the TFTP tree, and restarts services when you add or modify profiles and systems. If a sync fails silently, your configuration changes won't be applied and new machines will get stale settings. Use a Vigilmon heartbeat to verify syncs succeed:

1. In Vigilmon, click Add Monitor → Cron Heartbeat.
2. Set the expected ping interval to 24 hours (adjust to match your sync frequency).
3. Copy your heartbeat URL:

   https://vigilmon.online/heartbeat/abc123
   

4. Create a wrapper script:

#!/bin/bash
# /usr/local/bin/cobbler-sync-check.sh

# Run cobbler sync
cobbler sync 2>&1
SYNC_EXIT=$?

if [ $SYNC_EXIT -eq 0 ]; then
    # Verify TFTP and HTTP are serving content
    TFTP_FILE=/var/lib/tftpboot/pxelinux.0
    HTTP_CHECK=$(curl -s -o /dev/null -w "%{http_code}" http://localhost/cobbler/)

    if [ -f "$TFTP_FILE" ] && [ "$HTTP_CHECK" = "200" ]; then
        curl -s https://vigilmon.online/heartbeat/abc123
        echo "Cobbler sync OK"
    else
        echo "Cobbler sync ran but TFTP/HTTP check failed: HTTP=$HTTP_CHECK, TFTP=$([[ -f $TFTP_FILE ]] && echo 'present' || echo 'missing')" >&2
        exit 1
    fi
else
    echo "Cobbler sync FAILED with exit code $SYNC_EXIT" >&2
    exit 1
fi

5. Schedule with cron:

crontab -e
# Run nightly at 2 AM
0 2 * * * /usr/local/bin/cobbler-sync-check.sh >> /var/log/cobbler-sync.log 2>&1

Step 6: Monitor DHCP Service

Cobbler manages DHCP (via dhcpd or dnsmasq) to assign IP addresses and point machines to the TFTP server. A DHCP outage means new machines can never start PXE boot. Monitor the DHCP port:

1. Click Add Monitor → TCP Port.
2. Set Host to your Cobbler server.
3. Set Port to 67 (DHCP).
4. Set Check interval to 1 minute.
5. Click Save.

Like TFTP, DHCP uses UDP. For a more reliable check, add a script-based heartbeat that verifies the dhcpd process is running:

#!/bin/bash
# /usr/local/bin/cobbler-dhcp-check.sh

# Check dhcpd process
if pgrep -x "dhcpd" > /dev/null 2>&1; then
    curl -s https://vigilmon.online/heartbeat/dhcp-xyz789
    echo "DHCP OK: dhcpd running"
else
    echo "DHCP FAILED: dhcpd not running" >&2
    exit 1
fi

Schedule this every 5 minutes and set the Vigilmon heartbeat interval to 10 minutes.

Step 7: SSL Certificate Alerts

If Cobbler's web interface is served over HTTPS with a certificate, add SSL expiry monitoring:

1. Open the HTTP monitor for your Cobbler web interface.
2. Enable Monitor SSL certificate.
3. Set Alert when certificate expires in less than 21 days.
4. Click Save.

Cobbler often runs on internal networks with self-managed certificates — these are easy to forget about until they expire and break API clients and CI/CD systems that talk to Cobbler.

Step 8: Configure Alert Channels

1. Go to Alert Channels in Vigilmon and add Slack, email, or a webhook.
2. For the cobblerd API port (25151), set Consecutive failures to 1 — if the daemon is down, no sync or provisioning is possible.
3. For the web interface, set Consecutive failures to 2 — a single HTTP timeout may be transient.

# Suppress alerts during Cobbler upgrades
curl -X POST https://vigilmon.online/api/maintenance \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"monitor_id": "cobbler-api-monitor-id", "duration_minutes": 60}'

Summary

| Monitor | Target | What It Catches | |---|---|---| | HTTP web UI | /cobbler_web/ | Apache down, web interface inaccessible | | TCP cobblerd | :25151 | Cobbler daemon crash, API failures | | HTTP XMLRPC | /cobbler_api | API endpoint broken | | HTTP boot tree | /cobbler/ | Installation files unavailable | | TCP/UDP TFTP | :69 | PXE boot broken for all new machines | | Cron heartbeat | Sync check script | Failed sync, stale DHCP/TFTP config | | SSL certificate | Cobbler domain | Expired cert breaks API clients |

Cobbler is the silent workhorse of bare metal automation — it runs unattended and nobody thinks about it until a new server sits at the PXE prompt. With Vigilmon watching cobblerd, TFTP, HTTP boot, and sync jobs, you get alerted the moment the provisioning chain breaks, long before the next server deployment attempt reveals the problem.