Monitoring OpenStack Ironic with Vigilmon

OpenStack Ironic is the bare metal provisioning service that makes physical machines behave like cloud VMs: enroll a server, provide credentials, and Ironic handles PXE booting, deploying, and cleaning it on demand. When Ironic works, it's transparent. When it fails — API down, conductor crashed, IPMI unreachable — your entire bare metal fleet is frozen. Vigilmon gives you continuous visibility into Ironic's API, conductor health, and deployment pipelines so failures surface immediately instead of during the next deploy attempt.

What You'll Set Up

• HTTP uptime monitor for the Ironic API
• TCP port monitors for Ironic's internal services
• Cron heartbeat for Ironic conductor periodic tasks
• Alert channels for on-call notification

Prerequisites

• OpenStack Ironic installed (standalone or as part of an OpenStack deployment)
• Ironic API accessible (default port 6385)
• A free Vigilmon account

Step 1: Monitor the Ironic API

The Ironic REST API is the control plane for all bare metal operations. Add an HTTP monitor for its health endpoint:

1. Log in to vigilmon.online and click Add Monitor.
2. Set Type to HTTP / HTTPS.
3. Enter the Ironic API URL:

   http://your-ironic-host:6385/
   

   For a production deployment behind a load balancer:

   https://ironic.yourdomain.com/
   

4. Set Check interval to 1 minute.
5. Set Expected HTTP status to 200.
6. Click Save.

The Ironic API root returns a JSON object describing API versions ({"default_version": {...}, "versions": [...]}). A 200 confirms the ironic-api service is running and accepting requests.

Step 2: Monitor the Ironic API Version Endpoint

For a more specific check, monitor the versioned API endpoint that Ironic operators use:

1. Click Add Monitor → HTTP / HTTPS.
2. Enter:

   http://your-ironic-host:6385/v1/
   

3. Set Check interval to 2 minutes.
4. Enable Keyword check with keyword nodes.
5. Set Expected HTTP status to 200.
6. Click Save.

The /v1/ endpoint returns a list of available resources including nodes, ports, chassis, and drivers. The keyword check confirms the API is returning a real Ironic response rather than an error page from a proxy.

Step 3: Monitor the Ironic Conductor via TCP

The Ironic conductor handles the actual work: talking to IPMI/Redfish, running PXE deployments, and cleaning nodes. It doesn't expose a public HTTP endpoint, but you can monitor its presence via the RPC message broker (RabbitMQ) that the conductor uses for communication:

| Service | Default Port | |---|---| | Ironic API | 6385 | | RabbitMQ (Ironic RPC) | 5672 | | RabbitMQ management UI | 15672 |

Add TCP monitors for each:

1. Click Add Monitor → TCP Port.
2. Set Host to your controller node IP.
3. Set Port to 5672 (RabbitMQ AMQP).
4. Set Check interval to 1 minute.
5. Click Save.

If RabbitMQ goes down, the conductor cannot receive tasks — but the Ironic API may still respond to HTTP requests. Monitoring both catches this split-brain scenario where the API says "OK" but no work is actually being processed.

Step 4: Add a Health Check Script for the Ironic Conductor

Ironic doesn't have a built-in conductor health endpoint, but you can query the API for conductor state:

#!/bin/bash
# /usr/local/bin/ironic-conductor-check.sh

# Set up OpenStack credentials
source /etc/openstack/admin-openrc.sh

# Check that at least one conductor is active
CONDUCTORS=$(openstack baremetal conductor list -f json 2>/dev/null | \
  python3 -c "import sys,json; c=json.load(sys.stdin); print(len([x for x in c if x.get('Alive')]))")

if [ -z "$CONDUCTORS" ] || [ "$CONDUCTORS" -eq 0 ]; then
    echo "ERROR: No active Ironic conductors found" >&2
    exit 1
fi

echo "OK: $CONDUCTORS active conductor(s)"

Run this from a monitoring host and integrate it with Vigilmon's heartbeat (see Step 5).

Step 5: Heartbeat Monitor for Conductor Periodic Tasks

The Ironic conductor runs periodic tasks: syncing node power states, cleaning nodes after deployments, and checking for stuck provisioning operations. Use a Vigilmon heartbeat to verify these tasks run on schedule:

1. In Vigilmon, click Add Monitor → Cron Heartbeat.
2. Set the expected ping interval to 10 minutes.
3. Copy your heartbeat URL:

   https://vigilmon.online/heartbeat/abc123
   

4. Create a wrapper script:

#!/bin/bash
# /usr/local/bin/ironic-health-check.sh

source /etc/openstack/admin-openrc.sh

# Verify conductor(s) are alive
ALIVE=$(openstack baremetal conductor list -f json | \
  python3 -c "
import sys, json
conductors = json.load(sys.stdin)
alive = [c for c in conductors if c.get('Alive')]
print(len(alive))
")

# Verify API responds
API_STATUS=$(curl -s -o /dev/null -w "%{http_code}" http://localhost:6385/v1/)

if [ "$ALIVE" -gt 0 ] && [ "$API_STATUS" = "200" ]; then
    curl -s https://vigilmon.online/heartbeat/abc123
    echo "Ironic health OK: $ALIVE conductor(s) alive, API returned $API_STATUS"
else
    echo "Ironic health FAILED: conductors=$ALIVE, API=$API_STATUS" >&2
    exit 1
fi

5. Schedule with cron:

crontab -e
# Run every 5 minutes, expect ping every 10 minutes (allows one failure)
*/5 * * * * /usr/local/bin/ironic-health-check.sh >> /var/log/ironic-health.log 2>&1

Step 6: Monitor Node Provisioning Failures

Ironic nodes can get stuck in deploy failed or clean failed states without raising a system-level alert. Add a script that detects stuck nodes and pings a separate heartbeat only when no nodes are stuck:

#!/bin/bash
# /usr/local/bin/ironic-node-check.sh

source /etc/openstack/admin-openrc.sh

# Count nodes in error states
FAILED=$(openstack baremetal node list --provision-state deploy\ failed -f json | \
  python3 -c "import sys,json; print(len(json.load(sys.stdin)))")

CLEAN_FAILED=$(openstack baremetal node list --provision-state clean\ failed -f json | \
  python3 -c "import sys,json; print(len(json.load(sys.stdin)))")

TOTAL_FAILED=$((FAILED + CLEAN_FAILED))

if [ "$TOTAL_FAILED" -gt 0 ]; then
    echo "WARNING: $FAILED deploy-failed, $CLEAN_FAILED clean-failed nodes" >&2
    # Don't ping heartbeat — let it alert after the interval
    exit 1
else
    curl -s https://vigilmon.online/heartbeat/node-check-xyz789
    echo "All nodes healthy: 0 failed"
fi

Create a separate heartbeat monitor in Vigilmon for this check with a 30 minute interval.

Step 7: Configure Alert Channels

1. Go to Alert Channels in Vigilmon and configure Slack, PagerDuty, email, or a webhook.
2. For the Ironic API monitor (port 6385), set Consecutive failures before alert to 2.
3. For the RabbitMQ TCP monitor, set Consecutive failures to 1 — message broker failures are immediately critical.
4. Use Maintenance windows during OpenStack upgrades:

# Suppress alerts during a 30-minute maintenance window
curl -X POST https://vigilmon.online/api/maintenance \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"monitor_id": "ironic-api-monitor-id", "duration_minutes": 30}'

Summary

| Monitor | Target | What It Catches | |---|---|---| | HTTP API root | :6385/ | ironic-api service crash | | HTTP API v1 | :6385/v1/ | Partial failure, proxy returning errors | | TCP RabbitMQ | :5672 | Message broker down, conductor starved | | Cron heartbeat | Conductor check script | Conductor crash, no active conductors | | Node state heartbeat | Node check script | Stuck deploy/clean-failed nodes |

OpenStack Ironic makes bare metal as manageable as cloud VMs — but only while its own services stay healthy. With Vigilmon monitoring the API, the conductor's RPC channel, and node provisioning state, you catch Ironic failures before they become "why did the deployment hang for two hours?" investigations.