How to Monitor Metal3 Bare Metal Provisioning with Vigilmon (HTTP + TCP + Alerts)

Metal3 (Metal Kubed) lets you provision bare metal servers for Kubernetes using the same declarative API you use for cloud resources. It bridges bare metal hardware into the cloud-native world — but that also means a new class of failure modes that Kubernetes-internal probes simply cannot detect.

In this tutorial you'll set up comprehensive uptime monitoring for your Metal3-provisioned infrastructure using Vigilmon — free tier, no credit card required.

Why Metal3 environments need external monitoring

Metal3 provisions physical servers via IPMI/Redfish and manages them through the BareMetalHost CRD and the Ironic provisioning service. Your Kubernetes control plane may report everything healthy while the real world is completely different:

• Ironic API becomes unreachable — the provisioning API is down, so new servers cannot be enrolled or re-imaged; existing kubectl get baremetalhosts still returns results
• PXE/DHCP server failure — servers that need to be provisioned or re-provisioned are stuck; no alert fires inside the cluster
• BMC (IPMI/Redfish) endpoint goes silent — Metal3 loses out-of-band management access to hosts; the control plane doesn't immediately surface this as downtime
• Provisioning network partition — the management VLAN between Ironic and the target hosts is isolated; hosts are unreachable but the Kubernetes API is fine
• Ironic inspector hangs — hardware introspection is stuck; new nodes never join the cluster, and there is no user-facing error until a deployment times out

The pattern: Kubernetes reports Running while your bare metal provisioning pipeline is completely broken. External monitoring from multiple geographic vantage points is the only reliable safety net.

What you'll need

• A running Metal3 environment with Ironic deployed (often via Cluster API Provider Metal3)
• The Ironic API endpoint exposed (or a health proxy in front of it)
• A free Vigilmon account (sign up takes 30 seconds)

Step 1: Expose Metal3 health endpoints

Metal3 runs several components you should monitor externally. The key services are:

• Ironic API — typically on port 6385 (HTTP) or 13385 (HTTPS with TLS)
• Ironic Inspector — typically on port 5050
• CAPM3 controller — exposed via the Kubernetes API, indirectly via a LoadBalancer Service

First, confirm your Ironic API is responding:

# Check Ironic API health
curl http://192.168.0.10:6385/

# Expected: {"versions": [...]}

If Ironic is running inside Kubernetes, expose it so external monitors can reach it:

# ironic-service.yaml
apiVersion: v1
kind: Service
metadata:
  name: ironic
  namespace: baremetal-operator-system
spec:
  type: LoadBalancer
  selector:
    app: ironic
  ports:
    - name: ironic-api
      port: 6385
      targetPort: 6385
      protocol: TCP
    - name: ironic-inspector
      port: 5050
      targetPort: 5050
      protocol: TCP

kubectl apply -f ironic-service.yaml

# Get the external IP
kubectl get svc ironic -n baremetal-operator-system
# NAME     TYPE           CLUSTER-IP     EXTERNAL-IP    PORT(S)                          AGE
# ironic   LoadBalancer   10.96.8.12     203.0.113.80   6385:31385/TCP,5050:31050/TCP    2m

Add a lightweight health proxy if you don't want to expose Ironic directly:

# health_proxy.py — minimal Flask proxy for Ironic status
from flask import Flask, jsonify
import requests

app = Flask(__name__)
IRONIC_URL = "http://127.0.0.1:6385"

@app.route("/health")
def health():
    try:
        r = requests.get(IRONIC_URL, timeout=5)
        r.raise_for_status()
        return jsonify({"status": "ok", "ironic": "reachable"}), 200
    except Exception as e:
        return jsonify({"status": "error", "detail": str(e)}), 503

if __name__ == "__main__":
    app.run(host="0.0.0.0", port=8080)

Step 2: Set up HTTP monitoring in Vigilmon

With Ironic reachable, add it to Vigilmon:

1. Log in to vigilmon.online and go to Monitors → New Monitor
2. Choose HTTP / HTTPS as the type
3. Set the URL to your Ironic API endpoint: http://203.0.113.80:6385/ or your health proxy URL
4. Set the check interval to 1 minute
5. Under Expected response, set:
   • Status code: 200
   • (Optional) Response body contains: "versions"
6. Save the monitor

Vigilmon probes your endpoint from multiple geographic regions. If Ironic becomes unreachable — even while your Kubernetes API is healthy — an incident is triggered immediately.

Multi-region consensus prevents false positives

Bare metal environments are often hosted in single-site data centers with asymmetric network paths. Vigilmon's multi-region consensus model requires multiple independent probes from different geographic locations to agree before firing an alert. This eliminates false positives from transient network blips on a single transit path while still catching real outages within seconds.

Step 3: Monitor TCP connectivity to BMC endpoints

Bare metal provisioning depends on being able to reach BMC (IPMI/Redfish) interfaces. You can use Vigilmon's TCP monitors to verify that the management network is reachable:

# Verify a BMC endpoint is accepting connections
nc -zv 192.168.1.100 443    # Redfish over HTTPS
nc -zv 192.168.1.100 623    # IPMI UDP (TCP check is an approximation)

Add a TCP monitor in Vigilmon:

1. Go to Monitors → New Monitor
2. Choose TCP Port
3. Enter the BMC host and Redfish port: 192.168.1.100 / 443
4. Save the monitor

Repeat for each physical server's BMC or for a jumphost that proxies BMC access. This tells you immediately when the out-of-band management network has a connectivity issue — before you try to provision a server and hit a cryptic timeout.

You can also monitor the Ironic Inspector TCP port:

Host: 203.0.113.80
Port: 5050

Step 4: Configure alert channels

When Ironic goes down or a BMC becomes unreachable, your bare metal provisioning pipeline is silently broken — and you won't know until you try to provision a new node hours later.

Email alerts

1. In Vigilmon, go to Alert Channels → Add Channel → Email
2. Enter your infrastructure team's on-call email
3. Assign the channel to your Ironic and BMC monitors

Webhook alerts (Slack, PagerDuty, etc.)

1. Go to Alert Channels → Add Channel → Webhook
2. Enter your webhook URL
3. Assign the channel to your monitors

Example payload Vigilmon sends when Ironic goes down:

{
  "monitor_name": "Ironic API",
  "status": "down",
  "url": "http://203.0.113.80:6385/",
  "started_at": "2024-01-15T10:23:00Z",
  "duration_seconds": 300
}

Use this to trigger automated remediation or to enrich alerts with Metal3 context:

# Check BareMetalHost states when Ironic alert fires
kubectl get baremetalhosts -A
kubectl describe baremetalhost <name> -n baremetal-operator-system

Step 5: Create a public status page

If your Metal3 infrastructure supports multiple teams or tenants, a status page keeps everyone informed without granting kubectl access.

1. In Vigilmon, go to Status Pages → New Status Page
2. Give it a name: "Bare Metal Provisioning"
3. Add your monitors: Ironic API, Inspector, BMC endpoints
4. Group them: Provisioning API, Hardware Management, Network
5. Publish the page

Share the resulting URL with your platform team, development teams, and operations stakeholders.

Putting it all together

Here's a summary of the monitors to create for a typical Metal3 deployment:

| Monitor | Type | What it catches | |---------|------|-----------------| | http://203.0.113.80:6385/ | HTTP | Ironic API availability | | http://203.0.113.80:5050/ | HTTP | Ironic Inspector availability | | 203.0.113.80:6385 | TCP | Ironic API port reachability | | 192.168.1.100:443 | TCP | BMC Redfish endpoint | | http://203.0.113.80:8080/health | HTTP | Health proxy for Ironic |

And the corresponding Metal3 status checks to run when alerts fire:

# Check BareMetalHost states
kubectl get baremetalhosts -A -o wide

# Check Metal3 controller logs
kubectl logs -n baremetal-operator-system -l app=metal3-baremetal-operator

# Check Ironic pod status
kubectl get pods -n baremetal-operator-system

# Verify Ironic API from inside cluster
kubectl run tmp-curl --image=curlimages/curl --rm -it --restart=Never -- \
  curl http://ironic.baremetal-operator-system.svc.cluster.local:6385/

What's next

• SSL certificate monitoring — if you expose Ironic over HTTPS, Vigilmon monitors your TLS cert and alerts before it expires
• Heartbeat monitoring — if you run Metal3 provisioning jobs or reconciliation scripts on a schedule, Vigilmon's heartbeat monitors alert when they stop reporting in
• Multi-site coverage — if you have bare metal in multiple data centers, create separate monitor groups per site and use status page grouping to track each site independently

Get started free at vigilmon.online — no credit card, monitors start running in under a minute.