tutorial

How to Monitor Tinkerbell Bare Metal Provisioning with Vigilmon (HTTP + TCP + Alerts)

Tinkerbell is a cloud-native bare metal provisioning stack built to run entirely on Kubernetes. Its modular architecture — Hegel (metadata server), Tink Serv...

Tinkerbell is a cloud-native bare metal provisioning stack built to run entirely on Kubernetes. Its modular architecture — Hegel (metadata server), Tink Server (workflow orchestration), Boots (DHCP/PXE), and Rufio (BMC management) — gives you GitOps-friendly hardware provisioning. But that same distributed architecture creates multiple independent failure points that your Kubernetes-internal health checks won't surface.

In this tutorial you'll set up comprehensive uptime monitoring for your Tinkerbell stack using Vigilmon — free tier, no credit card required.

Why Tinkerbell stacks need external monitoring

Tinkerbell's components must all work together for provisioning to succeed. Any single failure breaks the entire pipeline, and most failures are silent from Kubernetes' perspective:

  • Boots DHCP stops responding — new hardware PXE-boots but never gets an IP address; the pod shows Running but no DHCP packets are served
  • Tink Server becomes unreachable — hardware workers connected to physical machines lose their workflow source; provisioning silently stalls
  • Hegel metadata server goes down — machines that booted successfully can't fetch their configuration; they hang waiting for cloud-init data
  • Rufio can't reach BMC endpoints — power-cycling and firmware operations silently fail; provisioning workflows that depend on BMC control time out
  • Network isolation — the provisioning VLAN is partitioned; Kubernetes reports all pods healthy while no bare metal machine can be reached

The gap between "Kubernetes says healthy" and "provisioning pipeline is working" is exactly where external monitoring adds value.

What you'll need

  • A running Tinkerbell stack (Tink Server, Boots, Hegel, and optionally Rufio)
  • At least one Tinkerbell service exposed outside the cluster (or a health proxy)
  • A free Vigilmon account (sign up takes 30 seconds)

Step 1: Expose Tinkerbell health endpoints

Tinkerbell's services run as Kubernetes workloads. Confirm they are up and expose them for external monitoring:

# Check Tinkerbell pods
kubectl get pods -n tink-system

# NAME                             READY   STATUS    RESTARTS   AGE
# tink-server-7d8c9b4f6-xk2np     1/1     Running   0          3d
# boots-5f9d7c8b4-mn3qr            1/1     Running   0          3d
# hegel-6c8d9f5b4-pq7rs            1/1     Running   0          3d

Expose Tink Server and Hegel via LoadBalancer:

# tink-services.yaml
apiVersion: v1
kind: Service
metadata:
  name: tink-server-external
  namespace: tink-system
spec:
  type: LoadBalancer
  selector:
    app: tink-server
  ports:
    - name: grpc
      port: 42113
      targetPort: 42113
      protocol: TCP
    - name: http
      port: 42114
      targetPort: 42114
      protocol: TCP
---
apiVersion: v1
kind: Service
metadata:
  name: hegel-external
  namespace: tink-system
spec:
  type: LoadBalancer
  selector:
    app: hegel
  ports:
    - name: http
      port: 50061
      targetPort: 50061
      protocol: TCP

kubectl apply -f tink-services.yaml
kubectl get svc -n tink-system

Add a minimal health endpoint if Tinkerbell components don't expose one natively:

# health-proxy.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: tinkerbell-health
  namespace: tink-system
spec:
  replicas: 1
  selector:
    matchLabels:
      app: tinkerbell-health
  template:
    metadata:
      labels:
        app: tinkerbell-health
    spec:
      containers:
        - name: health
          image: nginx:alpine
          ports:
            - containerPort: 80
          volumeMounts:
            - name: config
              mountPath: /etc/nginx/conf.d
      volumes:
        - name: config
          configMap:
            name: health-nginx-config
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: health-nginx-config
  namespace: tink-system
data:
  default.conf: |
    server {
      listen 80;
      location /health {
        return 200 '{"status":"ok"}';
        add_header Content-Type application/json;
      }
    }
---
apiVersion: v1
kind: Service
metadata:
  name: tinkerbell-health
  namespace: tink-system
spec:
  type: LoadBalancer
  selector:
    app: tinkerbell-health
  ports:
    - port: 80
      targetPort: 80

kubectl apply -f health-proxy.yaml
kubectl get svc tinkerbell-health -n tink-system
# NAME                  TYPE           CLUSTER-IP     EXTERNAL-IP     PORT(S)   AGE
# tinkerbell-health     LoadBalancer   10.96.15.3     203.0.113.90    80/TCP    1m

Step 2: Set up HTTP monitoring in Vigilmon

With health endpoints in place, add them to Vigilmon:

  1. Log in to vigilmon.online and go to Monitors → New Monitor
  2. Choose HTTP / HTTPS as the type
  3. Set the URL to your health endpoint: http://203.0.113.90/health
  4. Set the check interval to 1 minute
  5. Under Expected response, set:
    • Status code: 200
    • (Optional) Response body contains: "status":"ok"
  6. Save the monitor

Add a second monitor for the Hegel metadata server:

URL: http://203.0.113.91:50061/
Method: GET
Expected status: 200

Vigilmon probes from multiple geographic regions. If your Tinkerbell stack becomes unreachable — even while Kubernetes reports all pods as Running — an alert fires immediately.

Why multi-region consensus matters for bare metal

Tinkerbell environments are often single-site, single-datacenter deployments. A network partition or upstream ISP failure can make your provisioning stack unreachable to the outside world while Kubernetes sees everything as healthy. Vigilmon's multi-region consensus model requires multiple independent probes to agree before firing — eliminating false alarms from transient blips while catching real outages within seconds.

Step 3: Monitor TCP ports for Tink Server and Boots

Tinkerbell's gRPC and DHCP services are critical and can be monitored at the TCP layer for an additional layer of coverage:

Tink Server gRPC port:

  1. Go to Monitors → New Monitor
  2. Choose TCP Port
  3. Enter: Host 203.0.113.90, Port 42113
  4. Save the monitor

Hegel HTTP port:

  1. Choose TCP Port
  2. Enter: Host 203.0.113.91, Port 50061
  3. Save the monitor

TCP monitoring catches lower-level failures — a port that isn't listening, a firewall rule change, or a LoadBalancer routing failure — before they manifest as application-level errors.

For Boots, which serves DHCP, you can monitor the host's port 67 reachability from inside your provisioning network using a sidecar or test script:

# Test that Boots is serving DHCP from a test machine on the provisioning VLAN
nmap -sU -p 67 192.168.2.1

Step 4: Configure alert channels

A broken Tinkerbell stack means no new bare metal can be provisioned. This is often a silent failure that persists for hours until someone manually tries to add a server.

Email alerts

  1. In Vigilmon, go to Alert Channels → Add Channel → Email
  2. Enter your platform team's on-call email
  3. Assign the channel to your Tinkerbell monitors

Webhook alerts (Slack, PagerDuty, etc.)

  1. Go to Alert Channels → Add Channel → Webhook
  2. Enter your webhook URL
  3. Assign the channel to your monitors

Example payload when a Tinkerbell service goes down:

{
  "monitor_name": "Tinkerbell Health",
  "status": "down",
  "url": "http://203.0.113.90/health",
  "started_at": "2024-01-15T10:23:00Z",
  "duration_seconds": 180
}

Use this to trigger automated diagnostics:

# Run when Vigilmon fires an alert
kubectl get pods -n tink-system
kubectl logs -n tink-system -l app=tink-server --tail=50
kubectl logs -n tink-system -l app=boots --tail=50
kubectl logs -n tink-system -l app=hegel --tail=50

Step 5: Create a public status page

Give your platform and development teams visibility into the Tinkerbell provisioning stack without requiring Kubernetes access.

  1. In Vigilmon, go to Status Pages → New Status Page
  2. Give it a name: "Tinkerbell Provisioning Stack"
  3. Add your monitors: Health proxy, Tink Server gRPC, Hegel metadata
  4. Group them: Workflow Orchestration, Metadata Service, BMC Management
  5. Publish the page

Share the URL in your platform team's runbook and with any teams that depend on bare metal provisioning capacity.

Putting it all together

Here's a summary of the monitors to create for a typical Tinkerbell deployment:

| Monitor | Type | What it catches | |---------|------|-----------------| | http://203.0.113.90/health | HTTP | Overall Tinkerbell stack health | | http://203.0.113.91:50061/ | HTTP | Hegel metadata server availability | | 203.0.113.90:42113 | TCP | Tink Server gRPC port reachability | | 203.0.113.91:50061 | TCP | Hegel TCP reachability | | 203.0.113.90:42114 | TCP | Tink Server HTTP port |

And the diagnostics to run when an alert fires:

# Overall stack health
kubectl get pods -n tink-system -o wide

# Check Hardware and Template CRDs
kubectl get hardware -A
kubectl get template -A
kubectl get workflow -A

# Check Tink Server connectivity from a worker node
tink hardware list
tink workflow list

What's next

  • SSL certificate monitoring — if you expose Tink Server or Hegel over TLS, Vigilmon monitors the cert and alerts before it expires
  • Heartbeat monitoring — use Vigilmon's heartbeat monitors to verify that your Tinkerbell provisioning cron jobs and reconciliation loops are running on schedule
  • Multi-datacenter coverage — if you run Tinkerbell across multiple sites, create monitor groups per site to isolate failures geographically

Get started free at vigilmon.online — no credit card, monitors start running in under a minute.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles