tutorial

Monitoring Crossplane with Vigilmon: Health Checks for Your Kubernetes Infrastructure Control Plane

How to monitor Crossplane with Vigilmon — tracking provider health, composite resource API endpoints, webhook availability, and controller readiness from outside your cluster.

Crossplane turns your Kubernetes cluster into a universal infrastructure control plane. Instead of writing Terraform modules or clicking through cloud consoles, you declare AWS S3 buckets, GCP CloudSQL instances, and Azure VNets as Kubernetes custom resources — and Crossplane's providers reconcile them into reality. That power comes with a new reliability concern: if Crossplane's controllers, providers, or webhooks go down, infrastructure provisioning silently stalls. New environments don't get created, database claims stay pending, and the developers waiting on them have no idea why. Vigilmon gives you external monitoring that watches Crossplane from outside the cluster and alerts the moment something breaks.

What You'll Build

  • A Vigilmon HTTP monitor for Crossplane's webhook server
  • Healthcheck monitors for each installed provider
  • A heartbeat monitor to verify the Crossplane reconciliation loop is running
  • Alert channels so your platform team gets notified before developers notice

Prerequisites

  • Crossplane installed in a Kubernetes cluster (version 1.14+)
  • At least one provider installed (e.g., provider-aws, provider-gcp)
  • A reverse proxy or load balancer exposing Crossplane health endpoints over HTTPS
  • A free account at vigilmon.online

Why Monitoring Crossplane Matters

Crossplane runs as a set of Kubernetes controllers plus a mutating/validating webhook. Failures here are particularly damaging because they're invisible to application teams:

Provider pods crashing means no infrastructure gets reconciled. New Bucket or RDSInstance claims sit in Pending state forever with no clear error surfaced to the requester.

Webhook server downtime is catastrophic — Kubernetes rejects all writes to Crossplane resources (CRDs) if the webhook is unreachable, since it's registered with failurePolicy: Fail by default. This can block deployments that depend on infrastructure claims.

Composition function failures break complex resource pipelines silently. A function that generates 15 managed resources from one claim might fail at step 8 with no alert to the platform team.

External monitoring from Vigilmon catches these failures from the perspective of the users and automation that depend on Crossplane.

Step 1: Expose Health Endpoints

Crossplane and its providers expose health endpoints on their pods. The quickest way to make them accessible externally is to create a Kubernetes Service and Ingress for each.

The Crossplane manager pod exposes /healthz and /readyz on port 8081 by default:

# Check the healthz endpoint from inside the cluster
kubectl exec -n crossplane-system deploy/crossplane -- \
  wget -qO- http://localhost:8081/healthz

Expose it via an Ingress:

# crossplane-health-ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: crossplane-health
  namespace: crossplane-system
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /
spec:
  rules:
    - host: crossplane-health.internal.yourdomain.com
      http:
        paths:
          - path: /healthz
            pathType: Prefix
            backend:
              service:
                name: crossplane
                port:
                  number: 8081
          - path: /readyz
            pathType: Prefix
            backend:
              service:
                name: crossplane
                port:
                  number: 8081

Apply it:

kubectl apply -f crossplane-health-ingress.yaml

Verify the endpoint resolves externally:

curl https://crossplane-health.internal.yourdomain.com/healthz
# Expected: {"status":"ok"}

Step 2: Monitor Provider Health Endpoints

Each installed Crossplane provider runs as its own pod with its own /healthz endpoint. A provider crash stops reconciliation of every managed resource it owns.

List your providers and their health ports:

kubectl get pods -n crossplane-system -l pkg.crossplane.io/revision

Create a small script to expose all provider healthz endpoints:

#!/bin/bash
# expose-provider-health.sh
# For each provider pod, port-forward and verify health
for pod in $(kubectl get pods -n crossplane-system -l pkg.crossplane.io/revision -o name); do
  echo "Checking $pod"
  kubectl exec -n crossplane-system "$pod" -- \
    wget -qO- http://localhost:8081/healthz 2>/dev/null || echo "UNHEALTHY"
done

Create a Service for each provider to expose its health endpoint through your Ingress. For provider-aws:

# provider-aws-health-svc.yaml
apiVersion: v1
kind: Service
metadata:
  name: provider-aws-health
  namespace: crossplane-system
spec:
  selector:
    pkg.crossplane.io/revision: provider-aws
  ports:
    - port: 8081
      targetPort: 8081

Then add Vigilmon HTTP monitors for each provider health URL (Step 3 covers the UI steps).

Step 3: Add Vigilmon HTTP Monitors

In the Vigilmon dashboard:

  1. Go to Add Monitor → HTTP
  2. Create monitors for each endpoint:

| Monitor Name | URL | Expected Status | |---|---|---| | crossplane manager /healthz | https://crossplane-health.yourdomain.com/healthz | 200 | | crossplane manager /readyz | https://crossplane-health.yourdomain.com/readyz | 200 | | provider-aws /healthz | https://provider-aws-health.yourdomain.com/healthz | 200 | | provider-gcp /healthz | https://provider-gcp-health.yourdomain.com/healthz | 200 |

For each monitor:

  • Check interval: 1 minute
  • Timeout: 10 seconds
  • Expected status code: 200
  • Response body contains: ok
  • Alert threshold: Alert after 2 consecutive failures (reduces false positives)

Vigilmon probes from multiple geographic regions. You need both a cloud-local and truly external probe — configure the Ingress to restrict access by IP if needed while still allowing Vigilmon's probe IPs through.

Step 4: Monitor the Crossplane Webhook

The Crossplane webhook server handles admission for all Crossplane custom resources. If it's down, Kubernetes will reject writes to CRDs with failurePolicy: Fail. This is the highest-severity failure mode.

The webhook runs on port 9443 within the cluster. To monitor it externally:

# Verify webhook is responding inside the cluster
kubectl exec -n crossplane-system deploy/crossplane -- \
  wget -qO- --no-check-certificate https://localhost:9443/healthz 2>&1

Expose the webhook health endpoint via Ingress (separate from the main health endpoint so you can track them independently in Vigilmon):

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: crossplane-webhook-health
  namespace: crossplane-system
spec:
  rules:
    - host: crossplane-webhook.internal.yourdomain.com
      http:
        paths:
          - path: /healthz
            pathType: Prefix
            backend:
              service:
                name: crossplane
                port:
                  number: 9443

Add a Vigilmon monitor for this URL with a P0 alert channel — webhook downtime is your highest-priority alert.

Step 5: Heartbeat Monitor for Reconciliation

Provider health endpoints confirm the pod is alive, but not that reconciliation is actually succeeding. Use a Crossplane CronJob that creates and deletes a test resource to verify the full reconciliation loop:

# crossplane-smoke-test-cronjob.yaml
apiVersion: batch/v1
kind: CronJob
metadata:
  name: crossplane-smoke-test
  namespace: crossplane-system
spec:
  schedule: "*/15 * * * *"
  jobTemplate:
    spec:
      template:
        spec:
          restartPolicy: OnFailure
          serviceAccountName: crossplane-smoke-test
          containers:
            - name: smoke-test
              image: bitnami/kubectl:latest
              env:
                - name: HEARTBEAT_URL
                  valueFrom:
                    secretKeyRef:
                      name: vigilmon-secrets
                      key: crossplane-heartbeat-url
              command:
                - /bin/sh
                - -c
                - |
                  set -e
                  # Verify Crossplane CRDs are reachable and providers are healthy
                  kubectl get providers.pkg.crossplane.io -n crossplane-system
                  # Check all providers are HEALTHY=True
                  UNHEALTHY=$(kubectl get providers.pkg.crossplane.io \
                    -o jsonpath='{range .items[*]}{.metadata.name}{" "}{.status.conditions[?(@.type=="Healthy")].status}{"\n"}{end}' \
                    | grep -v "True" | wc -l)
                  if [ "$UNHEALTHY" -gt 0 ]; then
                    echo "FAIL: $UNHEALTHY providers are not healthy"
                    exit 1
                  fi
                  # Send heartbeat only if everything is healthy
                  wget -qO- "$HEARTBEAT_URL"
                  echo "Heartbeat sent"

Create the heartbeat monitor in Vigilmon:

  1. Go to Add Monitor → Heartbeat
  2. Name: crossplane reconciliation loop
  3. Expected interval: 15 minutes
  4. Grace period: 5 minutes
  5. Copy the heartbeat URL into a Kubernetes Secret:
kubectl create secret generic vigilmon-secrets \
  --from-literal=crossplane-heartbeat-url='https://vigilmon.online/heartbeat/YOUR_TOKEN' \
  -n crossplane-system

If the smoke test fails (a provider is unhealthy, CRDs are unreachable, or the job itself crashes), the heartbeat ping won't fire — Vigilmon opens an incident after the grace period.

Step 6: Configure Alert Channels

Route Crossplane alerts to your platform team's Slack channel:

  1. In Vigilmon: Alert Channels → Add Channel → Slack Webhook
  2. Paste your #platform-alerts channel webhook URL
  3. Assign the channel to all Crossplane monitors

For webhook downtime (the highest-severity scenario), add a PagerDuty channel as well:

  1. Alert Channels → Add Channel → PagerDuty
  2. Paste your PagerDuty integration key
  3. Assign to the crossplane webhook /healthz monitor only

This gives you tiered alerting: most Crossplane issues go to Slack, but webhook downtime pages on-call immediately.

What You're Now Monitoring

| Component | Monitor Type | What It Detects | |---|---|---| | Crossplane manager | HTTP /healthz | Controller pod crash | | Crossplane manager | HTTP /readyz | Controller not ready to reconcile | | Crossplane webhook | HTTP /healthz | Admission webhook down (blocks all CRD writes) | | provider-aws | HTTP /healthz | AWS provider crash | | provider-gcp | HTTP /healthz | GCP provider crash | | Reconciliation loop | Heartbeat | Providers healthy but not reconciling |

Between these monitors, you'll know about Crossplane failures within 1–2 minutes — long before your developers start wondering why their database claims are stuck in Pending.

Conclusion

Crossplane is powerful precisely because it abstracts infrastructure into Kubernetes resources — but that abstraction means failures are further removed from the developers who depend on it. External monitoring with Vigilmon gives your platform team the visibility needed to catch problems before they cascade into multi-hour provisioning delays.

Get started free at vigilmon.online. Your first Crossplane health monitor takes under two minutes to configure, and the free tier is sufficient to cover the core failure modes described above.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles