tutorial

Monitoring Crossplane Provider Terraform with Vigilmon: Keeping Your Kubernetes-Managed Terraform Workspaces Healthy

How to monitor Crossplane Provider Terraform with Vigilmon — tracking workspace controller health, Terraform execution state, workspace reconciliation, and provider endpoints from outside your cluster.

Crossplane Provider Terraform bridges the gap between Crossplane and the vast Terraform ecosystem — it lets you declare a Workspace resource in Kubernetes and have Crossplane run terraform apply against it, managing arbitrary Terraform configurations as Kubernetes custom resources. Teams use it to migrate existing Terraform codebases into Crossplane without rewriting providers, or to manage resources in ecosystems where no dedicated Crossplane provider exists. When Provider Terraform fails — workspace controller crash, backend connectivity loss, Terraform execution errors — all managed workspaces stall silently. Vigilmon gives you external monitoring that catches provider failures before they cascade into blocked infrastructure pipelines.

What You'll Build

  • A Vigilmon HTTP monitor for the Provider Terraform health endpoint
  • A heartbeat monitor to verify active workspace reconciliation
  • A workspace execution probe that validates Terraform can actually run
  • Alert channels routed to your platform team

Prerequisites

  • A Kubernetes cluster with Crossplane installed (version 1.14+)
  • provider-terraform installed and a ProviderConfig referencing a valid Terraform backend
  • A reverse proxy or Ingress exposing provider health endpoints over HTTPS
  • A free account at vigilmon.online

Why Monitoring Crossplane Provider Terraform Matters

Provider Terraform has a unique set of failure modes compared to standard Crossplane providers:

Workspace controller crashes stop all terraform plan and terraform apply operations across every Workspace resource. Infrastructure declared via Workspaces freezes in its last-known state. Applications waiting on outputs — IPs, ARNs, connection strings — never receive them.

Backend connectivity loss is the most common silent failure. If the Terraform backend (S3, GCS, Consul, Kubernetes Secret) becomes unreachable, workspace reconciliation fails with BackendInitialization errors. The provider pod stays healthy but no state operations succeed.

Git credential expiry affects workspaces sourcing modules from private Git repositories. When tokens expire, workspace sync fails on the module download step, and the actual apply never runs. From the Kubernetes resource perspective, the workspace appears Synced: False but only if you're watching it.

Terraform binary version mismatches cause subtle failures when a workspace's required Terraform version differs from the one embedded in the provider. Workspaces using required_version constraints fail initialization silently.

External monitoring with Vigilmon detects these conditions from outside the cluster.

Step 1: Expose the Provider Terraform Health Endpoint

Provider Terraform runs as a controller pod in crossplane-system with a standard health endpoint:

# Find the provider pod
kubectl get pods -n crossplane-system | grep provider-terraform

# Verify the health endpoint
kubectl exec -n crossplane-system \
  $(kubectl get pod -n crossplane-system -l pkg.crossplane.io/revision=provider-terraform -o name | head -1) \
  -- wget -qO- http://localhost:8081/healthz

Expose it via a Service:

# provider-terraform-health-svc.yaml
apiVersion: v1
kind: Service
metadata:
  name: provider-terraform-health
  namespace: crossplane-system
  labels:
    app: provider-terraform-health
spec:
  selector:
    pkg.crossplane.io/revision: provider-terraform
  ports:
    - name: health
      port: 8081
      targetPort: 8081

Add an Ingress:

# provider-terraform-health-ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: provider-terraform-health
  namespace: crossplane-system
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /healthz
spec:
  rules:
    - host: provider-terraform-health.internal.yourdomain.com
      http:
        paths:
          - path: /healthz
            pathType: Prefix
            backend:
              service:
                name: provider-terraform-health
                port:
                  number: 8081

Apply and verify:

kubectl apply -f provider-terraform-health-svc.yaml
kubectl apply -f provider-terraform-health-ingress.yaml

curl https://provider-terraform-health.internal.yourdomain.com/healthz
# Expected: {"status":"ok"}

Step 2: Add the Vigilmon HTTP Monitor

In the Vigilmon dashboard:

  1. Go to Add Monitor → HTTP
  2. Configure the monitor:

| Field | Value | |---|---| | Name | provider-terraform /healthz | | URL | https://provider-terraform-health.internal.yourdomain.com/healthz | | Method | GET | | Expected status | 200 | | Response body contains | ok | | Check interval | 1 minute | | Timeout | 10 seconds | | Alert after | 2 consecutive failures |

Two consecutive failures give the provider time to restart after a rolling update before triggering an alert.

Step 3: Heartbeat Monitor for Workspace Reconciliation

The pod health check doesn't validate that workspaces are actually reconciling. Add a CronJob that inspects workspace status:

# provider-terraform-reconcile-probe.yaml
apiVersion: batch/v1
kind: CronJob
metadata:
  name: provider-terraform-reconcile-probe
  namespace: crossplane-system
spec:
  schedule: "*/10 * * * *"
  jobTemplate:
    spec:
      template:
        spec:
          restartPolicy: OnFailure
          serviceAccountName: crossplane-probe
          containers:
            - name: reconcile-probe
              image: bitnami/kubectl:latest
              env:
                - name: HEARTBEAT_URL
                  valueFrom:
                    secretKeyRef:
                      name: vigilmon-secrets
                      key: provider-terraform-reconcile-heartbeat-url
              command:
                - /bin/sh
                - -c
                - |
                  set -e
                  # Verify provider package health
                  HEALTHY=$(kubectl get provider provider-terraform \
                    -o jsonpath='{.status.conditions[?(@.type=="Healthy")].status}' 2>/dev/null)
                  if [ "$HEALTHY" != "True" ]; then
                    echo "FAIL: provider-terraform is not healthy"
                    exit 1
                  fi

                  # Count workspaces in error state
                  FAILED=$(kubectl get workspace \
                    --all-namespaces \
                    -o jsonpath='{range .items[*]}{.status.conditions[?(@.type=="Synced")].status}{"\n"}{end}' \
                    2>/dev/null | grep -c "False" || true)

                  TOTAL=$(kubectl get workspace --all-namespaces --no-headers 2>/dev/null | wc -l || true)

                  # Alert if more than 20% of workspaces are failing
                  if [ "$TOTAL" -gt 0 ] && [ "$FAILED" -gt "$(( TOTAL / 5 ))" ]; then
                    echo "FAIL: $FAILED of $TOTAL workspaces not synced"
                    exit 1
                  fi

                  wget -qO- "$HEARTBEAT_URL"
                  echo "Workspace reconciliation heartbeat sent ($FAILED/$TOTAL failing)"

Create the heartbeat monitor in Vigilmon:

  1. Add Monitor → Heartbeat
  2. Name: provider-terraform workspaces
  3. Expected interval: 10 minutes
  4. Grace period: 5 minutes

Step 4: Validate Terraform Execution End-to-End

The most critical probe confirms that Terraform can actually plan and apply. Create a dedicated canary workspace:

# terraform-canary-workspace.yaml
apiVersion: tf.upbound.io/v1beta1
kind: Workspace
metadata:
  name: vigilmon-canary
  namespace: crossplane-system
spec:
  forProvider:
    source: Inline
    module: |
      output "canary" {
        value = "vigilmon-ok"
      }
  writeConnectionSecretToRef:
    namespace: crossplane-system
    name: vigilmon-canary-outputs
  providerConfigRef:
    name: default

Then add a CronJob to verify the canary workspace output stays current:

# terraform-canary-probe.yaml
apiVersion: batch/v1
kind: CronJob
metadata:
  name: terraform-canary-probe
  namespace: crossplane-system
spec:
  schedule: "*/20 * * * *"
  jobTemplate:
    spec:
      template:
        spec:
          restartPolicy: OnFailure
          serviceAccountName: crossplane-probe
          containers:
            - name: canary-probe
              image: bitnami/kubectl:latest
              env:
                - name: HEARTBEAT_URL
                  valueFrom:
                    secretKeyRef:
                      name: vigilmon-secrets
                      key: terraform-canary-heartbeat-url
              command:
                - /bin/sh
                - -c
                - |
                  set -e
                  # Force reconciliation of canary workspace
                  kubectl annotate workspace vigilmon-canary \
                    -n crossplane-system \
                    crossplane.io/paused="false" \
                    --overwrite

                  # Wait for the workspace to sync
                  for i in $(seq 1 12); do
                    STATUS=$(kubectl get workspace vigilmon-canary -n crossplane-system \
                      -o jsonpath='{.status.conditions[?(@.type=="Synced")].status}' 2>/dev/null)
                    if [ "$STATUS" = "True" ]; then
                      break
                    fi
                    sleep 5
                  done

                  # Read the canary output from the connection secret
                  OUTPUT=$(kubectl get secret vigilmon-canary-outputs \
                    -n crossplane-system \
                    -o jsonpath='{.data.canary}' 2>/dev/null | base64 -d)

                  if [ "$OUTPUT" != "vigilmon-ok" ]; then
                    echo "FAIL: Canary output mismatch: $OUTPUT"
                    exit 1
                  fi

                  wget -qO- "$HEARTBEAT_URL"
                  echo "Terraform execution heartbeat sent"

Apply the canary workspace and probe:

kubectl apply -f terraform-canary-workspace.yaml
kubectl apply -f terraform-canary-probe.yaml

kubectl create secret generic vigilmon-secrets \
  --from-literal=provider-terraform-reconcile-heartbeat-url='https://vigilmon.online/heartbeat/YOUR_RECONCILE_TOKEN' \
  --from-literal=terraform-canary-heartbeat-url='https://vigilmon.online/heartbeat/YOUR_CANARY_TOKEN' \
  -n crossplane-system

Step 5: Configure Alert Channels

Route Provider Terraform alerts to your platform team:

  1. In Vigilmon: Alert Channels → Add Channel → Slack Webhook
  2. Paste your #platform-alerts webhook URL
  3. Assign to all Provider Terraform monitors

For the canary execution monitor (which signals complete Terraform pipeline failure):

  1. Alert Channels → Add Channel → PagerDuty or email
  2. Assign only to the terraform-canary heartbeat

The canary failing means all workspace reconciliation may be broken — escalate immediately.

What You're Now Monitoring

| Component | Monitor Type | What It Detects | |---|---|---| | Provider pod | HTTP /healthz | Controller crash or restart loop | | Workspace reconciliation | Heartbeat (10 min) | Workspaces stuck, backend connectivity loss | | Terraform execution | Heartbeat (20 min) | Plan/apply broken end-to-end |

Conclusion

Crossplane Provider Terraform lets you adopt Crossplane incrementally without abandoning your existing Terraform investment. But it adds a new failure surface — Terraform execution inside Kubernetes — that conventional Kubernetes health checks don't reach. External monitoring with Vigilmon closes this gap, giving your platform team visibility into actual Terraform execution state rather than just pod liveness.

Get started free at vigilmon.online. The canary workspace pattern described here gives you an end-to-end execution signal in under ten minutes, and it generalizes to any Workspace-backed infrastructure your team manages.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles