Monitoring Terrajet (Upjet) with Vigilmon: Keeping Your Crossplane Provider Generation Pipeline Healthy

Terrajet — now known as Upjet — is the framework that turns any Terraform provider into a Crossplane managed resource provider. Teams building custom Crossplane providers on top of Terrajet generate entire suites of CRDs from Terraform resource schemas, enabling Kubernetes-native management of resources from providers that don't have a dedicated Crossplane implementation. When a Terrajet-generated provider breaks — pod crash, schema drift, Terraform binary errors — every managed resource it owns stops reconciling silently. Vigilmon gives you external monitoring that watches your Terrajet-generated providers from outside the cluster and alerts the moment your custom infrastructure provider is at risk.

What You'll Build

• A Vigilmon HTTP monitor for Terrajet provider health endpoints
• A heartbeat monitor to verify active managed resource reconciliation
• A schema-sync probe that detects Terraform provider schema drift
• Alert channels routed to your platform engineering team

Prerequisites

• A Kubernetes cluster with Crossplane installed (version 1.14+)
• A Terrajet/Upjet-generated provider installed and running
• A reverse proxy or Ingress exposing provider health endpoints over HTTPS
• A free account at vigilmon.online

Why Monitoring Terrajet Providers Matters

Terrajet-generated providers are more complex than hand-written Crossplane providers because they embed Terraform logic — they shell out to Terraform binaries, translate Kubernetes spec fields to Terraform HCL, and manage Terraform state in Kubernetes Secrets. This creates unique failure modes:

Terraform binary failures occur when the embedded Terraform provider binary crashes, is incompatible with the Terrajet runtime version, or fails to initialize. The controller pod may stay running while every terraform apply call inside it fails silently.

Schema drift happens when the upstream Terraform provider schema changes between the version used during generation and the version currently embedded. Fields disappear or change type, causing managed resource validation to fail with opaque errors.

State corruption in the Terraform workspace Secrets can leave managed resources permanently stuck — the Kubernetes object exists and appears healthy, but the underlying Terraform state is inconsistent and no reconciliation loop can fix it.

Pod OOM kills are common with Terrajet providers managing large Terraform states. A single large resource (dozens of outputs, complex state JSON) can push memory over the container limit.

External monitoring from Vigilmon catches these from the perspective of the systems depending on working infrastructure provisioning.

Step 1: Expose the Terrajet Provider Health Endpoint

Terrajet-generated providers expose a health endpoint on port 8081, identical to other Crossplane providers. Identify your provider's pod:

# List Terrajet provider pods
kubectl get pods -n crossplane-system -l pkg.crossplane.io/revision

# Verify the health endpoint
kubectl exec -n crossplane-system \
  $(kubectl get pod -n crossplane-system -l pkg.crossplane.io/revision=provider-upjet-gcp -o name | head -1) \
  -- wget -qO- http://localhost:8081/healthz

Create a Service to expose the health endpoint:

# upjet-provider-health-svc.yaml
apiVersion: v1
kind: Service
metadata:
  name: upjet-provider-health
  namespace: crossplane-system
  labels:
    app: upjet-provider-health
spec:
  selector:
    pkg.crossplane.io/revision: provider-upjet-gcp  # adjust to your provider
  ports:
    - name: health
      port: 8081
      targetPort: 8081

Expose it through an Ingress:

# upjet-provider-health-ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: upjet-provider-health
  namespace: crossplane-system
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /healthz
spec:
  rules:
    - host: upjet-provider-health.internal.yourdomain.com
      http:
        paths:
          - path: /healthz
            pathType: Prefix
            backend:
              service:
                name: upjet-provider-health
                port:
                  number: 8081

Apply both and verify:

kubectl apply -f upjet-provider-health-svc.yaml
kubectl apply -f upjet-provider-health-ingress.yaml

curl https://upjet-provider-health.internal.yourdomain.com/healthz
# Expected: {"status":"ok"}

Step 2: Add the Vigilmon HTTP Monitor

In the Vigilmon dashboard:

1. Go to Add Monitor → HTTP
2. Configure the monitor:

| Field | Value | |---|---| | Name | upjet-provider /healthz | | URL | https://upjet-provider-health.internal.yourdomain.com/healthz | | Method | GET | | Expected status | 200 | | Response body contains | ok | | Check interval | 1 minute | | Timeout | 10 seconds | | Alert after | 2 consecutive failures |

The 2-failure threshold absorbs transient pod restarts during provider upgrades while catching genuine provider crashes within 2 minutes.

Step 3: Heartbeat Monitor for Managed Resource Reconciliation

A running, healthy pod can still fail to reconcile if the embedded Terraform binary is broken. Add a CronJob that probes actual reconciliation activity:

# upjet-reconcile-probe.yaml
apiVersion: batch/v1
kind: CronJob
metadata:
  name: upjet-reconcile-probe
  namespace: crossplane-system
spec:
  schedule: "*/10 * * * *"
  jobTemplate:
    spec:
      template:
        spec:
          restartPolicy: OnFailure
          serviceAccountName: crossplane-probe
          containers:
            - name: reconcile-probe
              image: bitnami/kubectl:latest
              env:
                - name: PROVIDER_NAME
                  value: provider-upjet-gcp  # adjust to your provider
                - name: HEARTBEAT_URL
                  valueFrom:
                    secretKeyRef:
                      name: vigilmon-secrets
                      key: upjet-reconcile-heartbeat-url
              command:
                - /bin/sh
                - -c
                - |
                  set -e
                  # Verify provider package is installed and healthy
                  HEALTHY=$(kubectl get provider "$PROVIDER_NAME" \
                    -o jsonpath='{.status.conditions[?(@.type=="Healthy")].status}' 2>/dev/null)
                  if [ "$HEALTHY" != "True" ]; then
                    echo "FAIL: Provider $PROVIDER_NAME is not healthy (status: $HEALTHY)"
                    exit 1
                  fi

                  # Check reconciliation controller is active
                  INSTALLED=$(kubectl get provider "$PROVIDER_NAME" \
                    -o jsonpath='{.status.conditions[?(@.type=="Installed")].status}' 2>/dev/null)
                  if [ "$INSTALLED" != "True" ]; then
                    echo "FAIL: Provider $PROVIDER_NAME is not installed"
                    exit 1
                  fi

                  # Count managed resources stuck in error
                  STUCK=$(kubectl get managed \
                    --all-namespaces \
                    -o jsonpath='{range .items[*]}{.status.conditions[?(@.type=="Synced")].status}{"\n"}{end}' \
                    2>/dev/null | grep -c "False" || true)
                  if [ "$STUCK" -gt 3 ]; then
                    echo "FAIL: $STUCK managed resources not synced"
                    exit 1
                  fi

                  wget -qO- "$HEARTBEAT_URL"
                  echo "Upjet reconciliation heartbeat sent"

Create the heartbeat monitor in Vigilmon:

1. Add Monitor → Heartbeat
2. Name: upjet-provider reconciliation
3. Expected interval: 10 minutes
4. Grace period: 5 minutes

Store the heartbeat URL:

kubectl create secret generic vigilmon-secrets \
  --from-literal=upjet-reconcile-heartbeat-url='https://vigilmon.online/heartbeat/YOUR_TOKEN' \
  -n crossplane-system

Step 4: Detect Terraform Schema Drift

Schema drift is one of the hardest Terrajet failure modes to catch because it manifests as validation errors on new resource creation rather than pod-level health issues. Add a probe that creates and deletes a minimal managed resource:

# upjet-schema-probe.yaml
apiVersion: batch/v1
kind: CronJob
metadata:
  name: upjet-schema-probe
  namespace: crossplane-system
spec:
  schedule: "0 */6 * * *"
  jobTemplate:
    spec:
      template:
        spec:
          restartPolicy: OnFailure
          serviceAccountName: crossplane-probe
          containers:
            - name: schema-probe
              image: bitnami/kubectl:latest
              env:
                - name: HEARTBEAT_URL
                  valueFrom:
                    secretKeyRef:
                      name: vigilmon-secrets
                      key: upjet-schema-heartbeat-url
              command:
                - /bin/sh
                - -c
                - |
                  set -e
                  # Apply a minimal canary managed resource
                  cat <<EOF | kubectl apply -f -
                  apiVersion: storage.gcp.upbound.io/v1beta1
                  kind: BucketIAMMember
                  metadata:
                    name: upjet-schema-canary
                    namespace: crossplane-system
                    annotations:
                      vigilmon.io/canary: "true"
                  spec:
                    forProvider:
                      bucket: vigilmon-canary-nonexistent
                      member: serviceAccount:probe@example.iam.gserviceaccount.com
                      role: roles/storage.objectViewer
                    providerConfigRef:
                      name: default
                  EOF

                  # Wait briefly for admission webhook validation
                  sleep 5

                  # If the object was accepted by the webhook, schema is consistent
                  kubectl get bucketiammember upjet-schema-canary -n crossplane-system

                  # Clean up
                  kubectl delete bucketiammember upjet-schema-canary -n crossplane-system --ignore-not-found

                  wget -qO- "$HEARTBEAT_URL"
                  echo "Schema validation heartbeat sent"

This probe validates that the Terrajet provider's webhook and CRD schema accept the resource type correctly. If schema drift has caused CRD validation to break, the kubectl apply fails and the heartbeat is never sent.

Step 5: Configure Alert Channels

Route Terrajet provider alerts to your platform engineering team:

1. In Vigilmon: Alert Channels → Add Channel → Slack Webhook
2. Paste your #platform-alerts Slack webhook URL
3. Assign the channel to all Upjet monitors

For schema drift detection (which blocks new infrastructure creation):

1. Alert Channels → Add Channel → PagerDuty (or email)
2. Add your on-call rotation
3. Assign it only to the upjet-schema heartbeat monitor

Schema drift is a category-1 incident because it silently blocks all new resource provisioning while existing resources appear healthy.

What You're Now Monitoring

| Component | Monitor Type | What It Detects | |---|---|---| | Provider pod | HTTP /healthz | Pod crash or controller unhealthy | | Reconciliation loop | Heartbeat (10 min) | Provider stalled, managed resources not syncing | | CRD schema | Heartbeat (6 hr) | Terraform schema drift breaking resource admission |

With these monitors in place, your team catches Terrajet provider failures within minutes rather than discovering them when a cluster operator reports that new managed resources can no longer be created.

Conclusion

Terrajet-generated providers carry unique risk because they embed Terraform logic in the Kubernetes control plane — a layer most platform engineers don't routinely inspect. External monitoring with Vigilmon closes the gap between the Kubernetes health APIs and the actual state of your Terraform-backed infrastructure. Whether you're running provider-upjet-gcp, provider-upjet-azure, or a custom internal provider built on the Upjet framework, the patterns described here apply directly.

Get started free at vigilmon.online. Adding Terrajet provider health monitoring takes under five minutes, and the schema drift probe pattern catches the failure mode that pod-level health checks miss entirely.