Monitoring Crossplane Provider AWS with Vigilmon: Keeping Your AWS Infrastructure Reconciliation Healthy

Crossplane Provider AWS is the bridge between your Kubernetes cluster and your AWS account. Declare an RDSInstance, S3Bucket, or VPC as a Kubernetes resource, and the provider reconciles it against the real AWS API. When this provider goes down — pod crash, credentials expired, AWS API throttling — your infrastructure provisioning silently stalls. Developers waiting on database claims, teams blocked on network resources, and CI pipelines expecting fresh buckets all hit a wall with no obvious error. Vigilmon gives you external monitoring that watches Provider AWS from outside the cluster and alerts the moment reconciliation is at risk.

What You'll Build

• A Vigilmon HTTP monitor for Provider AWS's health endpoint
• A heartbeat monitor to verify active AWS resource reconciliation
• Credential rotation detection via a custom probe
• Alert channels routed to your platform team

Prerequisites

• Crossplane installed in a Kubernetes cluster (version 1.14+)
• provider-aws installed and configured with AWS credentials
• A reverse proxy or load balancer exposing provider health endpoints over HTTPS
• A free account at vigilmon.online

Why Monitoring Crossplane Provider AWS Matters

Provider AWS is a long-running Kubernetes controller with several distinct failure modes:

Pod crashes stop all AWS resource reconciliation immediately. Every managed resource the provider owns — EC2 instances, RDS databases, S3 buckets, IAM roles — stops being reconciled. Drift from the declared state accumulates silently until the provider restarts.

Expired or revoked AWS credentials cause the provider to start returning 401 Unauthorized errors from the AWS API. The pod stays running and healthy by pod metrics, but no actual AWS API calls succeed. Resources report Synced: False but only if you're watching them.

AWS API throttling degrades reconciliation throughput. High-traffic clusters can hit AWS rate limits, causing reconciliation backlogs that look like slow provisioning but are actually a systemic problem.

CRD webhook failures — if Provider AWS's conversion webhook is down, any writes to its CRDs fail with admission errors, blocking infrastructure provisioning pipelines entirely.

External monitoring from Vigilmon catches these from the perspective of the systems that depend on working AWS infrastructure.

Step 1: Expose the Provider AWS Health Endpoint

Provider AWS runs as a pod in the crossplane-system namespace with a health endpoint on port 8081:

# Verify the health endpoint from inside the cluster
kubectl exec -n crossplane-system \
  $(kubectl get pod -n crossplane-system -l pkg.crossplane.io/revision=provider-aws -o name | head -1) \
  -- wget -qO- http://localhost:8081/healthz

Create a Kubernetes Service to expose it:

# provider-aws-health-svc.yaml
apiVersion: v1
kind: Service
metadata:
  name: provider-aws-health
  namespace: crossplane-system
  labels:
    app: provider-aws-health
spec:
  selector:
    pkg.crossplane.io/revision: provider-aws
  ports:
    - name: health
      port: 8081
      targetPort: 8081

Then expose it through an Ingress:

# provider-aws-health-ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: provider-aws-health
  namespace: crossplane-system
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /healthz
spec:
  rules:
    - host: provider-aws-health.internal.yourdomain.com
      http:
        paths:
          - path: /healthz
            pathType: Prefix
            backend:
              service:
                name: provider-aws-health
                port:
                  number: 8081

Apply both:

kubectl apply -f provider-aws-health-svc.yaml
kubectl apply -f provider-aws-health-ingress.yaml

Verify the endpoint resolves from outside the cluster:

curl https://provider-aws-health.internal.yourdomain.com/healthz
# Expected: {"status":"ok"}

Step 2: Add the Vigilmon HTTP Monitor

In the Vigilmon dashboard:

1. Go to Add Monitor → HTTP
2. Configure the monitor:

| Field | Value | |---|---| | Name | provider-aws /healthz | | URL | https://provider-aws-health.internal.yourdomain.com/healthz | | Method | GET | | Expected status | 200 | | Response body contains | ok | | Check interval | 1 minute | | Timeout | 10 seconds | | Alert after | 2 consecutive failures |

The "2 consecutive failures" threshold prevents false positives from transient pod restarts during upgrades, while still catching genuine provider outages within 2 minutes.

Step 3: Monitor AWS Credential Validity

Provider AWS credential expiry is the most insidious failure mode — the pod reports healthy, but no AWS calls succeed. Add a probe that directly checks whether the provider can reach the AWS API.

Create a CronJob that uses the AWS CLI with the provider's credentials to make a lightweight, read-only API call:

# provider-aws-cred-probe.yaml
apiVersion: batch/v1
kind: CronJob
metadata:
  name: provider-aws-cred-probe
  namespace: crossplane-system
spec:
  schedule: "*/10 * * * *"
  jobTemplate:
    spec:
      template:
        spec:
          restartPolicy: OnFailure
          serviceAccountName: crossplane-probe
          containers:
            - name: aws-probe
              image: amazon/aws-cli:latest
              env:
                - name: AWS_ACCESS_KEY_ID
                  valueFrom:
                    secretKeyRef:
                      name: aws-provider-creds
                      key: access_key_id
                - name: AWS_SECRET_ACCESS_KEY
                  valueFrom:
                    secretKeyRef:
                      name: aws-provider-creds
                      key: secret_access_key
                - name: AWS_DEFAULT_REGION
                  value: us-east-1
                - name: HEARTBEAT_URL
                  valueFrom:
                    secretKeyRef:
                      name: vigilmon-secrets
                      key: provider-aws-creds-heartbeat-url
              command:
                - /bin/sh
                - -c
                - |
                  set -e
                  # Lightweight STS call to verify credentials are valid
                  aws sts get-caller-identity --output text
                  # Send heartbeat only if credentials work
                  wget -qO- "$HEARTBEAT_URL"
                  echo "AWS credential heartbeat sent"

This CronJob fires every 10 minutes. If the credentials are expired or revoked, aws sts get-caller-identity fails, the heartbeat ping is never sent, and Vigilmon opens an incident after the grace period.

Step 4: Heartbeat Monitor for Reconciliation Activity

A running pod with valid credentials still might not be reconciling. Use a CronJob that inspects whether Provider AWS is actively processing managed resources:

# provider-aws-reconcile-probe.yaml
apiVersion: batch/v1
kind: CronJob
metadata:
  name: provider-aws-reconcile-probe
  namespace: crossplane-system
spec:
  schedule: "*/15 * * * *"
  jobTemplate:
    spec:
      template:
        spec:
          restartPolicy: OnFailure
          serviceAccountName: crossplane-probe
          containers:
            - name: reconcile-probe
              image: bitnami/kubectl:latest
              env:
                - name: HEARTBEAT_URL
                  valueFrom:
                    secretKeyRef:
                      name: vigilmon-secrets
                      key: provider-aws-reconcile-heartbeat-url
              command:
                - /bin/sh
                - -c
                - |
                  set -e
                  # Check the provider is installed and healthy
                  PROVIDER_HEALTHY=$(kubectl get provider provider-aws \
                    -o jsonpath='{.status.conditions[?(@.type=="Healthy")].status}')
                  if [ "$PROVIDER_HEALTHY" != "True" ]; then
                    echo "FAIL: Provider AWS is not healthy"
                    exit 1
                  fi
                  # Check no managed resources are stuck in error state
                  ERRORS=$(kubectl get managed \
                    -l crossplane.io/provider=provider-aws \
                    --all-namespaces \
                    -o jsonpath='{range .items[*]}{.status.conditions[?(@.type=="Synced")].status}{"\n"}{end}' \
                    2>/dev/null | grep -c "False" || true)
                  if [ "$ERRORS" -gt 5 ]; then
                    echo "FAIL: $ERRORS managed resources not synced"
                    exit 1
                  fi
                  wget -qO- "$HEARTBEAT_URL"
                  echo "Reconciliation heartbeat sent"

Create the heartbeat monitor in Vigilmon:

1. Go to Add Monitor → Heartbeat
2. Name: provider-aws reconciliation
3. Expected interval: 15 minutes
4. Grace period: 5 minutes
5. Store the heartbeat URL in a Kubernetes Secret:

kubectl create secret generic vigilmon-secrets \
  --from-literal=provider-aws-reconcile-heartbeat-url='https://vigilmon.online/heartbeat/YOUR_TOKEN' \
  --from-literal=provider-aws-creds-heartbeat-url='https://vigilmon.online/heartbeat/YOUR_CREDS_TOKEN' \
  -n crossplane-system

Step 5: Configure Alert Channels

Route Provider AWS alerts to your platform team:

1. In Vigilmon: Alert Channels → Add Channel → Slack Webhook
2. Paste your #platform-alerts Slack webhook URL
3. Assign the channel to all Provider AWS monitors

For credential expiry (which blocks all AWS provisioning), add an email escalation:

1. Alert Channels → Add Channel → Email
2. Add your platform team's email address
3. Assign it to the provider-aws credentials heartbeat monitor only

This gives you immediate Slack notification for pod health issues and email escalation when credentials expire — the failure mode most likely to be overlooked because the pod stays green.

What You're Now Monitoring

| Component | Monitor Type | What It Detects | |---|---|---| | Provider AWS pod | HTTP /healthz | Pod crash or unhealthy controller | | AWS credentials | Heartbeat (10 min) | Expired or revoked AWS keys | | Reconciliation loop | Heartbeat (15 min) | Provider stalled, managed resources not syncing |

With these three monitors in place, you'll catch Provider AWS failures within minutes rather than discovering them when a developer reports their infrastructure claim has been Pending for an hour.

Conclusion

Crossplane Provider AWS is a critical dependency for any team using Kubernetes-native infrastructure management. Its failure modes are particularly dangerous because they're silent — the pod appears healthy while AWS provisioning is completely broken. External monitoring with Vigilmon closes this gap, giving your platform team early warning before provisioning delays cascade into blocked deployments.

Get started free at vigilmon.online. Adding Provider AWS health monitoring takes under three minutes, and the credential heartbeat pattern described here works equally well for provider-gcp, provider-azure, and any other Crossplane provider.