Monitoring KubeAdmiral with Vigilmon: Keeping Your Multi-Cluster Federation Controller Healthy

KubeAdmiral is a Kubernetes multi-cluster federation controller that propagates resources across cluster fleets using fine-grained scheduling and propagation policies. Platform teams use KubeAdmiral to declare workloads once in a hub cluster and have them distributed, replicated, and overridden across member clusters according to weight-based or topology-aware policies. When the KubeAdmiral controller goes down, propagation to member clusters stops — workloads in hub clusters sit undeployed, and policy-driven scaling events never reach member clusters. When a member cluster loses its sync connection, that cluster drifts from the desired state silently while the hub controller continues to report propagation as successful. Vigilmon gives you external monitoring that validates the KubeAdmiral control plane and confirms that resources are actually propagating to member clusters.

What You'll Build

• Vigilmon HTTP monitors for the KubeAdmiral API server and webhook
• Heartbeat monitors validating the propagation pipeline for each member cluster
• A policy execution probe to confirm scheduling decisions are taking effect
• Alert channels for your platform team

Prerequisites

• KubeAdmiral deployed in your hub Kubernetes cluster
• At least one member cluster joined via FederatedCluster
• A free account at vigilmon.online

Why Monitoring KubeAdmiral Matters

KubeAdmiral is the nervous system connecting your hub to your cluster fleet. Its failure modes are dangerous precisely because the hub cluster continues to look healthy while member clusters drift:

Controller manager unavailability silently stops all resource propagation. Federated deployments that should propagate to member clusters remain in the hub's desired state but never reach the member clusters. Existing workloads on member clusters continue running from previous reconciliations — giving a false impression that propagation is working — while new deployments or scaling events never land.

Propagation policy divergence is a subtle failure that occurs when a PropagationPolicy or ClusterPropagationPolicy changes but the controller fails mid-reconciliation. Some member clusters receive the updated resource; others retain the previous version. From the hub's perspective, reconciliation is "complete." From an operator's perspective, the fleet has split-brain state that may not be detected until a production incident.

Member cluster sync disruption happens when a member cluster's API server is briefly unavailable. KubeAdmiral marks the FederatedCluster as unreachable and stops propagating to it — but without external monitoring, this condition persists silently. Member clusters that are "paused" from the federation perspective continue serving stale workloads indefinitely.

Admission webhook latency affects every federated resource creation. KubeAdmiral's webhook validates propagation policies before resources are committed. If the webhook becomes slow or unavailable, federated resource creation blocks cluster-wide — a form of control-plane DoS that is easy to miss until CI pipelines start timing out on deployment operations.

Step 1: Monitor the KubeAdmiral API Server

KubeAdmiral exposes the Kubernetes API for federated resources. Monitor its health endpoint:

# Check the KubeAdmiral hub API server
curl -sk https://kubeadmiral.yourdomain.com/healthz
# Expected: ok

# Check the KubeAdmiral controller manager
kubectl get pods -n kubeadmiral-system -l app=kubeadmiral-controller-manager

In the Vigilmon dashboard:

1. Go to Add Monitor → HTTP
2. Configure:

| Field | Value | |---|---| | Name | KubeAdmiral hub /healthz | | URL | https://kubeadmiral.yourdomain.com/healthz | | Method | GET | | Expected status | 200 | | Response body contains | ok | | Check interval | 1 minute | | Timeout | 10 seconds | | Alert after | 2 consecutive failures |

Step 2: Monitor the KubeAdmiral Admission Webhook

KubeAdmiral installs admission webhooks for PropagationPolicy validation. Webhook latency directly blocks federated resource operations. Add a dedicated probe to the webhook endpoint:

# Identify the KubeAdmiral webhook service
kubectl get validatingwebhookconfigurations | grep kubeadmiral
kubectl get svc -n kubeadmiral-system | grep webhook

In Vigilmon, monitor the webhook's HTTPS endpoint:

| Field | Value | |---|---| | Name | KubeAdmiral webhook /readyz | | URL | https://kubeadmiral-webhook.yourdomain.com/readyz | | Method | GET | | Expected status | 200 | | Check interval | 1 minute | | Timeout | 5 seconds | | Alert after | 2 consecutive failures |

Set a tight timeout on the webhook probe — a slow webhook is nearly as damaging as an unavailable one for control-plane operations.

Step 3: Monitor Member Cluster Connectivity

KubeAdmiral maintains a FederatedCluster status for each member cluster. Add a heartbeat that verifies all member clusters are reachable:

# kubeadmiral-cluster-probe.yaml
apiVersion: batch/v1
kind: CronJob
metadata:
  name: kubeadmiral-cluster-probe
  namespace: kubeadmiral-system
spec:
  schedule: "*/5 * * * *"
  successfulJobsHistoryLimit: 3
  failedJobsHistoryLimit: 3
  jobTemplate:
    spec:
      template:
        spec:
          serviceAccountName: kubeadmiral-monitor
          restartPolicy: OnFailure
          containers:
            - name: cluster-probe
              image: bitnami/kubectl:latest
              env:
                - name: HEARTBEAT_URL
                  valueFrom:
                    secretKeyRef:
                      name: vigilmon-secrets
                      key: kubeadmiral-cluster-heartbeat-url
              command:
                - /bin/sh
                - -c
                - |
                  set -e

                  # Check all FederatedClusters are in Ready state
                  NOT_READY=$(kubectl get federatedcluster -o json 2>/dev/null | \
                    python3 -c "
import sys, json
data = json.load(sys.stdin)
not_ready = []
for item in data.get('items', []):
    name = item['metadata']['name']
    conditions = item.get('status', {}).get('conditions', [])
    ready = any(
        c.get('type') == 'ClusterReady' and c.get('status') == 'True'
        for c in conditions
    )
    if not ready:
        not_ready.append(name)
print('\n'.join(not_ready))
" 2>/dev/null || echo "")

                  if [ -z "$NOT_READY" ]; then
                    echo "All FederatedClusters are Ready"
                    wget -qO- "$HEARTBEAT_URL"
                    echo "Cluster connectivity heartbeat sent"
                  else
                    echo "ERROR: FederatedClusters not Ready:" >&2
                    echo "$NOT_READY" >&2
                    exit 1
                  fi

Create RBAC for the monitor:

# kubeadmiral-monitor-rbac.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: kubeadmiral-monitor
  namespace: kubeadmiral-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: kubeadmiral-monitor
rules:
  - apiGroups: ["core.kubeadmiral.io"]
    resources: ["federatedclusters", "federatedobjects"]
    verbs: ["get", "list"]
  - apiGroups: ["types.kubeadmiral.io"]
    resources: ["federateddeployments", "federatedconfigmaps"]
    verbs: ["get", "list"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: kubeadmiral-monitor
subjects:
  - kind: ServiceAccount
    name: kubeadmiral-monitor
    namespace: kubeadmiral-system
roleRef:
  kind: ClusterRole
  name: kubeadmiral-monitor
  apiGroup: rbac.authorization.k8s.io

Create the heartbeat monitor in Vigilmon:

1. Add Monitor → Heartbeat
2. Name: KubeAdmiral member cluster connectivity
3. Expected interval: 5 minutes
4. Grace period: 5 minutes

Store secrets:

kubectl create secret generic vigilmon-secrets \
  --from-literal=kubeadmiral-cluster-heartbeat-url='https://vigilmon.online/heartbeat/YOUR_CLUSTER_TOKEN' \
  --from-literal=kubeadmiral-propagation-heartbeat-url='https://vigilmon.online/heartbeat/YOUR_PROPAGATION_TOKEN' \
  -n kubeadmiral-system

Step 4: Heartbeat Monitor for Propagation Pipeline

Validate that federated resources are actually propagating to member clusters by verifying propagation status of a known federated object:

# kubeadmiral-propagation-probe.yaml
apiVersion: batch/v1
kind: CronJob
metadata:
  name: kubeadmiral-propagation-probe
  namespace: kubeadmiral-system
spec:
  schedule: "*/10 * * * *"
  jobTemplate:
    spec:
      template:
        spec:
          serviceAccountName: kubeadmiral-monitor
          restartPolicy: OnFailure
          containers:
            - name: propagation-probe
              image: bitnami/kubectl:latest
              env:
                - name: PROBE_NAMESPACE
                  value: "monitoring"
                - name: PROBE_OBJECT
                  value: "platform-probe-configmap"
                - name: HEARTBEAT_URL
                  valueFrom:
                    secretKeyRef:
                      name: vigilmon-secrets
                      key: kubeadmiral-propagation-heartbeat-url
              command:
                - /bin/sh
                - -c
                - |
                  set -e

                  # Check that a known FederatedConfigMap has propagated to all clusters
                  STATUS=$(kubectl get federatedconfigmap "$PROBE_OBJECT" \
                    -n "$PROBE_NAMESPACE" \
                    -o jsonpath='{.status.conditions[?(@.type=="Propagating")].status}' 2>/dev/null || echo "")

                  if [ "$STATUS" = "True" ]; then
                    echo "Propagation pipeline active: $PROBE_OBJECT is propagating"
                    wget -qO- "$HEARTBEAT_URL"
                    echo "Propagation heartbeat sent"
                  else
                    echo "ERROR: Propagation status for $PROBE_OBJECT is '$STATUS'" >&2
                    kubectl get federatedconfigmap "$PROBE_OBJECT" -n "$PROBE_NAMESPACE" -o yaml >&2
                    exit 1
                  fi

Pre-create a lightweight federated probe object to use as the canary:

# propagation-probe-object.yaml
apiVersion: types.kubeadmiral.io/v1alpha1
kind: FederatedConfigMap
metadata:
  name: platform-probe-configmap
  namespace: monitoring
spec:
  template:
    data:
      probe: "vigilmon-propagation-check"
  placement:
    clusterSelector: {}  # propagate to all clusters

Create the heartbeat monitor in Vigilmon:

1. Add Monitor → Heartbeat
2. Name: KubeAdmiral propagation pipeline
3. Expected interval: 10 minutes
4. Grace period: 5 minutes

Step 5: Configure Alert Channels

Route KubeAdmiral alerts to the right teams:

1. In Vigilmon: Alert Channels → Add Channel → Slack Webhook
2. Add your #platform-engineering webhook URL
3. Assign to all KubeAdmiral monitors

For member cluster connectivity failures and propagation pipeline failures (both directly impact deployed workloads):

1. Alert Channels → Add Channel → PagerDuty or email
2. Add your platform on-call rotation
3. Assign to both cluster connectivity and propagation pipeline heartbeats

What You're Now Monitoring

| Component | Monitor Type | What It Detects | |---|---|---| | KubeAdmiral hub /healthz | HTTP GET | Control plane down, API server unreachable | | Admission webhook /readyz | HTTP GET | Webhook latency, federated resource creation blocked | | Member cluster connectivity | Heartbeat (5 min) | FederatedCluster not Ready, member unreachable | | Propagation pipeline | Heartbeat (10 min) | Resource propagation stalled across fleet |

Conclusion

KubeAdmiral's federation model makes its failure modes particularly deceptive: the hub cluster stays healthy while member clusters quietly drift from desired state. A controller crash at 3am leaves your fleet running on stale propagations — new deployments never land, scaling events never fire, and policy changes never apply — yet every kubectl command against the hub returns Running. External monitoring with Vigilmon closes this gap by watching the control plane from outside and validating that propagation is actually reaching member clusters, not just being accepted by the hub.

Get started free at vigilmon.online. The hub health probe, member cluster connectivity heartbeat, and propagation pipeline heartbeat together give you KubeAdmiral's three most critical signals in under fifteen minutes.