tutorial

How to Monitor Karmada Multi-Cluster Kubernetes with Vigilmon

Karmada lets you manage applications across multiple Kubernetes clusters as if they were one — federated deployments, cross-cluster service discovery, and un...

Karmada lets you manage applications across multiple Kubernetes clusters as if they were one — federated deployments, cross-cluster service discovery, and unified policy enforcement. But this power amplifies the consequences of failures: a control plane issue can silently break workload propagation across every member cluster, a policy change might leave some clusters out of sync, and a network partition between clusters can cause the federation layer to show healthy while applications are actually degraded.

In this tutorial you'll set up external monitoring for your Karmada federation using Vigilmon, covering control plane health, member cluster connectivity, and cross-cluster workload availability.

Why Karmada deployments need external monitoring

Karmada introduces a federation control plane layer above your individual clusters. Failures can happen at the Karmada level, the member cluster level, or both:

  • Karmada API server down — workload propagation stops; existing applications keep running on member clusters, but no new deployments or updates can be scheduled; kubectl commands time out silently
  • Member cluster disconnected — Karmada shows the cluster as NotReady; workloads targeted to that cluster don't get updates; failover to healthy clusters depends on your PropagationPolicy configuration
  • Policy propagation failure — a ClusterPropagationPolicy silently fails to apply; some clusters run version N while others are stuck on version N-1; no alarm fires from inside the cluster
  • Cross-cluster service unreachable — Karmada's multi-cluster service federation is in place but a DNS or network policy change breaks service discovery between clusters; application-layer errors occur while all pods show Running
  • Override policy conflict — a ClusterOverridePolicy creates a conflict; the Karmada scheduler logs warnings but no user-facing alert fires
  • Etcd health degradation — the Karmada control plane's etcd cluster loses quorum; the API server becomes read-only; workload changes queue up silently

External monitoring from outside the Karmada control plane is the only way to confirm that workloads are actually reachable and that the federation layer is functioning end-to-end.

What you'll need

  • A running Karmada control plane
  • At least two member clusters registered with Karmada
  • A federated workload with an HTTP endpoint accessible from outside the clusters
  • A free Vigilmon account (sign up takes 30 seconds)

Step 1: Expose the Karmada API server health endpoint

The Karmada API server exposes /healthz and /readyz endpoints. These confirm the control plane is running and connected to etcd:

# Get your Karmada API server address
kubectl config view --kubeconfig ~/.kube/karmada.config \
  -o jsonpath='{.clusters[0].cluster.server}'
# https://karmada-api.yourdomain.com:6443

# Test health (requires valid client cert or token)
curl -k https://karmada-api.yourdomain.com:6443/healthz
# ok

curl -k https://karmada-api.yourdomain.com:6443/readyz
# ok

For monitoring, expose a simplified health proxy that doesn't require client certificate auth. Add a small health proxy deployment to the Karmada host cluster:

# karmada-health-proxy.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: karmada-health-proxy
  namespace: karmada-system
spec:
  replicas: 1
  selector:
    matchLabels:
      app: karmada-health-proxy
  template:
    metadata:
      labels:
        app: karmada-health-proxy
    spec:
      serviceAccountName: karmada-health-proxy
      containers:
      - name: proxy
        image: nginx:alpine
        ports:
        - containerPort: 8080
        volumeMounts:
        - name: nginx-config
          mountPath: /etc/nginx/conf.d
      volumes:
      - name: nginx-config
        configMap:
          name: karmada-health-proxy-config
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: karmada-health-proxy-config
  namespace: karmada-system
data:
  default.conf: |
    server {
        listen 8080;
        location /health {
            return 200 '{"status":"ok","component":"karmada-control-plane"}';
            add_header Content-Type application/json;
        }
    }
---
apiVersion: v1
kind: Service
metadata:
  name: karmada-health-proxy
  namespace: karmada-system
spec:
  type: LoadBalancer
  selector:
    app: karmada-health-proxy
  ports:
  - port: 80
    targetPort: 8080

For a more robust health signal, use a small Go sidecar that actually queries the Karmada API:

// health-proxy/main.go
package main

import (
    "encoding/json"
    "net/http"
    "os/exec"
)

func healthHandler(w http.ResponseWriter, r *http.Request) {
    // Check karmada API server connectivity
    cmd := exec.Command("kubectl", "--kubeconfig", "/etc/karmada/kubeconfig",
        "get", "--raw", "/healthz")
    if err := cmd.Run(); err != nil {
        w.WriteHeader(http.StatusServiceUnavailable)
        json.NewEncoder(w).Encode(map[string]string{
            "status": "degraded",
            "error":  "karmada API server unreachable",
        })
        return
    }

    w.WriteHeader(http.StatusOK)
    json.NewEncoder(w).Encode(map[string]string{
        "status": "ok",
    })
}

func main() {
    http.HandleFunc("/health", healthHandler)
    http.ListenAndServe(":8080", nil)
}

Step 2: Monitor member cluster connectivity

Karmada tracks member cluster health via its own controller. Expose cluster status via a small API endpoint in your ops tooling, or use the Karmada API server directly through a proxy:

# Check member cluster status via Karmada kubectl plugin
kubectl karmada get clusters
# NAME        VERSION   MODE   READY   AGE
# cluster-eu  v1.28.0   Push   True    45d
# cluster-us  v1.28.0   Push   True    45d
# cluster-ap  v1.28.0   Push   False   45d  ← problem!

Create a /clusters/health endpoint in your health proxy that parses this and returns a simple status:

# cluster_health_api.py (FastAPI)
from fastapi import FastAPI
import subprocess
import json

app = FastAPI()

@app.get("/clusters/health")
async def cluster_health():
    result = subprocess.run(
        ["kubectl", "karmada", "get", "clusters", "-o", "json",
         "--kubeconfig", "/etc/karmada/kubeconfig"],
        capture_output=True, text=True
    )
    clusters = json.loads(result.stdout)
    
    not_ready = [
        c["metadata"]["name"]
        for c in clusters["items"]
        if not any(
            cond["type"] == "Ready" and cond["status"] == "True"
            for cond in c.get("status", {}).get("conditions", [])
        )
    ]
    
    if not_ready:
        return {"status": "degraded", "not_ready_clusters": not_ready}, 503
    
    return {
        "status": "ok",
        "cluster_count": len(clusters["items"]),
        "all_ready": True
    }

This gives Vigilmon a single endpoint that returns 200 when all member clusters are connected, and 503 when any cluster is NotReady.

Step 3: Monitor federated workload endpoints

The most important thing to monitor is whether your actual workloads are reachable via their federation-managed endpoints. For a federated Nginx deployment:

# federated-nginx.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-federation
  namespace: default
spec:
  replicas: 2
  selector:
    matchLabels:
      app: nginx-federation
  template:
    metadata:
      labels:
        app: nginx-federation
    spec:
      containers:
      - name: nginx
        image: nginx:alpine
        ports:
        - containerPort: 80
---
apiVersion: policy.karmada.io/v1alpha1
kind: PropagationPolicy
metadata:
  name: nginx-propagation
  namespace: default
spec:
  resourceSelectors:
  - apiVersion: apps/v1
    kind: Deployment
    name: nginx-federation
  placement:
    clusterAffinity:
      clusterNames:
      - cluster-eu
      - cluster-us
      - cluster-ap
    replicaScheduling:
      replicaSchedulingType: Divided
      replicaDivisionPreference: Weighted
      weightPreference:
        staticClusterWeight:
        - targetCluster:
            clusterNames: [cluster-eu]
          weight: 40
        - targetCluster:
            clusterNames: [cluster-us]
          weight: 40
        - targetCluster:
            clusterNames: [cluster-ap]
          weight: 20

After applying, get the external IPs for each cluster's service:

# On each member cluster
kubectl get svc nginx-federation --context=cluster-eu
# NAME               TYPE           CLUSTER-IP      EXTERNAL-IP     PORT(S)
# nginx-federation   LoadBalancer   10.96.50.10     34.65.12.100    80:31234/TCP

kubectl get svc nginx-federation --context=cluster-us
# nginx-federation   LoadBalancer   10.96.50.11     52.10.45.200    80:31234/TCP

kubectl get svc nginx-federation --context=cluster-ap
# nginx-federation   LoadBalancer   10.96.50.12     13.229.80.55    80:31234/TCP

Step 4: Set up multi-cluster monitoring in Vigilmon

Add a monitor for each cluster endpoint independently. This gives you per-cluster visibility — if cluster-ap loses connectivity to the federation, the EU and US monitors stay green while the AP monitor goes red.

Monitor 1: EU cluster workload

  1. Log in to vigilmon.online → Monitors → New Monitor
  2. Type: HTTP / HTTPS
  3. URL: http://34.65.12.100/ (or your DNS name)
  4. Check interval: 1 minute
  5. Expected response: status code 200
  6. Name: Nginx Federation — EU
  7. Save

Monitor 2: US cluster workload

Repeat with http://52.10.45.200/, name: Nginx Federation — US

Monitor 3: AP cluster workload

Repeat with http://13.229.80.55/, name: Nginx Federation — AP

Monitor 4: Karmada control plane health

  1. URL: https://karmada-health.yourdomain.com/health
  2. Check interval: 2 minutes
  3. Expected: 200, body contains "status":"ok"
  4. Name: Karmada Control Plane

Monitor 5: Member cluster connectivity

  1. URL: https://karmada-health.yourdomain.com/clusters/health
  2. Check interval: 2 minutes
  3. Expected: 200, body contains "status":"ok"
  4. Name: Karmada Member Clusters

Step 5: Configure cross-cluster alert grouping

With monitors for 3+ clusters, you want to distinguish "one cluster is down" from "the whole federation is down." Use Vigilmon's monitor grouping and status page to create this picture:

  1. Status Pages → New Status Page
  2. Add all 5 monitors
  3. Group them into sections:
    • Control Plane: Karmada Health, Member Clusters
    • Federated Workloads: EU, US, AP
  4. Name the page "Karmada Federation Status"

When cluster-ap goes down, the status page shows a partial outage — EU and US remain green — giving on-call engineers an immediate picture of blast radius.

Step 6: Alert routing for multi-cluster incidents

Single cluster failure (partial outage):

  • Alert: Slack notification immediately
  • Do not page on-call unless 2+ clusters fail simultaneously

Control plane failure (federation-wide):

  • Alert: PagerDuty immediately (no delay)
  • Impact: no new workload changes can propagate

All workload monitors failing:

  • Alert: PagerDuty immediately with high urgency
  • Impact: complete service unavailability

Configure this in Vigilmon:

  1. Create a Slack channel for Karmada alerts
  2. Assign all 5 monitors to this Slack channel
  3. Create a PagerDuty channel with a 5-minute delay for single-cluster failures
  4. Create a separate PagerDuty channel with 0-minute delay for the Karmada control plane monitor

Step 7: Policy propagation smoke test

To catch silent policy propagation failures, add a monitor that probes your workloads through a consistent API that all clusters should serve:

# Canary endpoint that returns cluster identity
curl http://federation.yourdomain.com/api/cluster-info
# {"cluster": "cluster-eu", "version": "v1.4.2", "region": "europe"}

Monitor this endpoint from Vigilmon and set a response body contains check for the current expected version string (e.g., "version":"v1.4.2"). If a cluster fails to receive the latest deployment, it will still serve the old version, and this check will fail — alerting you to a propagation lag before it becomes a larger problem.

Karmada monitoring coverage summary

| Failure scenario | Vigilmon monitor | Detection method | |---|---|---| | Karmada API server down | HTTP /health | Non-200 or connection error | | Member cluster NotReady | HTTP /clusters/health | Returns 503 | | Workload unreachable in EU | HTTP EU endpoint | Connection error or non-200 | | Workload unreachable in US | HTTP US endpoint | Connection error or non-200 | | Workload unreachable in AP | HTTP AP endpoint | Connection error or non-200 | | Policy propagation failure | Version canary endpoint | Old version in response body | | Partial vs full outage | Status page grouping | Visual cluster-by-cluster view |

Conclusion

Karmada's multi-cluster federation adds a powerful management layer over Kubernetes, but it also adds a new class of failure modes: control plane issues, cluster disconnections, and policy propagation failures that internal probes can't see. With Vigilmon you get:

  • Per-cluster workload monitoring with independent monitors for each member cluster, giving precise blast-radius visibility
  • Control plane health checks that confirm the Karmada API server and member cluster connectivity
  • Policy propagation smoke tests that catch version drift across clusters
  • Multi-cluster status pages that communicate federation health to engineering and operations teams

Sign up for a free Vigilmon account and have your Karmada federation monitored in under 15 minutes.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles