tutorial

How to Monitor KubeStell Multi-Cluster Workload Orchestration with Vigilmon (HTTP + TCP + Alerts)

KubeStell (KubeStelllar) is a multi-cluster workload orchestration system that lets you deploy and manage applications across many Kubernetes clusters from a...

KubeStell (KubeStelllar) is a multi-cluster workload orchestration system that lets you deploy and manage applications across many Kubernetes clusters from a single control plane. It uses BindingPolicy resources to declare where workloads should run across your fleet, propagating configuration to spoke clusters and synchronizing status back to the hub. When the KubeStell control plane fails or a spoke cluster loses connectivity, workloads that appear deployed may stop running on one or more clusters — and there is no built-in cross-cluster health alerting.

In this tutorial you'll set up comprehensive uptime monitoring for your KubeStell multi-cluster environment using Vigilmon — free tier, no credit card required.

Why KubeStell deployments need external monitoring

KubeStell orchestrates across cluster boundaries, which creates failure modes that neither the hub nor the spoke can fully detect on its own:

  • KubeStell control plane (kubestellar-controller-manager) crashes — BindingPolicy reconciliation halts; existing workloads continue running on spokes but no updates, new deployments, or removals are propagated
  • Space provider disruption — if the underlying OCM (Open Cluster Management) hub cluster or its API is degraded, KubeStell cannot communicate with spoke clusters; the hub looks healthy but all cross-cluster operations silently fail
  • Spoke cluster unreachable — if a spoke cluster loses connectivity to the hub, KubeStell stops propagating to it; workloads on the spoke keep running on their last state but diverge from Git and receive no updates
  • BindingPolicy selector drift — if cluster labels change or spoke clusters are de-registered, BindingPolicies may match zero clusters; workloads appear configured on the hub but run nowhere
  • Sync object status lag — status sync from spokes to hub may fall behind under load; the hub shows stale workload health that masks real failures on spoke clusters

These failures compound across clusters: a single control plane issue can silently stop all workload propagation across dozens of clusters simultaneously.

What you'll need

  • A KubeStell hub cluster with at least one spoke cluster registered
  • Workloads running on spoke clusters exposed via Service or Ingress
  • A free Vigilmon account (sign up takes 30 seconds)

Step 1: Expose health endpoints for monitoring

Verify the KubeStell control plane and spoke cluster connectivity:

# Check KubeStell control plane on the hub
kubectl get pods -n kubestellar -l app=kubestellar-controller-manager
# NAME                                              READY   STATUS    RESTARTS   AGE
# kubestellar-controller-manager-7f8d9c6b5-xk2mn   1/1     Running   0          3d

# Check registered spoke clusters (ManagedClusters via OCM)
kubectl get managedcluster
# NAME         HUB ACCEPTED   MANAGED CLUSTER URLS       JOINED   AVAILABLE   AGE
# cluster-eu   true           https://eu-cluster:6443    True     True        3d
# cluster-us   true           https://us-cluster:6443    True     True        3d
# cluster-ap   true           https://ap-cluster:6443    True     True        2d

# Check BindingPolicies
kubectl get bindingpolicy -A
# NAMESPACE   NAME             SYNCED   READY   AGE
# default     web-app-policy   True     True    2d

Deploy a health check workload that KubeStell propagates to your spoke clusters:

# kubestellar-health.yaml — deploy this to the KubeStell hub; BindingPolicy distributes it
apiVersion: apps/v1
kind: Deployment
metadata:
  name: ks-health
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      app: ks-health
  template:
    metadata:
      labels:
        app: ks-health
    spec:
      containers:
        - name: health
          image: nginx:alpine
          ports:
            - containerPort: 80
          volumeMounts:
            - name: config
              mountPath: /etc/nginx/conf.d
          readinessProbe:
            httpGet:
              path: /health
              port: 80
            initialDelaySeconds: 5
            periodSeconds: 10
          livenessProbe:
            httpGet:
              path: /health
              port: 80
            initialDelaySeconds: 10
            periodSeconds: 30
      volumes:
        - name: config
          configMap:
            name: ks-health-nginx
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: ks-health-nginx
  namespace: default
data:
  default.conf: |
    server {
      listen 80;
      location /health {
        return 200 '{"status":"ok","orchestrator":"kubestellar"}';
        add_header Content-Type application/json;
      }
    }
---
apiVersion: v1
kind: Service
metadata:
  name: ks-health
  namespace: default
spec:
  type: LoadBalancer
  selector:
    app: ks-health
  ports:
    - port: 80
      targetPort: 80
      protocol: TCP
---
# BindingPolicy to propagate to all spoke clusters
apiVersion: control.kubestellar.io/v1alpha1
kind: BindingPolicy
metadata:
  name: ks-health-policy
  namespace: default
spec:
  clusterSelectors:
    - matchLabels:
        fleet: production
  downsync:
    - apiGroup: apps
      resource: deployments
      namespaces: [default]
      labelSelectors:
        - matchLabels:
            app: ks-health
    - apiGroup: ""
      resource: services
      namespaces: [default]
      labelSelectors:
        - matchLabels:
            app: ks-health
    - apiGroup: ""
      resource: configmaps
      namespaces: [default]
      labelSelectors:
        - matchLabels:
            app: ks-health

kubectl apply -f kubestellar-health.yaml

# Verify propagation to spoke clusters
# (use kubeconfig for each spoke cluster)
kubectl --kubeconfig=cluster-eu.kubeconfig get pods -n default -l app=ks-health
kubectl --kubeconfig=cluster-us.kubeconfig get pods -n default -l app=ks-health

# Get LoadBalancer IPs from each spoke
kubectl --kubeconfig=cluster-eu.kubeconfig get svc ks-health
# NAME         TYPE           EXTERNAL-IP       PORT(S)
# ks-health    LoadBalancer   203.0.113.110     80/TCP

kubectl --kubeconfig=cluster-us.kubeconfig get svc ks-health
# NAME         TYPE           EXTERNAL-IP       PORT(S)
# ks-health    LoadBalancer   203.0.113.120     80/TCP

# Verify each spoke endpoint
curl http://203.0.113.110/health
# {"status":"ok","orchestrator":"kubestellar"}
curl http://203.0.113.120/health
# {"status":"ok","orchestrator":"kubestellar"}

Step 2: Set up HTTP monitoring in Vigilmon

Add monitors for each spoke cluster's health endpoint:

  1. Log in to vigilmon.online and go to Monitors → New Monitor
  2. Choose HTTP / HTTPS as the type
  3. Set the URL to the EU spoke endpoint: http://203.0.113.110/health
  4. Set the check interval to 1 minute
  5. Under Expected response, set:
    • Status code: 200
    • Response body contains: "status":"ok"
  6. Save the monitor
  7. Repeat for the US spoke: http://203.0.113.120/health

Repeat for every spoke cluster in your fleet.

Each per-spoke monitor validates the complete KubeStell propagation path: the hub control plane must be healthy, the BindingPolicy must be matching the cluster, OCM connectivity must be working, the workload must be running on the spoke, and the LoadBalancer must be routing traffic.

Why per-spoke external monitoring is essential

KubeStell's hub cluster can report all BindingPolicies as Synced and Ready even when a spoke cluster has lost network connectivity and is running a stale workload. The hub shows what it last successfully propagated, not the current state of the spoke. Vigilmon probes each spoke independently from outside the cluster fleet, giving you ground truth for every region. Its multi-region consensus model eliminates false positives from transient network issues while catching real per-cluster outages within minutes.

Step 3: Monitor TCP ports and the KubeStell control plane

Per-spoke TCP checks:

  1. Go to Monitors → New Monitor
  2. Choose TCP Port
  3. Enter: Host 203.0.113.110, Port 80 (EU cluster)
  4. Save; repeat for 203.0.113.120:80 (US cluster)

KubeStell control plane health:

# Port-forward to KubeStell controller metrics
kubectl port-forward -n kubestellar deploy/kubestellar-controller-manager 8080:8080

# Check health
curl http://localhost:8080/healthz
# ok

# Check readiness
curl http://localhost:8080/readyz
# ok

Expose via Ingress for direct Vigilmon probing:

# ks-controller-ingress.yaml
apiVersion: v1
kind: Service
metadata:
  name: ks-controller-health
  namespace: kubestellar
spec:
  selector:
    app: kubestellar-controller-manager
  ports:
    - port: 8080
      targetPort: 8080
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ks-controller-health
  namespace: kubestellar
spec:
  rules:
    - host: ks-health.your-hub.com
      http:
        paths:
          - path: /healthz
            pathType: Exact
            backend:
              service:
                name: ks-controller-health
                port:
                  number: 8080

Add an HTTP monitor in Vigilmon for https://ks-health.your-hub.com/healthz.

Spoke connectivity heartbeat:

# ks-spoke-heartbeat-cronjob.yaml
apiVersion: batch/v1
kind: CronJob
metadata:
  name: ks-spoke-heartbeat
  namespace: default
spec:
  schedule: "*/5 * * * *"
  jobTemplate:
    spec:
      template:
        spec:
          restartPolicy: Never
          containers:
            - name: heartbeat
              image: bitnami/kubectl:latest
              command:
                - sh
                - -c
                - |
                  # Check all spoke clusters are available
                  NOT_READY=$(kubectl get managedcluster -o json | \
                    jq -r '.items[] | select(.status.conditions[] | select(.type=="ManagedClusterConditionAvailable" and .status!="True")) | .metadata.name')
                  if [ -n "$NOT_READY" ]; then
                    echo "Spoke clusters not available: $NOT_READY"
                    exit 1
                  fi
                  curl -fsS https://uptime.vigilmon.online/api/v1/heartbeat/YOUR_HEARTBEAT_KEY

Step 4: Configure alert channels

A KubeStell control plane failure affects all clusters simultaneously. A single spoke failure affects all users in that region. Both require immediate response.

Email alerts

  1. In Vigilmon, go to Alert Channels → Add Channel → Email
  2. Enter your platform team's on-call email
  3. Assign the channel to all KubeStell monitors

Webhook alerts (Slack, PagerDuty, etc.)

  1. Go to Alert Channels → Add Channel → Webhook
  2. Enter your webhook URL
  3. Assign the channel to your monitors

Example payload when a spoke health endpoint goes down:

{
  "monitor_name": "KubeStell EU Cluster Health",
  "status": "down",
  "url": "http://203.0.113.110/health",
  "started_at": "2024-01-15T16:20:00Z",
  "duration_seconds": 90
}

Run these diagnostics when an alert fires:

# Check KubeStell hub control plane
kubectl get pods -n kubestellar
kubectl logs -n kubestellar -l app=kubestellar-controller-manager --tail=100

# Check all spoke availability
kubectl get managedcluster
kubectl describe managedcluster cluster-eu

# Check BindingPolicy status
kubectl get bindingpolicy -A
kubectl describe bindingpolicy ks-health-policy -n default

# Check propagation status
kubectl get manifestwork -A | grep ks-health

# Check spoke cluster using its own kubeconfig
kubectl --kubeconfig=cluster-eu.kubeconfig get pods -n default
kubectl --kubeconfig=cluster-eu.kubeconfig get nodes

# Check OCM connectivity
kubectl get managedclusteraddons -A
kubectl get addondeploymentconfigs -A

Step 5: Create a public status page

KubeStell typically manages production workloads across multiple regions. Give your teams and users per-region visibility.

  1. In Vigilmon, go to Status Pages → New Status Page
  2. Give it a name: "Multi-Cluster Fleet Status"
  3. Add your monitors: per-spoke HTTP monitors, TCP checks, Control Plane Health, Spoke Connectivity Heartbeat
  4. Group them by region: EU Cluster, US Cluster, AP Cluster, Control Plane
  5. Publish the page

Share this URL with your engineering teams and any users who need regional availability visibility.

Putting it all together

Here's a summary of the monitors to create for a KubeStell multi-cluster deployment:

| Monitor | Type | What it catches | |---------|------|-----------------| | http://203.0.113.110/health (EU) | HTTP | Full propagation path to EU spoke | | http://203.0.113.120/health (US) | HTTP | Full propagation path to US spoke | | 203.0.113.110:80 (EU) | TCP | EU spoke service reachability | | 203.0.113.120:80 (US) | TCP | US spoke service reachability | | https://ks-health.your-hub.com/healthz | HTTP | KubeStell hub control plane | | Spoke connectivity heartbeat | Heartbeat | All ManagedClusters remain Available |

And the KubeStell diagnostics to run when an alert fires:

# Quick triage: hub or spoke?
kubectl get pods -n kubestellar        # hub control plane
kubectl get managedcluster             # spoke availability

# Which spoke is affected?
kubectl get managedcluster -o custom-columns=NAME:.metadata.name,AVAILABLE:.status.conditions[?(@.type==\"ManagedClusterConditionAvailable\")].status

# Is the workload propagated?
kubectl get manifestwork -A | grep ks-health
kubectl describe manifestwork -n cluster-eu ks-health

# Check BindingPolicy matching
kubectl get bindingpolicy ks-health-policy -n default -o yaml | grep -A10 clusterSelectors

# Force a re-sync
kubectl annotate bindingpolicy ks-health-policy reconcile.kubestellar.io/requestedAt="$(date -u +%Y-%m-%dT%H:%M:%SZ)"

# Check events on the hub for propagation errors
kubectl get events -n kubestellar | grep -i "error\|fail\|warn"

What's next

  • SSL certificate monitoring — add SSL monitors for HTTPS endpoints on each spoke cluster
  • Heartbeat monitoring — use Vigilmon heartbeats to verify that workload propagation to each spoke runs on schedule
  • Per-region status pages — create a Vigilmon status page group per spoke cluster so regional failures are immediately identifiable without noisy cross-cluster dashboards

Get started free at vigilmon.online — no credit card, monitors start running in under a minute.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles