Monitoring Kosmos with Vigilmon: Control Plane Health, Leaf Cluster Connectivity, Cross-Cluster Network Tunnels & Scheduling Heartbeats

Kosmos is a Kubernetes multi-cluster orchestration engine that virtualizes multiple independent clusters into a single unified cluster. The Kosmos control plane runs on a host cluster and manages a tree of leaf clusters through its kosmos-controller-manager and kosmos-scheduler. Leaf clusters join via Clusterlink, an overlay network that stitches cross-cluster pod networking together transparently. When the Kosmos API is unreachable, cross-cluster pod scheduling stops and workloads that need to overflow to leaf clusters queue indefinitely. When a Clusterlink tunnel degrades, cross-cluster service calls fail silently with connection timeouts while both pods show Running on their respective clusters. Vigilmon gives you external visibility into the Kosmos control plane API, leaf cluster tree controller status, Clusterlink network tunnels, SSL certificates, and cross-cluster scheduling heartbeats so you catch multi-cluster failures before they stall workloads or break network connectivity between clusters.

What You'll Build

• A monitor on the Kosmos control plane health endpoint
• A Kosmos API group check for multi-cluster CRDs
• A leaf cluster tree controller status check
• SSL certificate monitoring for the Kosmos control plane
• A heartbeat monitor confirming cross-cluster scheduling pipelines complete on schedule

Prerequisites

• A running Kosmos installation with the control plane exposed at a URL
• Kosmos API accessible over HTTPS (via an Ingress or LoadBalancer)
• A free account at vigilmon.online

Step 1: Verify the Kosmos Control Plane Health Endpoint

The Kosmos control plane extends the Kubernetes API and exposes the standard health endpoint:

curl https://kosmos.example.com/healthz

A healthy Kosmos control plane returns:

ok

This confirms the Kosmos API server is alive and the core controllers — kosmos-controller-manager (which manages the cluster tree and Clusterlink tunnels) and kosmos-scheduler (which places pods across leaf clusters) — can receive reconciliation work.

Step 2: Create a Vigilmon HTTP Monitor for the Kosmos Control Plane

1. Log in to Vigilmon → Add Monitor → HTTP.
2. URL: https://kosmos.example.com/healthz.
3. Check interval: 60 seconds.
4. Response timeout: 10 seconds.
5. Expected status: 200.
6. Keyword: ok.
7. Label: Kosmos Control Plane Health.
8. Click Save.

Set alerts to trigger after 1 consecutive failure — a Kosmos control plane outage blocks all cross-cluster pod scheduling and prevents the Clusterlink controller from reconciling network tunnels across the cluster tree.

Step 3: Monitor the Kosmos API Group

Kosmos registers its custom resource group on the host cluster. Confirming this group is available verifies that CRD installation is intact and the multi-cluster tree API is functional:

curl https://kosmos.example.com/apis/kosmos.io/v1alpha1 \
  -H "Authorization: Bearer your-token"

A healthy response lists resources including Cluster and ClusterNode. Add a monitor:

1. Add Monitor → HTTP.
2. URL: https://kosmos.example.com/apis/kosmos.io/v1alpha1.
3. Check interval: 2 minutes.
4. Expected status: 200.
5. Keyword: ClusterNode.
6. HTTP Headers: Authorization: Bearer your-token.
7. Label: Kosmos API Group.
8. Click Save.

Authentication: Use kubectl create token on the host cluster's kosmos-system namespace service account to generate a bearer token.

Step 4: Monitor the Leaf Cluster Tree Registry

The Cluster list endpoint confirms the Kosmos tree controller is tracking leaf clusters and the host cluster etcd backend is healthy. A degraded tree registry means newly joined leaf clusters cannot receive workloads:

curl https://kosmos.example.com/apis/kosmos.io/v1alpha1/clusters \
  -H "Authorization: Bearer your-token"

A healthy response includes:

{"kind":"ClusterList","apiVersion":"kosmos.io/v1alpha1","items":[...]}

Add a monitor:

1. Add Monitor → HTTP.
2. URL: https://kosmos.example.com/apis/kosmos.io/v1alpha1/clusters.
3. Check interval: 2 minutes.
4. Expected status: 200.
5. Keyword: ClusterList.
6. Label: Kosmos Leaf Cluster Registry.
7. Click Save.

Step 5: Monitor Clusterlink Network Tunnel Endpoints

Kosmos Clusterlink establishes overlay network tunnels between the host cluster and each leaf cluster. A TCP connectivity check on the gateway endpoint confirms the tunnel is alive at the network layer — catching failures that don't surface through the Kubernetes API:

# Identify gateway endpoints
kubectl get gateway -n kosmos-system

Add a TCP monitor for each leaf cluster gateway:

1. Add Monitor → TCP Port.
2. Host: The gateway's external IP (e.g., 203.0.113.10).
3. Port: 4500 (or your configured Clusterlink gateway port).
4. Check interval: 60 seconds.
5. Label: Kosmos Clusterlink Gateway leaf-01.
6. Click Save.

Repeat for each leaf cluster gateway. A TCP failure while the Kosmos API is healthy indicates a network tunnel failure below the Kubernetes API layer — the class of failure that leaves pods Running on both sides while cross-cluster service calls time out.

Step 6: Set Up Cross-Cluster Scheduling Heartbeat Monitoring

Cross-cluster scheduling pipelines place workloads onto leaf clusters based on capacity and placement policies. A scheduler crash or leaf cluster disconnection leaves new pods in Pending indefinitely with no obvious error. Use Vigilmon heartbeat monitoring to confirm scheduling pipelines complete on schedule:

Step 6a — Create a Heartbeat monitor in Vigilmon:

1. Add Monitor → Heartbeat.
2. Name: Kosmos cross-cluster scheduling pipeline.
3. Expected interval: 15 minutes (or match your workload deployment cadence).
4. Grace period: 5 minutes.
5. Copy the generated ping URL (e.g., https://vigilmon.online/ping/abc123).

Step 6b — Add a ping step to your scheduling validation pipeline:

#!/bin/bash
# Kosmos cross-cluster scheduling smoke test

# Deploy a test pod targeting a leaf cluster
kubectl --context host-cluster apply -f - <<EOF
apiVersion: v1
kind: Pod
metadata:
  name: kosmos-scheduling-probe
  namespace: kosmos-system
  labels:
    kosmos-probe: "true"
spec:
  schedulerName: kosmos-scheduler
  nodeName: ""
  containers:
    - name: probe
      image: busybox:latest
      command: ["sh", "-c", "sleep 30"]
EOF

# Wait for pod to be scheduled onto a leaf cluster node
kubectl --context host-cluster wait pod/kosmos-scheduling-probe \
  -n kosmos-system --for=condition=Ready --timeout=120s

# Clean up probe pod
kubectl --context host-cluster delete pod/kosmos-scheduling-probe -n kosmos-system

# Notify Vigilmon on success
curl -fsS -m 10 https://vigilmon.online/ping/abc123

If the scheduling probe fails or times out before reaching the ping step, Vigilmon fires an alert after the grace period, surfacing Kosmos scheduler failures or leaf cluster disconnections.

Step 7: Monitor SSL Certificates

The Kosmos control plane API is consumed by kosmos-agent processes on every leaf cluster. An expired certificate breaks leaf cluster reconnections and prevents new Clusterlink tunnel establishment:

1. Add Monitor → SSL Certificate.
2. Domain: kosmos.example.com.
3. Alert when expiry is within: 30 days.
4. Alert again at: 14 days, 7 days, 3 days, 1 day.
5. Click Save.

Step 8: Configure Alerting

In Vigilmon under Settings → Notifications, configure your alert channels:

| Monitor | Trigger | Action | |---|---|---| | Control plane health | Non-200 or ok missing | Check kosmos-controller-manager pod; kubectl get pods -n kosmos-system | | Kosmos API group | Non-200 or ClusterNode absent | CRD missing; check kosmos-controller-manager logs | | Leaf cluster registry | Non-200 or ClusterList absent | Tree controller down; no new leaf clusters can join | | Clusterlink TCP gateway | Connection refused | Network tunnel endpoint unreachable; cross-cluster traffic broken | | Scheduling heartbeat | No ping within window | Kosmos scheduler stalled or leaf cluster disconnected | | SSL certificate | < 30 days | Renew; leaf cluster agent reconnections will fail |

Common Kosmos Failure Modes and What Vigilmon Catches

| Scenario | Vigilmon monitor | |---|---| | Kosmos control plane pod crash | Health endpoint unreachable; alert within 60 s | | CRD migration failure after upgrade | API group returns 404; alert immediately | | Tree controller OOM killed | Leaf cluster registry returns 503; alert fires | | Clusterlink tunnel dropped on leaf cluster restart | TCP gateway monitor fires; alert before app timeouts | | Cross-cluster service call timeouts | TCP gateway down; alert precedes app-level errors | | Kosmos scheduler crash | Scheduling heartbeat times out; alert after grace period | | SSL certificate expiry | SSL monitor fires at 30 days; leaf agent reconnects fail | | Host cluster etcd pressure | All API monitors return 500; alert fires fleet-wide |

Kosmos's multi-cluster abstraction operates at both the scheduling and networking layers simultaneously — which means its failure modes span the Kubernetes control plane, the Clusterlink overlay network, and the leaf cluster agents. A silent Clusterlink tunnel failure doesn't crash pods; it makes cross-cluster service calls time out while both sides show Running. A stalled Kosmos scheduler doesn't error; it leaves pods in Pending with no owner. Vigilmon's layered monitoring of the control plane health, API group availability, leaf cluster registry, Clusterlink TCP tunnel endpoints, scheduling heartbeats, and SSL certificates gives you external visibility across every layer of the Kosmos multi-cluster stack, so you catch failures before they propagate to application workloads or leave clusters operating in isolation.

Start monitoring Kosmos in under 5 minutes — register free at vigilmon.online.