tutorial

Verify API server health for each peered cluster

--- title: "Monitoring Liqo Multi-Cluster Resource Sharing with Vigilmon: Peering Health, API Server Checks & Cross-Cluster Connectivity" published: false d...

title: "Monitoring Liqo Multi-Cluster Resource Sharing with Vigilmon: Peering Health, API Server Checks & Cross-Cluster Connectivity" published: false description: How to monitor Liqo multi-cluster Kubernetes resource sharing with Vigilmon -- peer cluster API health, virtual node availability, namespace offloading status, and cross-cluster network connectivity. tags: liqo, kubernetes, multi-cluster, networking, monitoring, devops, cloud-native cover_image:

Liqo enables Kubernetes clusters to share compute resources and workloads across organizational and geographic boundaries -- extending your scheduler to offload pods to peer clusters as naturally as scheduling to local nodes. When the Liqo control plane or its peering sessions degrade, workloads silently lose access to remote capacity, cross-cluster services become unreachable, and namespace offloading fails without surfacing errors to the original scheduler. Vigilmon gives you external visibility into the health of every component Liqo depends on: your local API server, peered cluster API endpoints, virtual node availability, and the cross-cluster network fabric.

What You'll Build

  • HTTP monitors on each peered cluster's API server for federation health
  • A check on Liqo's gateway service for cross-cluster network tunnel availability
  • TCP monitors for WireGuard tunnel port reachability between clusters
  • SSL certificate monitoring for all cluster API server endpoints
  • An alerting table mapping each Liqo failure mode to the right remediation action

Prerequisites

  • Liqo installed on at least two Kubernetes clusters with an active peering session
  • kubectl access to your home cluster
  • API server endpoints accessible over HTTPS (e.g., https://api.cluster1.example.com:6443)
  • A free account at vigilmon.online

Step 1: Understand Liqo's Critical Infrastructure Components

A Liqo multi-cluster deployment has several layers that must stay healthy for resource sharing to work:

Home Cluster                         Remote Cluster
+------------------+                 +------------------+
| liqo-controller  |<-- peering ---->| liqo-controller  |
| liqo-gateway     |<-- WireGuard -->| liqo-gateway     |
| virtual-kubelet  |<-- API calls -->| kube-apiserver   |
| kube-apiserver   |                 |                  |
+------------------+                 +------------------+

Monitoring targets:

  • Cluster API servers: Both local and peered cluster API endpoints must be reachable for Liqo's virtual-kubelet to schedule pods on remote nodes
  • Liqo gateway: The WireGuard-based network gateway that tunnels pod traffic between clusters
  • Peering health: The network fabric enabling cross-cluster service discovery and communication
  • SSL certificates: Each cluster API server's TLS certificate affects the virtual-kubelet's ability to authenticate with remote clusters

Step 2: Monitor Your Cluster API Server Endpoints

Liqo's virtual-kubelet authenticates against each peered cluster's API server to schedule pods and retrieve status. If a peered cluster's API server is unreachable, virtual nodes representing that cluster enter NotReady state and pending workloads are not offloaded:

curl -k https://api.cluster1.example.com:6443/healthz
# Returns: ok

curl -k https://api.cluster2.example.com:6443/readyz
# Returns: ok

For each peered cluster's API server:

  1. Log in to Vigilmon -> Add Monitor -> HTTP.
  2. URL: https://api.cluster1.example.com:6443/healthz.
  3. Check interval: 60 seconds.
  4. Response timeout: 15 seconds.
  5. Expected status: 200.
  6. Keyword: ok.
  7. Label: Liqo peer: cluster1 API server.
  8. Click Save.

Repeat for each peered cluster, using descriptive labels like Liqo peer: cluster2 API server.

Why this matters for Liqo: The virtual-kubelet must be able to authenticate to remote API servers to:

  • Create and delete pods on remote nodes
  • Watch pod status and propagate it back to the home cluster
  • Sync ConfigMaps and Secrets needed by offloaded workloads

When the remote API server is unreachable, offloaded pods enter Pending state on the remote virtual node without surfacing an obvious error to the home cluster operators.

Step 3: Monitor the Liqo Gateway Service

Liqo establishes WireGuard tunnels between clusters via its gateway component. The gateway service peers clusters to exchange data plane traffic once control-plane peering is established.

Check that your Liqo gateway service is reachable from outside your cluster:

# Get the gateway external IP or LoadBalancer address
kubectl get svc -n liqo liqo-gateway -o jsonpath='{.status.loadBalancer.ingress[0].ip}'

# Verify the gateway HTTPS management endpoint
curl https://<GATEWAY_IP>:443/healthz

If your gateway exposes an HTTP health endpoint:

  1. Add Monitor -> HTTP.
  2. URL: https://<liqo-gateway-external-ip>/healthz.
  3. Check interval: 2 minutes.
  4. Response timeout: 10 seconds.
  5. Expected status: 200.
  6. Label: Liqo gateway health.
  7. Click Save.

If your gateway does not expose an HTTP health endpoint, use a TCP monitor instead (see Step 4).

Step 4: Add TCP Monitors for WireGuard Tunnel Ports

Liqo uses WireGuard for cross-cluster network tunnels. WireGuard operates over UDP, but you can monitor the gateway service port to confirm tunnel endpoints are reachable at the network level:

# Check gateway port reachability
nc -zv <gateway-external-ip> 51820
  1. Add Monitor -> TCP.
  2. Host: <liqo-gateway-external-ip>.
  3. Port: 443 (or 51820 if the gateway uses a custom port).
  4. Check interval: 60 seconds.
  5. Label: Liqo gateway TCP.
  6. Click Save.

Add a TCP monitor for each cluster gateway endpoint. When a gateway TCP monitor fires but the cluster API server monitor is green, the data plane tunnel has failed while the control plane remains functional -- a common failure mode during node IP changes or cloud load balancer reconfiguration.

Step 5: Monitor the Local Cluster API Server

In addition to peered clusters, monitor your home cluster API server. Liqo's own control plane components depend on the local API server for CRD access, peering state, and virtual node reconciliation:

curl -k https://api.home-cluster.example.com:6443/healthz
# Returns: ok

curl -k https://api.home-cluster.example.com:6443/readyz
# Returns: ok
  1. Add Monitor -> HTTP.
  2. URL: https://api.home-cluster.example.com:6443/healthz.
  3. Check interval: 60 seconds.
  4. Response timeout: 15 seconds.
  5. Expected status: 200.
  6. Keyword: ok.
  7. Label: Home cluster API server.
  8. Click Save.

Step 6: Monitor SSL Certificates for All API Servers

Each cluster API server TLS certificate is used by the virtual-kubelet when authenticating to remote clusters. An expired or invalid certificate causes authentication failures that surface as virtual node NotReady conditions:

# Check API server certificate expiry
openssl s_client -connect api.cluster1.example.com:6443 2>/dev/null | \
  openssl x509 -noout -dates

# Check all peered cluster certs
for cluster in cluster1 cluster2 cluster3; do
  echo "=== $cluster ==="
  openssl s_client -connect api.${cluster}.example.com:6443 2>/dev/null | \
    openssl x509 -noout -enddate
done

For each cluster API server:

  1. Add Monitor -> SSL Certificate.
  2. Domain: api.cluster1.example.com.
  3. Port: 6443.
  4. Alert when expiry is within: 30 days.
  5. Alert again: 14 days, 7 days, 3 days.
  6. Click Save.

Kubernetes API server cert rotation: Many managed Kubernetes providers (EKS, GKE, AKS) auto-rotate API server certificates. However, self-hosted clusters using kubeadm require manual certificate renewal with kubeadm certs renew. A 30-day alert window ensures you have time to schedule maintenance.

Step 7: Configure Alerting

In Vigilmon under Settings -> Notifications, configure your alert channels:

| Monitor | Trigger | Remediation action | |---|---|---| | Peered cluster API server | Non-200 or timeout | Check remote cluster health; liqoctl status peer; verify network path | | Liqo gateway health | Non-200 response | Check kubectl get pods -n liqo; restart gateway if crashed | | Gateway TCP | Connection refused or timeout | Cloud load balancer or firewall change; verify external IP routing | | Home cluster API server | Non-200 or timeout | Local cluster emergency; check control plane nodes | | SSL certificates | < 30 days to expiry | Renew API server cert; verify virtual-kubelet kubeconfig after renewal |

Alert sensitivity:

  • API server monitors: 1 consecutive failure -- API server outages directly impact workload scheduling
  • Gateway TCP: 2 consecutive failures -- brief load balancer health check transitions can cause transient failures
  • SSL: 30-day lead time for coordinating certificate rotation across clusters

Verifying Liqo Health After an Alert

When a monitor fires, use liqoctl to correlate external availability with Liqo internal state:

# Check overall peering status
liqoctl status peer

# Check virtual node status for each peered cluster
kubectl get virtualnodes

# Check Liqo pod health in the liqo namespace
kubectl get pods -n liqo

# Check namespace offloading status
kubectl get namespacemaps -A

# Inspect gateway logs for tunnel errors
kubectl logs -n liqo -l app.kubernetes.io/component=gateway --tail=50

Common Liqo Failure Modes and What Vigilmon Catches

| Scenario | Vigilmon monitor | |---|---| | Remote cluster API server unreachable | Peered cluster API monitor fires; virtual nodes go NotReady | | WireGuard tunnel drops (node IP change) | Gateway TCP monitor fires; cross-cluster pod traffic drops | | Liqo gateway pod OOM killed | Gateway health monitor fires; data plane disrupted | | API server certificate expires | SSL monitor fires; virtual-kubelet authentication fails | | Cloud load balancer reconfiguration drops gateway IP | Gateway TCP monitor fires; tunnel reestablishment required | | Namespace offloading fails silently | API server monitor fires; offloaded pods stuck Pending | | Network policy blocks Liqo gateway UDP | Gateway TCP fires; control plane OK but data plane broken | | Peered cluster upgraded, API version changed | API server returns 503 during upgrade window | | DNS misconfiguration for cluster endpoints | All cluster API monitors fire simultaneously | | Resource quota exceeded on remote cluster | Virtual-kubelet pod creation fails; API server still healthy |

Liqo's promise -- seamless workload distribution across clusters -- depends on every peer API server, every WireGuard tunnel, and every TLS certificate staying healthy. Failures in any one layer are often invisible to the home cluster operators until workloads fail to schedule or cross-cluster services become unreachable. Vigilmon gives you external, continuous monitoring of every peered API server, gateway endpoint, and certificate in your Liqo federation, so you catch the outage before your multi-cluster scheduling silently falls back to a single cluster.

Start monitoring your Liqo federation in under 5 minutes -- register free at vigilmon.online.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles