Verify broker cluster API server health

title: "Monitoring Submariner Cross-Cluster Networking with Vigilmon: Gateway Health, Broker API Checks & Tunnel Connectivity" published: false description: How to monitor Submariner cross-cluster Kubernetes networking with Vigilmon -- broker API health, gateway pod availability, cable driver tunnel connectivity, and service discovery endpoint checks. tags: submariner, kubernetes, multi-cluster, networking, monitoring, devops, cloud-native cover_image:

Submariner provides direct L3 network connectivity between pods and services in separate Kubernetes clusters, eliminating the need to route cross-cluster traffic through ingress controllers or API gateways. When Submariner's broker, gateways, or cable drivers degrade, cross-cluster service calls silently time out, multi-cluster DNS resolution fails, and workloads that span clusters lose connectivity without surfacing clear errors to application teams. Vigilmon gives you external visibility into the infrastructure Submariner depends on: the broker API server, gateway endpoint reachability, service discovery health, and SSL certificate validity.

What You'll Build

HTTP monitors on the Submariner broker's Kubernetes API for control-plane health
TCP monitors for each cluster gateway IP to verify tunnel endpoint reachability
A check on the Lighthouse DNS service discovery component
SSL certificate monitoring for all cluster API endpoints in your Submariner mesh
An alerting table mapping each Submariner failure mode to its remediation action

Prerequisites

Submariner installed across at least two Kubernetes clusters with a broker cluster
subctl CLI installed for diagnostics (subctl version confirms installation)
Gateway node external IPs accessible from outside your clusters
A free account at vigilmon.online

Step 1: Understand Submariner Architecture and What to Monitor

Submariner has three main components that must stay healthy for cross-cluster connectivity:

Broker Cluster                    Member Cluster A         Member Cluster B
+--------------------+            +----------------+       +----------------+
| broker namespace   |            | submariner-gw  |       | submariner-gw  |
| (API for endpoint  |<-- sync -->| (IPsec/WG/VXLAN|<----->| (IPsec/WG/VXLAN|
|  discovery)        |            |  cable driver) |       |  cable driver) |
+--------------------+            | lighthouse-dns |       | lighthouse-dns |
                                  +----------------+       +----------------+

Monitoring targets:

Broker API server: The Kubernetes API of the broker cluster stores Endpoint and Cluster CRDs that member clusters sync. Broker unavailability blocks new gateway registrations but does not immediately break existing tunnels
Gateway nodes: Each member cluster has one or more gateway nodes with external IPs. If the gateway node IP is unreachable, the IPsec/WireGuard/VXLAN tunnel drops
Lighthouse CoreDNS: Submariner's Lighthouse component enables cross-cluster service DNS resolution via ServiceImport. If Lighthouse is unhealthy, cross-cluster DNS queries fail
Cable driver health: The cable driver (IPsec, WireGuard, or VXLAN) is the tunnel mechanism; monitoring gateway reachability is the external proxy for cable driver health

Step 2: Monitor the Broker Cluster API Server

The broker cluster hosts the submariner-k8s-broker namespace where endpoint and cluster CRDs are synchronized. Member clusters connect to the broker API server to register their gateways and discover peer endpoints:

curl -k https://api.broker-cluster.example.com:6443/healthz
# Returns: ok

curl -k https://api.broker-cluster.example.com:6443/readyz
# Returns: ok

Log in to Vigilmon -> Add Monitor -> HTTP.
URL: https://api.broker-cluster.example.com:6443/healthz.
Check interval: 2 minutes.
Response timeout: 15 seconds.
Expected status: 200.
Keyword: ok.
Label: Submariner broker API server.
Click Save.

Impact when broker is down:

Existing tunnels remain operational -- cable drivers maintain established connections
New gateway registrations fail -- clusters that restart their gateway nodes cannot re-register
Cluster topology changes are not propagated -- adding new member clusters fails
subctl show connections may report stale endpoint data

Monitor the broker with a 2-minute interval rather than 60 seconds because existing tunnels survive short broker outages. Alert after 3 consecutive failures to avoid false alarms during broker cluster maintenance windows.

Step 3: Monitor Gateway Node Endpoints

Submariner gateway nodes have external IPs (or Elastic IPs/floating IPs in cloud environments) that peer gateways use to establish tunnels. Monitoring these IPs confirms that the tunnel endpoints are reachable at the network level.

Get gateway node IPs for each member cluster:

# List gateway nodes and their external IPs
kubectl get nodes -l submariner.io/gateway=true \
  -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.status.addresses[?(@.type=="ExternalIP")].address}{"\n"}{end}'

For each gateway node IP, add a TCP monitor:

Add Monitor -> TCP.
Host: <gateway-node-external-ip>.
Port: 4500 (IPsec NAT-T port, default for IPsec cable driver) or 4800 (VXLAN) or 4900 (WireGuard).
Check interval: 60 seconds.
Label: Submariner gateway: cluster-a node-1.
Click Save.

Repeat for every gateway node across all member clusters.

Port reference by cable driver: | Cable driver | Default port | Protocol | |---|---|---| | IPsec (default) | 4500 | UDP | | WireGuard | 4900 | UDP | | VXLAN | 4800 | UDP |

Firewall requirements: If TCP monitoring fails for UDP-only ports, monitor port 8080 (the Submariner gateway metrics port) or monitor the gateway node's SSH port (22) as a proxy for node reachability. Adjust firewall rules to allow Vigilmon's probe IPs to reach your gateway nodes.

Step 4: Monitor the Lighthouse Service Discovery Endpoint

Submariner's Lighthouse CoreDNS plugin enables cross-cluster service discovery. Services exported via ServiceExport are resolvable across clusters as <service>.<namespace>.svc.clusterset.local. When Lighthouse is unhealthy, cross-cluster DNS queries fail:

# Check Lighthouse CoreDNS pod health
kubectl get pods -n submariner-operator -l app=submariner-lighthouse-coredns

# Test cross-cluster DNS resolution
kubectl run dns-test --image=busybox --restart=Never --rm -i -- \
  nslookup my-service.my-namespace.svc.clusterset.local

If your Lighthouse CoreDNS service is exposed externally (e.g., through a LoadBalancer or NodePort), add an HTTP monitor:

Add Monitor -> HTTP.
URL: http://<lighthouse-dns-external-ip>:8080/health (Lighthouse exposes a health endpoint on port 8080).
Check interval: 2 minutes.
Response timeout: 10 seconds.
Expected status: 200.
Label: Lighthouse DNS health.
Click Save.

If Lighthouse is not externally exposed, monitor the CoreDNS pod indirectly by monitoring one of the cross-cluster services that Lighthouse resolves for -- if that service becomes unreachable, Lighthouse is likely the culprit.

Step 5: Monitor All Member Cluster API Servers

Each member cluster's API server is required for Submariner's operator to manage gateway deployments, sync endpoint CRDs, and react to cluster topology changes:

# Check member cluster API servers
curl -k https://api.cluster-a.example.com:6443/healthz
curl -k https://api.cluster-b.example.com:6443/healthz

For each member cluster:

Add Monitor -> HTTP.
URL: https://api.cluster-a.example.com:6443/healthz.
Check interval: 60 seconds.
Response timeout: 15 seconds.
Expected status: 200.
Keyword: ok.
Label: Submariner member: cluster-a API server.
Click Save.

Step 6: Monitor SSL Certificates

Submariner components authenticate to each other and to Kubernetes API servers using TLS. Expired certificates cause:

Broker client connections to fail (member clusters cannot sync endpoints)
subctl commands to fail with TLS verification errors
Gateway-to-gateway authentication failures for IPsec IKE (which uses certificates in cert-based mode)

# Check broker cluster cert
openssl s_client -connect api.broker-cluster.example.com:6443 2>/dev/null | \
  openssl x509 -noout -dates

# Check all member cluster certs
for cluster in cluster-a cluster-b; do
  echo "=== $cluster ==="
  openssl s_client -connect api.${cluster}.example.com:6443 2>/dev/null | \
    openssl x509 -noout -enddate
done

Add Monitor -> SSL Certificate.
Domain: api.broker-cluster.example.com.
Port: 6443.
Alert when expiry is within: 30 days.
Alert again: 14 days, 7 days, 3 days.
Click Save.

Repeat for all member cluster API server domains.

Step 7: Configure Alerting

In Vigilmon under Settings -> Notifications, configure your alert channels:

| Monitor | Trigger | Remediation action | |---|---|---| | Broker API server | Non-200 or timeout | Check broker cluster nodes; new gateways cannot register but existing tunnels survive | | Gateway node TCP | Connection refused or timeout | Gateway node unreachable; check node health, cloud routing, security groups | | Lighthouse DNS health | Non-200 response | Cross-cluster DNS broken; check kubectl get pods -n submariner-operator | | Member cluster API | Non-200 or timeout | Submariner operator cannot manage that cluster; check cluster control plane | | SSL certificates | < 30 days to expiry | Renew API server certs; verify Submariner broker client configs after renewal |

Alert sensitivity:

Gateway TCP monitors: 1 consecutive failure -- tunnel endpoint unreachable means cross-cluster traffic is disrupted
Broker API monitor: 3 consecutive failures -- short broker outages do not break existing tunnels
Member cluster API: 1 consecutive failure -- operator cannot reconcile gateway state

Verifying Submariner Health After an Alert

When a Vigilmon monitor fires, correlate it with Submariner internal state using subctl:

# Check connection status across all clusters
subctl show connections

# Run Submariner diagnostics
subctl diagnose all

# Check gateway pod status
kubectl get pods -n submariner-operator -l app=submariner-gateway

# Check cable driver logs for tunnel errors
kubectl logs -n submariner-operator -l app=submariner-gateway --tail=50

# Check endpoint sync status
kubectl get endpoints.submariner.io -n submariner-k8s-broker

Common Submariner Failure Modes and What Vigilmon Catches

| Scenario | Vigilmon monitor | |---|---| | Gateway node OOM killed or rebooted | Gateway TCP monitor fires; tunnel drops until node recovers | | Broker API server unreachable | Broker API monitor fires; new registrations fail; existing tunnels survive | | Cloud security group blocks UDP 4500 | Gateway TCP monitor fires; all IPsec tunnels fail to establish | | Gateway node external IP changes | Gateway TCP monitor fires for old IP; reconfigure Submariner and update monitor | | Lighthouse CoreDNS crash | Lighthouse health monitor fires; cross-cluster service DNS fails | | IPsec certificate rotation (cert-based mode) | SSL monitor flags cert expiry; tunnel renegotiation fails after expiry | | Member cluster API server unreachable | Member API monitor fires; Submariner operator cannot reconcile state | | Cable driver version mismatch after upgrade | Gateway TCP monitor fires; tunnel compatibility broken | | DNS misconfiguration for cluster endpoints | All API monitors fire simultaneously | | VXLAN MTU mismatch causes packet drops | TCP monitor passes but application-layer latency spikes |

Submariner's cross-cluster networking is foundational infrastructure: when it fails, workloads that span clusters lose connectivity silently, cross-cluster service calls time out without clear error messages, and multi-cluster service meshes degrade gracefully in ways that are hard to diagnose. Vigilmon gives you external eyes on the broker, gateway endpoints, Lighthouse DNS, and API server certificates across your entire Submariner mesh -- so you know the moment cross-cluster connectivity is at risk, before your application teams start filing bug reports about timeouts.

Start monitoring your Submariner network in under 5 minutes -- register free at vigilmon.online.