tutorial

Monitoring Skupper with Vigilmon

Skupper creates virtual application networks across Kubernetes clusters without VPNs or firewall changes — but when a Skupper router goes offline, its inter-cluster links drop, or its control plane loses connectivity, cross-cluster services silently fail. Here's how to monitor Skupper router availability, inter-cluster link health, service proxy reachability, and network liveness with Vigilmon.

Skupper is a layer-7 service interconnect for Kubernetes that enables cross-cluster communication without VPNs, shared networks, or complex firewall rules. It deploys a router per namespace, exchanges link tokens between clusters, and proxies service traffic over mutual-TLS AMQP connections. Teams using Skupper for multi-cloud or multi-region Kubernetes deployments get cluster isolation without application-level cross-cluster routing complexity — but when a Skupper router pod is evicted, a link token expires, or a proxy binding silently unbinds from its backing service, cross-cluster service calls begin failing with connection errors and the only signal is application-level 503s in a completely different cluster. Vigilmon gives you external monitoring for Skupper router availability, inter-cluster link status, service proxy reachability, and network liveness so Skupper connectivity failures surface within minutes rather than during an incident.

What You'll Set Up

  • Skupper router pod availability via HTTP health check
  • Inter-cluster link status monitoring via heartbeat
  • Service proxy endpoint reachability
  • Skupper console reachability
  • Alert channel configuration for multi-cluster networking failures

Prerequisites

  • Skupper 1.5+ deployed in one or more Kubernetes namespaces
  • kubectl access to all Skupper namespaces
  • skupper CLI installed on your monitoring host
  • A free Vigilmon account

Step 1: Monitor the Skupper Router Health Endpoint

Skupper's router pod exposes an HTTP management endpoint on port 9090. When the router pod crashes, is evicted by the Kubernetes scheduler, or fails its readiness probe, all cross-cluster connections through that router drop immediately — services in other clusters that depend on proxied services in this namespace begin timing out. Monitor router availability as the first signal:

  1. Expose the Skupper router management endpoint via a Kubernetes service or ingress:
# skupper-router-monitor-svc.yaml
apiVersion: v1
kind: Service
metadata:
  name: skupper-router-monitor
  namespace: your-app-namespace
spec:
  selector:
    skupper.io/component: router
  ports:
    - name: management
      port: 9090
      targetPort: 9090
  type: ClusterIP
  1. Expose via an ingress or kubectl port-forward for external monitoring:
# For testing — in production use an ingress
kubectl port-forward -n your-app-namespace svc/skupper-router 9090:9090 &
  1. In Vigilmon, click Add Monitor → HTTP/HTTPS.
  2. Set the URL to https://skupper-router.your-org.internal:9090/healthz.
  3. Set Check interval to 1 minute.
  4. Under Expected response code, set to 200.
  5. Set Timeout to 10 seconds.

Alternatively, if you have an HTTP-accessible application endpoint in your namespace, configure Skupper to expose the /skupper/healthz path through the console service:

# Check if the Skupper console is accessible
kubectl get svc -n your-app-namespace skupper
# The skupper service typically exposes ports 8080 (console) and 8081

Step 2: Monitor Inter-Cluster Link Status

Skupper links are the AMQP connections between routers in different clusters. A link can drop due to a network partition, an expired certificate, a firewall rule change, or a router pod restart that doesn't fully re-establish the connection. When a link drops, services proxied across that link begin failing — but the failure is in a different cluster, making root cause identification slow. Monitor link health from both sides:

#!/bin/bash
# /usr/local/bin/skupper-link-check.sh
# Run this on a host with kubectl access to the monitored cluster

HEARTBEAT_URL="https://vigilmon.online/heartbeat/abc123"
NAMESPACE="your-app-namespace"
KUBECONFIG="/path/to/kubeconfig"    # Adjust to your kubeconfig path

# Check all Skupper links in the namespace
LINK_STATUS=$(kubectl --kubeconfig="$KUBECONFIG" \
  -n "$NAMESPACE" \
  get links.skupper.io \
  -o json 2>/dev/null)

if [ -z "$LINK_STATUS" ] || [ "$LINK_STATUS" = "null" ]; then
  # Try the skupper CLI as fallback
  LINK_STATUS=$(skupper --namespace "$NAMESPACE" link status --output json 2>/dev/null)
fi

if [ -z "$LINK_STATUS" ]; then
  echo "Could not retrieve Skupper link status"
  exit 1
fi

# Count connected vs total links
RESULT=$(echo "$LINK_STATUS" | python3 -c "
import sys, json

data = json.load(sys.stdin)

# Handle both CRD list and skupper CLI output formats
items = []
if isinstance(data, dict) and 'items' in data:
    items = data['items']
elif isinstance(data, list):
    items = data

total = len(items)
connected = 0
down_links = []

for item in items:
    # CRD format
    status = item.get('status', {})
    if isinstance(status, dict):
        state = status.get('configured', False) or status.get('operational', False)
        if state:
            connected += 1
        else:
            name = item.get('metadata', {}).get('name', 'unknown')
            down_links.append(name)
    # CLI format
    elif item.get('connected', False):
        connected += 1
    else:
        down_links.append(item.get('name', 'unknown'))

print(f'{connected}/{total}')
if down_links:
    print('DOWN: ' + ', '.join(down_links), file=sys.stderr)
" 2>/tmp/skupper-link-errors)

TOTAL=$(echo "$RESULT" | cut -d/ -f2)
CONNECTED=$(echo "$RESULT" | cut -d/ -f1)

if [ -z "$TOTAL" ] || [ "$TOTAL" -eq 0 ]; then
  echo "No Skupper links configured in namespace ${NAMESPACE}"
  curl -s "$HEARTBEAT_URL"
  exit 0
fi

if [ "$CONNECTED" -eq "$TOTAL" ]; then
  echo "All Skupper links connected: ${CONNECTED}/${TOTAL}"
  curl -s "$HEARTBEAT_URL"
else
  echo "Skupper link failure: ${CONNECTED}/${TOTAL} connected"
  cat /tmp/skupper-link-errors
  exit 1
fi

Schedule this as a cron job with the Vigilmon heartbeat set to 3 minutes expected interval:

*/2 * * * * /usr/local/bin/skupper-link-check.sh >> /var/log/skupper-link-check.log 2>&1

Deploy this script for each cluster in your Skupper network — a link outage is only visible from the cluster that owns the link, not from the cluster on the other end.

Step 3: Monitor Proxied Service Endpoint Reachability

Skupper's value is making services in one cluster reachable from another. Even with links connected, a service proxy can be misconfigured, a selector can become stale after a namespace migration, or a skupper expose can fail silently if the backing pods are removed. Directly test the actual cross-cluster service endpoint from the consuming cluster:

  1. Identify a proxied service endpoint in your remote-cluster namespace:
kubectl -n your-app-namespace get services
# Look for services created by Skupper — they have skupper.io annotations
kubectl -n your-app-namespace get svc -l skupper.io/proxied-service=true
  1. In Vigilmon, click Add Monitor → HTTP/HTTPS.
  2. Set the URL to the health endpoint of a critical cross-cluster service (e.g., https://api.your-org.internal/health).
  3. Set Check interval to 1 minute.
  4. Under Expected response code, set to 200.
  5. Set Timeout to 15 seconds — cross-cluster traffic has higher latency than intra-cluster.

This monitor catches: link drops, proxy binding failures, backing service crashes in the remote cluster, and slow cross-cluster connections (high latency alerts before full failure).

For TCP-based services (databases, message queues) exposed via Skupper:

  1. In Vigilmon, click Add Monitor → TCP.
  2. Set Host to the proxied service hostname and Port to the service port.
  3. Set Check interval to 2 minutes.

Step 4: Monitor the Skupper Console

Skupper ships a web console that shows network topology, link status, and service traffic statistics. The console is a critical operational tool during connectivity incidents. Monitor its availability:

  1. In Vigilmon, click Add Monitor → HTTP/HTTPS.
  2. Set the URL to https://skupper.your-org.internal:8080.
  3. Set Check interval to 5 minutes.
  4. Under Expected response, add keyword Skupper.
  5. Set Timeout to 10 seconds.

Get the console URL from your cluster:

kubectl -n your-app-namespace get svc skupper -o jsonpath='{.status.loadBalancer.ingress[0].hostname}'
# or for NodePort:
kubectl -n your-app-namespace get svc skupper -o jsonpath='{.spec.ports[?(@.name=="console")].nodePort}'

Step 5: Monitor Router Certificate Expiry

Skupper uses mutual TLS for all inter-cluster links, generating certificates during skupper init and link token creation. These certificates have a default 3-year validity, but in environments with strict certificate rotation policies or after skupper delete and reinit cycles, certificates can expire unexpectedly. An expired router certificate causes all links to drop simultaneously:

#!/bin/bash
# /usr/local/bin/skupper-cert-check.sh

HEARTBEAT_URL="https://vigilmon.online/heartbeat/def456"
NAMESPACE="your-app-namespace"
MIN_DAYS_VALID=30   # Alert if any cert expires within 30 days

# Check Skupper TLS secrets in the namespace
SECRETS=$(kubectl -n "$NAMESPACE" get secrets \
  -l skupper.io/type=connection-token \
  --no-headers \
  -o custom-columns=NAME:.metadata.name 2>/dev/null)

if [ -z "$SECRETS" ]; then
  # Try without the label filter — different Skupper versions use different labels
  SECRETS=$(kubectl -n "$NAMESPACE" get secrets \
    --no-headers \
    -o custom-columns=NAME:.metadata.name 2>/dev/null | grep -i skupper | grep -i ca\|cert\|token || true)
fi

ALL_OK=true

while IFS= read -r SECRET; do
  [ -z "$SECRET" ] && continue

  # Extract certificate from secret
  CERT_B64=$(kubectl -n "$NAMESPACE" get secret "$SECRET" \
    -o jsonpath='{.data.tls\.crt}' 2>/dev/null)

  [ -z "$CERT_B64" ] && continue

  EXPIRY=$(echo "$CERT_B64" | base64 -d | \
    openssl x509 -noout -enddate 2>/dev/null | \
    sed 's/notAfter=//')

  if [ -z "$EXPIRY" ]; then
    continue
  fi

  EXPIRY_TS=$(date -d "$EXPIRY" +%s 2>/dev/null)
  NOW_TS=$(date +%s)
  DAYS_LEFT=$(( (EXPIRY_TS - NOW_TS) / 86400 ))

  if [ "$DAYS_LEFT" -lt "$MIN_DAYS_VALID" ]; then
    echo "Skupper cert ${SECRET} expires in ${DAYS_LEFT} days"
    ALL_OK=false
  else
    echo "Skupper cert ${SECRET} OK: ${DAYS_LEFT} days remaining"
  fi
done <<< "$SECRETS"

if [ "$ALL_OK" = "true" ]; then
  curl -s "$HEARTBEAT_URL"
else
  exit 1
fi

Set the heartbeat expected interval to 25 hours. This check runs daily and you'll be alerted if it stops running (namespace deletion, kubeconfig expiry) before the next check.

Step 6: Set Up Alert Channels

Configure Vigilmon to route Skupper alerts to the right team:

  1. Go to Alert Channels in Vigilmon.
  2. Add a Slack webhook to #platform-engineering or #networking — Skupper failures affect cross-cluster services and are typically platform team responsibility.
  3. Add PagerDuty for the link status heartbeat and proxied service monitors — a broken inter-cluster link causes application-level failures in the consuming cluster.
  4. Add email for the certificate expiry monitor — expiring certs need action within days, not minutes.
  5. Set Consecutive failures before alert to 2 for the router health monitor — pod restarts during rolling updates cause brief gaps.

Add maintenance windows during Skupper network topology changes:

curl -X POST https://vigilmon.online/api/maintenance \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -d '{
    "monitor_id": "SKUPPER_LINK_HEARTBEAT_ID",
    "duration_minutes": 20,
    "reason": "Skupper network link rotation — adding cluster-3"
  }'

Summary

| Monitor | Target | What It Catches | |---|---|---| | HTTP monitor (router health) | Router /healthz endpoint | Router pod crash, eviction, readiness failure | | HTTP monitor (console) | Skupper console port | Console unavailable during connectivity incidents | | HTTP monitor (proxied service) | Cross-cluster service endpoint | Link drops, proxy binding failures, service removal | | TCP monitor (proxied TCP service) | Proxied port | Database/queue proxy failures | | Cron heartbeat (link status) | All links connected check | Link disconnection, token expiry, network partition | | Cron heartbeat (cert expiry) | TLS certificate validity | Certificate expiry before inter-cluster links drop |

Skupper connects services across clusters without VPNs — but when its router pod crashes, a link drops, or a certificate expires, cross-cluster service calls begin failing in a different cluster than where the root cause lives. Vigilmon gives you external monitoring for every layer of Skupper so network failures surface within minutes rather than being discovered by a developer wondering why their cross-cluster service call is returning 503.

Start monitoring for free at vigilmon.online.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles