tutorial

How to Monitor Admiralty with Vigilmon (HTTP + TCP + Alerts)

Admiralty is a Kubernetes operator for multi-cluster scheduling and workload federation. Instead of deploying pods directly to a target cluster, it intercept...

Admiralty is a Kubernetes operator for multi-cluster scheduling and workload federation. Instead of deploying pods directly to a target cluster, it intercepts pod creation in a source cluster and uses a "proxy pod" pattern: a lightweight proxy pod stays in the source cluster while Admiralty's controllers schedule the real pod onto one or more registered target clusters. The source cluster runs the multicluster-scheduler controller; each target cluster runs the multicluster-scheduler agent, and the source cluster sees each target as a virtual node. When the source controller crashes, new cross-cluster scheduling stops silently — pods stay Pending with no explanation. When the target agent crashes, proxy pods are created but never transition to Running. These failures have no representation in either cluster's standard health checks.

In this tutorial you'll set up comprehensive uptime monitoring for your Admiralty deployment using Vigilmon — free tier, no credit card required.

Why Admiralty deployments need external monitoring

Admiralty introduces a category of scheduling failure that neither the source nor the target cluster exposes through normal Kubernetes observability:

  • Source controller crash silently stops cross-cluster scheduling — the multicluster-scheduler controller in the source cluster exits; new pods that should be federated to target clusters stay Pending indefinitely with a SchedulerName that no local scheduler recognizes; no event fires, no node condition changes
  • Target agent crash means proxy pods never become Running — the multicluster-scheduler agent in the target cluster stops reconciling; proxy pods in the source cluster report Pending or show stale status; actual workload pods in the target cluster may still be running but no new ones can be started
  • Virtual node registration drops — Admiralty registers each target cluster as a virtual node (e.g. admiralty-my-target-cluster) in the source cluster; if that registration lapses, Kubernetes marks the node NotReady and stops scheduling to it, breaking the entire target cluster's capacity without alerting the target cluster operators
  • Cross-cluster authentication tokens expire — Admiralty uses kubeconfig secrets to authenticate from the source controller to target clusters; when these tokens rotate or expire, scheduling attempts fail silently inside the controller with auth errors that never surface as Kubernetes events or pod conditions
  • Candidate pod feedback loop breaks — Admiralty uses a CandidatePod mechanism to negotiate placement across clusters; if the feedback loop between source and target breaks (network partition, API server degradation), pods are left in an intermediate Pending state that never resolves without manual intervention

These failures are especially damaging in mixed environments where some workloads schedule locally and others federate — a partial failure looks like normal operation until a specific workload type fails to start.

What you'll need

  • An Admiralty deployment with at least one source cluster running the multicluster-scheduler controller and one target cluster running the agent
  • A federated workload or test deployment that routes through at least one virtual node, exposed via Service or Ingress
  • A free Vigilmon account (sign up takes 30 seconds)

Step 1: Expose a health endpoint for monitoring

Admiralty's controller and agent both expose metrics and health endpoints on port 8080. First, verify your components and virtual nodes are healthy:

# Check the multicluster-scheduler controller (context: source)
kubectl --context source get pods -n admiralty
# NAME                                      READY   STATUS    RESTARTS   AGE
# multicluster-scheduler-6d9f8c7b5-xk2pq   1/1     Running   0          7d

# Check the multicluster-scheduler agent (context: target)
kubectl --context target get pods -n admiralty
# NAME                                      READY   STATUS    RESTARTS   AGE
# multicluster-scheduler-7f8b9d6c4-mn3rt   1/1     Running   0          7d

# Check virtual nodes registered in the source cluster
kubectl --context source get nodes
# NAME                            STATUS   ROLES    AGE   VERSION
# admiralty-my-target-cluster     Ready    agent    7d    v0.0.0-admiralty
# source-node-1                   Ready    worker   14d   v1.27.3

# Port-forward to check controller health endpoint
kubectl --context source port-forward -n admiralty deploy/multicluster-scheduler 8080:8080
curl http://localhost:8080/healthz
# ok

Deploy a federated probe workload that Admiralty schedules onto the target cluster. This validates the full scheduling path end-to-end:

# admiralty-probe.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: admiralty-probe
  namespace: admiralty
  annotations:
    multicluster.admiralty.io/elect: ""
spec:
  replicas: 1
  selector:
    matchLabels:
      app: admiralty-probe
  template:
    metadata:
      labels:
        app: admiralty-probe
      annotations:
        multicluster.admiralty.io/elect: ""
    spec:
      schedulerName: admiralty
      containers:
        - name: probe
          image: nginx:alpine
          ports:
            - containerPort: 80
          volumeMounts:
            - name: config
              mountPath: /etc/nginx/conf.d
          readinessProbe:
            httpGet:
              path: /healthz
              port: 80
            initialDelaySeconds: 5
            periodSeconds: 10
      volumes:
        - name: config
          configMap:
            name: admiralty-probe-nginx
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: admiralty-probe-nginx
  namespace: admiralty
data:
  default.conf: |
    server {
      listen 80;
      location /healthz {
        return 200 '{"status":"ok","component":"admiralty-probe","scheduler":"admiralty"}';
        add_header Content-Type application/json;
      }
    }
---
apiVersion: v1
kind: Service
metadata:
  name: admiralty-probe
  namespace: admiralty
spec:
  type: LoadBalancer
  selector:
    app: admiralty-probe
  ports:
    - port: 80
      targetPort: 80
      protocol: TCP

kubectl --context source apply -f admiralty-probe.yaml

# Get the LoadBalancer IP (assigned in the target cluster, surfaced via Admiralty)
kubectl --context source get svc admiralty-probe -n admiralty
# NAME              TYPE           CLUSTER-IP      EXTERNAL-IP      PORT(S)   AGE
# admiralty-probe   LoadBalancer   10.96.31.200    203.0.113.84    80/TCP    3m

# Verify proxy pod in source cluster
kubectl --context source get pods -n admiralty -l app=admiralty-probe
# NAME                               READY   STATUS    RESTARTS   AGE
# admiralty-probe-5d8c7f9b4-ab3mn    1/1     Running   0          3m

# Verify real pod in target cluster
kubectl --context target get pods -n admiralty -l app=admiralty-probe
# NAME                               READY   STATUS    RESTARTS   AGE
# admiralty-probe-5d8c7f9b4-ab3mn    1/1     Running   0          2m

# Confirm the health endpoint responds
curl http://203.0.113.84/healthz
# {"status":"ok","component":"admiralty-probe","scheduler":"admiralty"}

Step 2: Set up HTTP monitoring in Vigilmon

With your probe endpoint exposed, add it to Vigilmon:

  1. Log in to vigilmon.online and go to Monitors → New Monitor
  2. Choose HTTP / HTTPS as the type
  3. Set the URL to your probe endpoint: http://203.0.113.84/healthz
  4. Set the check interval to 1 minute
  5. Under Expected response, set:
    • Status code: 200
    • Response body contains: "status":"ok"
  6. Save the monitor

This monitor validates the entire Admiralty scheduling path: the source controller must be running and reconciling, the virtual node for the target cluster must be Ready, the target agent must be placing real pods, and the LoadBalancer must be routing from the target cluster back to the public endpoint. A single failed 200 means at least one of these components has degraded.

Why external monitoring catches what Kubernetes misses

When Admiralty's source controller crashes, pods that should federate simply stay Pending in the source cluster. Kubernetes generates no alerts — the pod is scheduled to the admiralty scheduler, which is simply absent. Vigilmon detects the resulting outage in under a minute because the probe pod stops responding as soon as the target's instance is unreachable. Its multi-region consensus model eliminates false positives from transient probe-path blips.

Step 3: Monitor TCP ports and the Admiralty controller health endpoint

Add monitors to detect failures at the controller and virtual node level:

Probe service TCP check:

  1. Go to Monitors → New Monitor
  2. Choose TCP Port
  3. Enter: Host 203.0.113.84, Port 80
  4. Save the monitor

Admiralty controller health via Ingress:

Expose the controller's /healthz endpoint through an Ingress in the source cluster so Vigilmon can probe it directly:

# admiralty-controller-ingress.yaml
apiVersion: v1
kind: Service
metadata:
  name: admiralty-controller-health
  namespace: admiralty
spec:
  selector:
    app: multicluster-scheduler
  ports:
    - port: 8080
      targetPort: 8080
      name: metrics
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: admiralty-controller-health
  namespace: admiralty
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /healthz
spec:
  rules:
    - host: admiralty-health.your-source-cluster.com
      http:
        paths:
          - path: /healthz
            pathType: Exact
            backend:
              service:
                name: admiralty-controller-health
                port:
                  number: 8080

kubectl --context source apply -f admiralty-controller-ingress.yaml

Add an HTTP monitor in Vigilmon for https://admiralty-health.your-source-cluster.com/healthz expecting status 200 and body ok.

Also add a TCP monitor for the controller metrics port to detect process-level failures even when the HTTP layer is unreachable:

  1. Go to Monitors → New Monitor
  2. Choose TCP Port
  3. Enter: Host admiralty-health.your-source-cluster.com, Port 8080
  4. Save the monitor

Step 4: Configure alert channels

An Admiralty failure silently stops cross-cluster workload placement — new deployments pend forever while on-call has no signal.

Email alerts

  1. In Vigilmon, go to Alert Channels → Add Channel → Email
  2. Enter your platform team's on-call email
  3. Assign the channel to all Admiralty monitors

Webhook alerts (Slack, PagerDuty, etc.)

  1. Go to Alert Channels → Add Channel → Webhook
  2. Enter your webhook URL
  3. Assign the channel to your monitors

Example payload when the Admiralty probe goes down:

{
  "monitor_name": "Admiralty Probe Health",
  "status": "down",
  "url": "http://203.0.113.84/healthz",
  "started_at": "2024-01-15T14:05:00Z",
  "duration_seconds": 120
}

Run these diagnostics immediately when an alert fires:

# Check source controller
kubectl --context source get pods -n admiralty
kubectl --context source logs -n admiralty -l app=multicluster-scheduler --tail=100

# Check virtual node status in source cluster
kubectl --context source get node admiralty-my-target-cluster
kubectl --context source describe node admiralty-my-target-cluster

# Check target agent
kubectl --context target get pods -n admiralty
kubectl --context target logs -n admiralty -l app=multicluster-scheduler --tail=100

# Check proxy pods and their status annotations
kubectl --context source get pods -n admiralty -l app=admiralty-probe -o yaml | grep -A10 annotations

# Check for pending federated pods stuck in source
kubectl --context source get pods -A --field-selector status.phase=Pending

# Check authentication secrets are present and not expired
kubectl --context source get secrets -n admiralty | grep kubeconfig

Step 5: Create a public status page

Multi-cluster platforms built on Admiralty often underpin internal developer platforms where teams self-service workload placement. Give them a status page.

  1. In Vigilmon, go to Status Pages → New Status Page
  2. Give it a name: "Admiralty Multi-Cluster Scheduler Status"
  3. Add your monitors: Admiralty Probe HTTP, Probe TCP 80, Controller Health HTTP, Controller TCP 8080
  4. Group them: Workload Federation, Controller Health, Port Availability
  5. Publish the page

Share this URL with application teams whose deployments rely on Admiralty for cross-cluster scheduling.

Putting it all together

Here's a summary of the monitors to create for an Admiralty deployment:

| Monitor | Type | What it catches | |---------|------|-----------------| | http://203.0.113.84/healthz | HTTP | Full federation path (controller → virtual node → target agent → pod → response) | | 203.0.113.84:80 | TCP | Probe service TCP reachability | | https://admiralty-health.your-source-cluster.com/healthz | HTTP | Source controller process health | | admiralty-health.your-source-cluster.com:8080 | TCP | Controller metrics port reachability |

And the Admiralty diagnostics to run when an alert fires:

# Quick triage: controller or target agent?
kubectl --context source get pods -n admiralty
kubectl --context target get pods -n admiralty

# Check virtual node readiness in source
kubectl --context source get nodes | grep admiralty
kubectl --context source get node admiralty-my-target-cluster -o yaml | grep -A10 conditions

# Check controller logs for auth or scheduling errors
kubectl --context source logs -n admiralty deploy/multicluster-scheduler | grep -E "error|401|403|token|expired"

# Check proxy pod placement annotations in source
kubectl --context source get pods -n admiralty -l app=admiralty-probe -o jsonpath='{.items[*].metadata.annotations}' | python3 -m json.tool

# Check real pod in target cluster
kubectl --context target get pods -n admiralty -o wide

# Check events across both clusters
kubectl --context source get events -n admiralty --sort-by='.lastTimestamp' | tail -20
kubectl --context target get events -n admiralty --sort-by='.lastTimestamp' | tail -20

What's next

  • SSL certificate monitoring — add SSL monitors for any HTTPS endpoints served from workloads federated through Admiralty to target clusters
  • Heartbeat monitoring — run a scheduled job that federates to a target cluster on a cron schedule and sends a Vigilmon heartbeat on completion, confirming the scheduling path works end-to-end
  • Per-target coverage — if you register multiple target clusters as virtual nodes, create a dedicated monitor set per target to isolate which cluster's agent is failing when an alert fires

Get started free at vigilmon.online — no credit card, monitors start running in under a minute.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles