fleet_health.py

Prerequisites

• Rancher Fleet 0.9.x or later (standalone or integrated with Rancher Manager)
• kubectl with access to the Fleet management cluster
• A free account at vigilmon.online

Part 1: Understanding what to monitor in Rancher Fleet

Fleet has two core failure surfaces:

1. Fleet manager (fleet-manager) — the central controller running in the management cluster that reads GitRepo CRDs and creates Bundle objects
2. Fleet agents — one agent per downstream cluster that applies Bundle objects to the cluster

Beyond pod readiness, the most important health signal is whether GitRepo synchronization is completing on schedule. A GitRepo that stays in a Modified, NotReady, or OutOfSync state for too long means the cluster has diverged from the desired state in Git.

Part 2: Build a Fleet HTTP health endpoint

Create fleet_health.py

import os
import subprocess
import json
import datetime
from fastapi import FastAPI, Response

app = FastAPI()

FLEET_NAMESPACE = os.environ.get("FLEET_NAMESPACE", "cattle-fleet-system")
GITREPO_STALL_MINUTES = int(os.environ.get("GITREPO_STALL_MINUTES", "30"))


def kubectl_json(*args):
    result = subprocess.run(
        ["kubectl", *args, "-o", "json"],
        capture_output=True, text=True, timeout=15,
    )
    if result.returncode != 0:
        raise RuntimeError(result.stderr.strip()[:300])
    return json.loads(result.stdout)


@app.get("/health")
def health(response: Response):
    checks = {}

    # Fleet manager pod readiness
    try:
        pods = kubectl_json(
            "get", "pods", "-n", FLEET_NAMESPACE,
            "-l", "app=fleet-manager",
        )
        items = pods.get("items", [])
        not_ready = [
            p["metadata"]["name"]
            for p in items
            if not all(
                c.get("status") == "True"
                for c in p.get("status", {}).get("conditions", [])
                if c.get("type") == "Ready"
            )
        ]
        if not_ready:
            checks["fleet_manager"] = f"degraded: {', '.join(not_ready)}"
        else:
            checks["fleet_manager"] = f"ok ({len(items)} ready)"
    except Exception as e:
        checks["fleet_manager"] = f"error: {e}"

    # GitRepo sync status
    try:
        gitrepos = kubectl_json("get", "gitrepos", "-A")
        stalled = []
        errored = []
        for gr in gitrepos.get("items", []):
            ns = gr["metadata"]["namespace"]
            name = gr["metadata"]["name"]
            conditions = gr.get("status", {}).get("conditions", [])
            ready = next(
                (c for c in conditions if c.get("type") == "Ready"), None
            )
            if ready and ready.get("status") != "True":
                errored.append(f"{ns}/{name}: {ready.get('message', 'unknown')[:80]}")

            # Check last sync time
            last_synced = gr.get("status", {}).get("lastSyncedAt", "")
            if last_synced:
                synced_at = datetime.datetime.fromisoformat(
                    last_synced.replace("Z", "+00:00")
                )
                age_min = (
                    datetime.datetime.now(datetime.timezone.utc) - synced_at
                ).total_seconds() / 60
                if age_min > GITREPO_STALL_MINUTES:
                    stalled.append(f"{ns}/{name} (last synced {int(age_min)}m ago)")

        if errored:
            checks["gitrepo_errors"] = errored
        else:
            checks["gitrepo_errors"] = "ok"

        if stalled:
            checks["gitrepo_stalled"] = f"warning: {'; '.join(stalled)}"
        else:
            checks["gitrepo_stalled"] = "ok"

    except Exception as e:
        checks["gitrepo_sync"] = f"error: {e}"

    # Bundle deployment health
    try:
        bundles = kubectl_json("get", "bundles", "-A")
        not_ready_bundles = [
            f"{b['metadata']['namespace']}/{b['metadata']['name']}"
            for b in bundles.get("items", [])
            if b.get("status", {}).get("summary", {}).get("notReady", 0) > 0
        ]
        if not_ready_bundles:
            checks["bundles"] = f"warning: {len(not_ready_bundles)} bundles not ready"
        else:
            checks["bundles"] = "ok"
    except Exception as e:
        checks["bundles"] = f"error: {e}"

    manager_ok = "ok" in str(checks.get("fleet_manager", ""))
    gitrepo_ok = checks.get("gitrepo_errors") == "ok"
    healthy = manager_ok and gitrepo_ok

    response.status_code = 200 if healthy else 503
    return {"status": "ok" if healthy else "degraded", "checks": checks}

Run the health service

pip install fastapi uvicorn
uvicorn fleet_health:app --host 0.0.0.0 --port 8098

Deploy in-cluster

# fleet-health-deploy.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: fleet-health
  namespace: cattle-fleet-system
spec:
  replicas: 1
  selector:
    matchLabels:
      app: fleet-health
  template:
    metadata:
      labels:
        app: fleet-health
    spec:
      serviceAccountName: fleet-health-sa
      containers:
        - name: health
          image: python:3.12-slim
          command:
            - sh
            - -c
            - |
              pip install -q fastapi uvicorn &&
              uvicorn fleet_health:app --host 0.0.0.0 --port 8098
          ports:
            - containerPort: 8098
          env:
            - name: FLEET_NAMESPACE
              value: cattle-fleet-system
            - name: GITREPO_STALL_MINUTES
              value: "30"
---
apiVersion: v1
kind: Service
metadata:
  name: fleet-health
  namespace: cattle-fleet-system
spec:
  selector:
    app: fleet-health
  ports:
    - port: 8098
      targetPort: 8098
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: fleet-health-sa
  namespace: cattle-fleet-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: fleet-health-reader
rules:
  - apiGroups: [""]
    resources: ["pods"]
    verbs: ["get", "list"]
  - apiGroups: ["fleet.cattle.io"]
    resources: ["gitrepos", "bundles", "bundledeployments", "clusters"]
    verbs: ["get", "list"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: fleet-health-reader
subjects:
  - kind: ServiceAccount
    name: fleet-health-sa
    namespace: cattle-fleet-system
roleRef:
  kind: ClusterRole
  name: fleet-health-reader
  apiGroup: rbac.authorization.k8s.io

kubectl apply -f fleet-health-deploy.yaml

Verify the endpoint

kubectl port-forward svc/fleet-health 8098:8098 -n cattle-fleet-system
curl http://localhost:8098/health

Expected healthy response:

{
  "status": "ok",
  "checks": {
    "fleet_manager": "ok (2 ready)",
    "gitrepo_errors": "ok",
    "gitrepo_stalled": "ok",
    "bundles": "ok"
  }
}

Part 3: Set up HTTP monitoring in Vigilmon

Monitor Fleet manager and GitRepo sync

1. Log in to vigilmon.online and click Add Monitor.
2. Choose HTTP(S) monitor.
3. Enter: http://your-cluster-ingress:8098/health
4. Set interval to 2 minutes.
5. Add a keyword check: must contain "status":"ok".
6. Add your alert channel.
7. Click Save.

Monitor Fleet per workspace

In multi-workspace Fleet deployments, add a workspace-scoped monitor:

| Monitor name | URL | Notes | |---|---|---| | Fleet — production workspace | http://fleet-health:8098/health?workspace=fleet-default | Main workspace | | Fleet — staging workspace | http://fleet-health:8098/health?workspace=fleet-staging | Staging clusters |

Monitor downstream cluster agent connectivity

Fleet agents in downstream clusters phone home to the management cluster. If an agent loses connectivity, that cluster diverges silently.

Add a TCP monitor in Vigilmon for the Fleet agent registration endpoint:

1. Add Monitor → TCP port monitor.
2. Host: fleet-management.example.com
3. Port: 443
4. Interval: 5 minutes.

Part 4: Monitor GitRepo sync freshness with a standalone check

For critical GitRepos that must sync on a tight schedule (e.g., security policy updates), add a dedicated check.

# gitrepo_freshness.py
import os
import subprocess
import json
import datetime
from fastapi import FastAPI, Response

app = FastAPI()

CRITICAL_GITREPO_NS = os.environ.get("CRITICAL_GITREPO_NS", "fleet-default")
CRITICAL_GITREPO_NAME = os.environ.get("CRITICAL_GITREPO_NAME", "security-policies")
MAX_SYNC_AGE_MINUTES = int(os.environ.get("MAX_SYNC_AGE_MINUTES", "15"))


@app.get("/health/gitrepo")
def gitrepo_health(response: Response):
    try:
        result = subprocess.run(
            [
                "kubectl", "get", "gitrepo", CRITICAL_GITREPO_NAME,
                "-n", CRITICAL_GITREPO_NS, "-o", "json",
            ],
            capture_output=True, text=True, timeout=15,
        )
        if result.returncode != 0:
            response.status_code = 503
            return {"status": "error", "detail": result.stderr.strip()[:200]}

        gr = json.loads(result.stdout)
        last_synced = gr.get("status", {}).get("lastSyncedAt", "")
        if not last_synced:
            response.status_code = 503
            return {"status": "error", "detail": "lastSyncedAt not available"}

        synced_at = datetime.datetime.fromisoformat(
            last_synced.replace("Z", "+00:00")
        )
        age_min = (
            datetime.datetime.now(datetime.timezone.utc) - synced_at
        ).total_seconds() / 60

        if age_min > MAX_SYNC_AGE_MINUTES:
            response.status_code = 503
            return {
                "status": "stale",
                "last_synced_at": last_synced,
                "age_minutes": int(age_min),
                "threshold_minutes": MAX_SYNC_AGE_MINUTES,
            }

        return {
            "status": "ok",
            "last_synced_at": last_synced,
            "age_minutes": int(age_min),
        }

    except Exception as e:
        response.status_code = 503
        return {"status": "error", "detail": str(e)}

Add a Vigilmon monitor for /health/gitrepo with a 5-minute interval and keyword "status":"ok".

Part 5: Docker Compose for local development

# docker-compose.yml
version: "3.8"

services:
  fleet-health:
    image: python:3.12-slim
    command: >
      sh -c "pip install -q fastapi uvicorn &&
             uvicorn fleet_health:app --host 0.0.0.0 --port 8098"
    ports:
      - "8098:8098"
    volumes:
      - ./fleet_health.py:/app/fleet_health.py
      - ~/.kube:/root/.kube:ro
    working_dir: /app
    environment:
      FLEET_NAMESPACE: cattle-fleet-system
      GITREPO_STALL_MINUTES: "30"
      KUBECONFIG: /root/.kube/config

Part 6: Webhook alerts for Fleet control plane outages

Fleet manages hundreds of clusters. A control plane outage means all of them stop receiving updates — security patches, configuration changes, and new rollouts are silently blocked.

// webhook-receiver.ts
import express from 'express';

const app = express();
app.use(express.json());

app.post('/webhook/vigilmon', async (req, res) => {
  const { monitor_name, status, url, checked_at } = req.body;

  if (status === 'down') {
    console.error('[FLEET] Rancher Fleet control plane DOWN', {
      monitor: monitor_name,
      url,
      at: checked_at,
    });

    await alertPlatformTeam({
      title: `Fleet DOWN: ${monitor_name}`,
      severity: 'critical',
      impact: 'GitOps synchronization halted — all downstream clusters stop receiving updates',
      since: checked_at,
      runbook: 'https://wiki.example.com/runbooks/fleet-recovery',
      affectedClusters: 'all',
    });

    await createIncident({
      service: 'gitops-fleet',
      title: 'Rancher Fleet manager unreachable',
      severity: 'SEV1',
    });

  } else if (status === 'up') {
    console.info('[FLEET] Rancher Fleet recovered', { monitor: monitor_name });
    await resolveAlert(monitor_name);
    await notifyPlatformTeam(
      'Fleet has recovered — verifying GitRepo sync catchup for all workspaces'
    );
  }

  res.sendStatus(204);
});

Vigilmon sends this payload on status change:

{
  "monitor_id": "mon_fleet",
  "monitor_name": "Fleet — Manager health",
  "status": "down",
  "url": "http://fleet-mgmt.internal:8098/health",
  "checked_at": "2026-07-02T16:00:00Z",
  "response_code": 503,
  "response_time_ms": 15003
}

Part 7: SSL monitoring for Fleet management endpoints

Fleet agents authenticate to the management cluster over HTTPS. An expired certificate breaks agent registration and prevents new clusters from joining.

1. In Vigilmon, click Add Monitor.
2. Choose SSL monitor.
3. Enter: fleet-management.example.com
4. Set alert threshold to 21 days before expiry — Fleet agent reconnection after a certificate rotation requires manually re-registering agents on affected clusters.
5. Add your alert channel.

If you use Rancher Manager with an embedded cert-manager, also monitor the Rancher UI certificate separately:

• rancher.example.com — Rancher UI and Fleet API gateway

Summary

Your Rancher Fleet deployment now has layered monitoring:

1. Fleet manager pod health endpoint — confirms the central controller is ready, polled every 2 minutes by Vigilmon.
2. GitRepo sync staleness detection — flags repositories that have not synced within the configured threshold, catching silent drift accumulation.
3. Bundle not-ready tracking — surfaces downstream clusters where deployed workloads are failing to apply.
4. Critical GitRepo freshness check — tighter per-repository monitoring for time-sensitive workloads like security policies.
5. Webhook escalation with SEV1 incident creation — triggers immediate platform team response and incident tracking when the Fleet control plane goes down.
6. SSL monitors with extended lead time — alerts 21 days before expiry to account for the complexity of Fleet agent re-registration after certificate rotation.

Vigilmon handles check scheduling, multi-region polling, and alert routing. When Fleet's GitOps synchronization stalls, your platform team knows within 2 minutes — before a missed security patch or a stalled rollout becomes a production incident.

Monitor your Rancher Fleet GitOps platform free at vigilmon.online

#rancherfleet #gitops #kubernetes #multicluster #platformengineering #monitoring