tutorial

Monitoring Argo Rollouts with Vigilmon: Controller Health, Rollout Phase Checks, and Analysis Provider Availability

How to monitor Argo Rollouts with Vigilmon — controller availability, rollout phase progress tracking, analysis provider reachability, and rollback rate alerting for progressive delivery pipelines.

How to Monitor Argo Rollouts with Vigilmon

Argo Rollouts brings progressive delivery to Kubernetes — canary deployments, blue/green switches, and automated analysis gates that promote or roll back based on real metrics. When it's working, deployments are safer than raw Kubernetes rollouts. When it's not, a stuck rollout can silently leave your application at an intermediate canary weight indefinitely, or worse, auto-promote a bad release because the analysis provider was unreachable.

This tutorial covers monitoring Argo Rollouts' health surfaces with Vigilmon: controller availability, rollout phase progress, analysis provider reachability, and rollback trigger rates.

Why Argo Rollouts needs external monitoring

The Rollouts controller manages the delivery lifecycle, but it can fail in ways Kubernetes doesn't surface:

  • Controller pod down — all rollout phase transitions stop; running rollouts freeze at current weight
  • AnalysisTemplate provider unreachable — Prometheus/Datadog/CloudWatch endpoints time out; CA defaults to Inconclusive or Failed depending on configuration — potentially triggering unintended rollbacks
  • Rollout stuck in Paused/Progressing — a canary that never promotes and never rolls back means half your traffic stays on the canary version forever
  • Rollback storm — multiple rollouts auto-rolling back simultaneously indicates a systemic metric or provider issue, not individual bad releases
  • Rollout controller API unreachable — the argo-rollouts Service endpoint is down; kubectl argo rollouts commands all fail

What you'll need

  • Kubernetes cluster with Argo Rollouts installed (v1.6+)
  • At least one active Rollout and AnalysisTemplate
  • A free Vigilmon account

Step 1: Expose the Rollouts controller metrics endpoint

Argo Rollouts exposes Prometheus metrics on port 8090 and a health endpoint on port 8080:

# Confirm controller is running
kubectl get pods -n argo-rollouts

# Check health endpoint in-cluster
kubectl exec -n argo-rollouts \
  $(kubectl get pod -n argo-rollouts -l app.kubernetes.io/name=argo-rollouts -o name | head -1) \
  -- wget -qO- http://localhost:8080/healthz

Expose it externally:

# rollouts-health-service.yaml
apiVersion: v1
kind: Service
metadata:
  name: argo-rollouts-health
  namespace: argo-rollouts
spec:
  selector:
    app.kubernetes.io/name: argo-rollouts
  ports:
    - name: health
      port: 8080
      targetPort: 8080
    - name: metrics
      port: 8090
      targetPort: 8090
  type: LoadBalancer

kubectl apply -f rollouts-health-service.yaml
export ROLLOUTS_IP=$(kubectl get svc argo-rollouts-health \
  -n argo-rollouts -o jsonpath='{.status.loadBalancer.ingress[0].ip}')

curl http://$ROLLOUTS_IP:8080/healthz
# OK

Step 2: Monitor rollout phase health

Write a rollout-phase checker that reads all Rollout resources and alerts when any is stuck in a non-terminal state beyond a threshold:

# rollout_health_server.py
import subprocess, json, time, http.server, threading
from datetime import datetime, timezone

STUCK_THRESHOLD = 600   # 10 minutes stuck in Progressing/Paused = alert
CHECK_INTERVAL = 30

state = {"stuck": [], "degraded": [], "last_check": 0}

def check_rollouts():
    while True:
        try:
            result = subprocess.run(
                ["kubectl", "get", "rollouts", "--all-namespaces", "-o", "json"],
                capture_output=True, text=True, timeout=15
            )
            rollouts = json.loads(result.stdout).get("items", [])
            stuck, degraded = [], []
            now = datetime.now(timezone.utc)

            for r in rollouts:
                meta = r.get("metadata", {})
                status = r.get("status", {})
                phase = status.get("phase", "")
                name = f"{meta.get('namespace')}/{meta.get('name')}"

                if phase == "Degraded":
                    degraded.append({"name": name, "phase": phase,
                                     "message": status.get("message", "")})
                elif phase in ("Progressing", "Paused"):
                    ts_str = meta.get("creationTimestamp", "")
                    # Use last transition time from conditions
                    conditions = status.get("conditions", [])
                    prog_cond = next(
                        (c for c in conditions if c.get("type") == "Progressing"), None
                    )
                    if prog_cond and prog_cond.get("lastUpdateTime"):
                        ts = datetime.fromisoformat(
                            prog_cond["lastUpdateTime"].replace("Z", "+00:00")
                        )
                        age = (now - ts).total_seconds()
                        if age > STUCK_THRESHOLD:
                            stuck.append({"name": name, "phase": phase,
                                          "stuck_seconds": int(age)})

            state["stuck"] = stuck
            state["degraded"] = degraded
            state["last_check"] = time.time()
        except Exception as e:
            state["stuck"] = [{"error": str(e)}]
        time.sleep(CHECK_INTERVAL)

class Handler(http.server.BaseHTTPRequestHandler):
    def do_GET(self):
        if self.path != "/healthz":
            self.send_response(404); self.end_headers(); return
        body = json.dumps(state).encode()
        if state["stuck"] or state["degraded"]:
            self.send_response(503)
        else:
            self.send_response(200)
        self.send_header("Content-Type", "application/json")
        self.end_headers()
        self.wfile.write(body)
    def log_message(self, *args): pass

threading.Thread(target=check_rollouts, daemon=True).start()
http.server.HTTPServer(("", 9092), Handler).serve_forever()

Apply RBAC so this pod can read Rollout resources:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: rollout-reader
rules:
  - apiGroups: ["argoproj.io"]
    resources: ["rollouts", "analysisruns"]
    verbs: ["get", "list"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: rollout-health-reader
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: rollout-reader
subjects:
  - kind: ServiceAccount
    name: rollout-health-sa
    namespace: argo-rollouts

Step 3: Monitor your analysis providers

AnalysisTemplates pull metrics from external providers. Monitor those endpoints directly:

# If using Prometheus as analysis provider
curl -s http://prometheus.monitoring.svc:9090/-/ready
# OK

# If using Datadog
curl -s "https://api.datadoghq.com/api/v1/validate" \
  -H "DD-API-KEY: $DATADOG_API_KEY" | jq .valid

For each provider your AnalysisTemplates use, create a Vigilmon HTTP monitor pointing at the provider's health/validate endpoint. If the provider is unreachable, your analysis gates will fail — and depending on your inconclusiveLimit settings, rollouts may stall or auto-rollback.

Step 4: Track rollback rates

Rollbacks should be rare. Frequent rollbacks mean either your analysis thresholds are too tight or releases are consistently bad. Monitor rollback count via Prometheus:

# Rollback events in the last hour
increase(rollout_info{phase="Degraded"}[1h])

# Analysis runs that failed (leading to rollbacks)
increase(analysis_run_info{phase="Failed"}[1h])

# Current canary weight across all rollouts
rollout_info{strategy="canary"}

Create a Prometheus recording rule and expose an alerting endpoint that Vigilmon can poll:

# prometheus-rules.yaml
groups:
  - name: argo-rollouts
    rules:
      - alert: RollbackStorm
        expr: increase(rollout_info{phase="Degraded"}[30m]) > 3
        for: 0m
        labels:
          severity: critical
        annotations:
          summary: "Multiple rollouts degraded in 30 minutes"

Step 5: Set up Vigilmon monitors

Monitor 1 — Controller health

| Field | Value | |---|---| | Type | HTTP | | URL | http://<ROLLOUTS-IP>:8080/healthz | | Expected status | 200 | | Keyword check | OK | | Interval | 30 seconds | | Alert after | 1 failure |

Monitor 2 — Rollout phase health

| Field | Value | |---|---| | Type | HTTP | | URL | http://<rollout-health-IP>:9092/healthz | | Expected status | 200 | | Keyword check | "stuck":[] | | Interval | 60 seconds |

Monitor 3 — Analysis provider (Prometheus)

| Field | Value | |---|---| | Type | HTTP | | URL | https://prometheus.example.com/-/ready | | Expected status | 200 | | Interval | 60 seconds |

Monitor 4 — Rollouts metrics endpoint

| Field | Value | |---|---| | Type | HTTP | | URL | http://<ROLLOUTS-IP>:8090/metrics | | Expected status | 200 | | Keyword check | rollout_info | | Interval | 60 seconds |

Step 6: Configure alerts by severity

Controller pod down           → P1 immediately (all rollouts frozen)
Rollout stuck > 10 minutes    → P1 (canary traffic split unresolved)
Analysis provider unreachable → P1 (rollout gates inoperative)
Rollback rate > 3 in 30 min  → P1 (systemic release or metric issue)
Controller metrics slow       → P2 (controller may be under load)

What to monitor — quick reference

| Signal | Check | Alert | |---|---|---| | Controller alive | HTTP /healthz 200 | 1 failure = P1 | | Rollout phase | Custom HTTP 200 | Any stuck/degraded | | Analysis provider | Provider health URL 200 | 1 failure = P1 | | Rollback rate | Prometheus + HTTP gateway | >3 in 30 min | | Metrics scrape | HTTP /metrics 200 | 2 failures |

Vigilmon for progressive delivery pipelines

Argo Rollouts is designed to reduce deployment risk — but the delivery pipeline itself needs to be monitored. A frozen rollout controller or an unreachable analysis provider can leave your application in a partial state that's invisible to end users until they hit the canary pod. Vigilmon's external probes give you an independent signal from outside the cluster, closing the gap between internal Kubernetes health and real delivery pipeline health.

Start monitoring at vigilmon.online — free tier, no card required.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles