Monitoring Flagger with Vigilmon: Canary Analysis, Rollout Health, Prometheus Metrics & Promotion Alerts

Flagger is the Kubernetes progressive delivery operator that automates canary releases, A/B tests, and blue-green deployments by shifting traffic incrementally and validating success metrics before promoting. When Flagger's controller stops reconciling, canary rollouts freeze mid-promotion and traffic splits stall indefinitely. When a canary deployment fails its analysis phase silently, production receives a bad build without any automated rollback. When Prometheus is unreachable, Flagger cannot evaluate any metric templates and will either block all future promotions or enter an undefined state depending on your configuration. Vigilmon gives you external visibility into Flagger's health and the canary endpoints it manages, so you detect promotion failures and controller outages before your users do.

What You'll Build

• An HTTP monitor on Flagger's webhook validation endpoint to detect controller failures
• Canary endpoint availability checks that confirm each deployment phase serves traffic
• Prometheus metric threshold alerting for canary analysis pass/fail rates
• SSL certificate monitoring for canary-exposed ingress hosts
• An alerting runbook that maps Flagger failure modes to Vigilmon monitor states

Prerequisites

• A Kubernetes cluster running Flagger 1.x (installed via Helm or Flux)
• At least one Canary custom resource deployed and managed by Flagger
• Prometheus installed and scraping Flagger metrics (standard with the Flagger Helm chart)
• Ingress controller (NGINX, Istio, or Contour) exposing canary services externally
• A free account at vigilmon.online

Step 1: Understand Flagger's Health Surface

Flagger exposes a health endpoint and Prometheus metrics that together tell you whether the controller is alive and whether canary analysis is succeeding. The controller's leader election health check is available at:

kubectl -n flagger-system port-forward deployment/flagger 8080:8080 &
curl http://localhost:8080/healthz
# Returns: OK

For external monitoring, Flagger's behavior is most usefully observed through the canary services it creates. For every Canary resource named myapp in namespace production, Flagger creates:

• myapp — the primary stable service (receives ~100% traffic in steady state)
• myapp-canary — the canary service (receives shifted traffic during analysis)
• myapp-primary — the underlying primary deployment service

The canary service is the right target for external health checks during a rollout window.

Flagger also emits Prometheus metrics that reveal analysis health:

# Check Flagger controller readiness
kubectl -n flagger-system get pods -l app.kubernetes.io/name=flagger

# View canary status
kubectl -n production get canary myapp -o jsonpath='{.status.phase}'
# Returns: Initialized | Progressing | Promoting | Finalising | Succeeded | Failed

Step 2: Monitor the Primary Service Endpoint

The primary service is the stable production endpoint. It should always be healthy regardless of whether a canary rollout is in progress. Set up a baseline monitor:

1. Log in to Vigilmon → Add Monitor → HTTP.
2. URL: https://myapp.example.com/healthz (your primary app's health endpoint).
3. Check interval: 60 seconds.
4. Response timeout: 10 seconds.
5. Expected status: 200.
6. Keyword: your expected health response (e.g., ok, healthy, or a JSON field like "status":"up").
7. Label: myapp primary (Flagger stable).
8. Click Save.

This monitor catches:

• Flagger rollback failures that leave the primary in a degraded state
• Kubernetes node failures that take down primary replicas
• Configuration drift between Flagger's primary deployment and the original spec
• Ingress controller failures that sever external access to the primary service

Alert after: 1 consecutive failure. The primary service should never be down during a Flagger-managed deployment.

Step 3: Monitor the Canary Service During Rollouts

The canary service receives traffic only during an active analysis window. Monitoring it tells you whether Flagger's traffic split is working and whether the new build is serving requests at all:

# Inspect what Flagger creates for your canary service
kubectl -n production get svc myapp-canary -o yaml | grep -A5 "selector:"

1. Add Monitor → HTTP.
2. URL: https://myapp-canary.example.com/healthz (if your ingress exposes the canary host separately).
3. Check interval: 30 seconds.
4. Response timeout: 10 seconds.
5. Expected status: 200.
6. Label: myapp canary (Flagger analysis).
7. Click Save.

Ingress header-based routing: Some Flagger configurations expose the canary service via a header (X-Canary: true) rather than a separate hostname. In that case, use Vigilmon's custom header support to add X-Canary: true to the HTTP monitor request, directing it to the canary backend.

When this monitor fires during an active rollout, it means the canary build is returning errors — Flagger's analysis phase will likely catch this too, but the Vigilmon alert arrives with external network context that Prometheus metrics may not capture (e.g., DNS resolution failures or TLS handshake errors on the canary ingress).

Step 4: Monitor Prometheus for Canary Metric Availability

Flagger relies on Prometheus to evaluate metric templates during analysis. If Prometheus is unreachable, Flagger cannot determine whether the canary is healthy. Monitor Prometheus to catch this dependency failure early:

# Verify Prometheus is scraping Flagger metrics
curl http://prometheus.monitoring.svc.cluster.local:9090/api/v1/query \
  --data-urlencode 'query=flagger_canary_status{name="myapp",namespace="production"}'

For external monitoring of Prometheus (if exposed via ingress):

1. Add Monitor → HTTP.
2. URL: https://prometheus.example.com/-/healthy.
3. Check interval: 2 minutes.
4. Response timeout: 10 seconds.
5. Expected status: 200.
6. Keyword: Prometheus Server is Healthy.
7. Label: Prometheus (Flagger dependency).
8. Click Save.

Flagger-specific metrics to watch in Prometheus directly:

# Canary weight (traffic percentage currently on canary)
flagger_canary_weight{name="myapp", namespace="production"}

# Whether the canary is in a failed state (1 = failed)
flagger_canary_status{name="myapp", namespace="production", phase="Failed"}

# Number of metric analysis iterations
flagger_canary_iterations_total{name="myapp", namespace="production"}

Step 5: Configure Webhook Endpoint Monitoring

Flagger supports pre-rollout, rollout, and post-rollout webhooks for custom validation. These webhooks are HTTP endpoints that Flagger calls during analysis — if they're unreachable or returning errors, Flagger will fail the canary promotion. Monitor your webhook endpoints to catch this failure mode:

# Example Flagger Canary webhook configuration
analysis:
  webhooks:
    - name: load-test
      url: http://flagger-loadtester.test/
      timeout: 5s
    - name: acceptance-test
      url: http://myapp-acceptance.test/api/check
      timeout: 30s
      type: pre-rollout

For each external webhook endpoint:

1. Add Monitor → HTTP.
2. URL: http://flagger-loadtester.example.com/ (your webhook's health endpoint).
3. Check interval: 2 minutes.
4. Response timeout: 10 seconds.
5. Expected status: 200.
6. Label: Flagger load-tester webhook.
7. Click Save.

Step 6: SSL Certificate Monitoring for Canary Hosts

Flagger-managed canary deployments often use separate ingress hostnames (e.g., myapp-canary.example.com). Certificate expiry on the canary host will cause analysis failures because Flagger's internal metric collection uses HTTPS:

1. Add Monitor → SSL Certificate.
2. Domain: myapp.example.com.
3. Alert when expiry is within: 30 days.
4. Alert again: 14 days, 7 days, 3 days.
5. Click Save.

Add a second SSL monitor for the canary hostname if it differs from the primary.

Step 7: Configure Alerting for Promotion Failures

In Vigilmon under Settings → Notifications, configure your alert channels and map them to the correct response runbook:

| Monitor | Trigger | Immediate action | |---|---|---| | Primary service /healthz | Non-200 or keyword missing | Check kubectl get canary -A; look for Failed phase; consider manual rollback | | Canary service /healthz | Non-200 during rollout | New build returning errors; Flagger should auto-rollback; verify with kubectl describe canary myapp | | Prometheus | Non-200 or keyword missing | Flagger cannot evaluate metrics; all active canaries will stall; restore Prometheus immediately | | Webhook endpoint | Non-200 | Pre/post rollout validation broken; future promotions will fail at webhook step | | SSL certificate | < 30 days to expiry | Renew certificate; check cert-manager; Flagger metric collection over HTTPS will fail on expiry |

Alert grouping: Group the primary and canary monitors under a myapp-flagger monitor group so a single deployment failure generates one aggregated alert rather than multiple simultaneous notifications.

Common Flagger Failure Modes and What Vigilmon Catches

| Scenario | Vigilmon signal | |---|---| | Flagger controller pod crash-loops | Primary monitor may still pass (deployment untouched); canary stalls mid-rollout | | New build returns 5xx errors | Canary monitor fires within 30 s; Flagger analysis also catches and rolls back | | Prometheus unreachable during analysis | Prometheus monitor fires; canary stalls pending metric evaluation | | Webhook endpoint down | Webhook monitor fires; promotions blocked at pre-rollout phase | | Canary ingress TLS error | SSL monitor fires; Flagger HTTPS metric collection fails silently | | Primary deployment OOM killed by Kubernetes | Primary health monitor fires immediately | | Canary traffic weight never reaches 100% | Not detectable externally; check flagger_canary_weight in Prometheus directly | | DNS misconfiguration on canary host | Canary monitor fires with DNS error; primary unaffected | | Rollback leaves canary service misconfigured | Both primary and canary monitors may show degraded state | | Node pool scaling removes canary replicas | Canary monitor fires; Flagger may retry analysis with insufficient replicas |

Flagger automates the hardest part of progressive delivery — but when the controller stalls, Prometheus goes dark, or a canary build silently fails, the consequences are felt across every future deployment. Vigilmon gives you external eyes on every layer: the primary service your users depend on, the canary endpoint during analysis windows, the Prometheus dependency that powers metric evaluation, and the SSL certificates protecting every HTTPS check. When Flagger can't tell you something is wrong, Vigilmon can.

Start monitoring your Flagger deployments in under 5 minutes — register free at vigilmon.online.