tutorial

How to Monitor Popeye with Vigilmon

Popeye is a Kubernetes cluster sanitizer that scans a live cluster and reports potential issues with resource configurations. It inspects namespaces, pods, d...

Popeye is a Kubernetes cluster sanitizer that scans a live cluster and reports potential issues with resource configurations. It inspects namespaces, pods, deployments, services, ConfigMaps, and dozens of other resource types, then produces a graded report of misconfigurations, missing resource limits, deprecated API versions, and orphaned resources. Many teams run Popeye in CI pipelines or as a periodic cluster health job to catch drift before it causes production problems.

But when Popeye itself becomes unavailable — whether as a CLI run via a CronJob or as a continually-running scan server — your configuration-hygiene visibility disappears. In this tutorial you'll set up uptime and response-time monitoring for Popeye using Vigilmon — free tier, no credit card required.

Why Popeye needs external monitoring

Popeye failures expose gaps that internal cluster health checks miss:

  • CronJob failure — if Popeye runs as a Kubernetes CronJob, a failed job leaves no evidence in standard cluster dashboards; the last successful scan report may be hours or days old while misconfigurations accumulate
  • Popeye HTTP server down — when deployed as a continuously running server with its built-in HTTP endpoint, a crash means the /sanitize and /metrics paths return nothing while the pod appears Running
  • Report export failure — Popeye can publish scan results to S3 or a push gateway; a broken export pipeline produces no output and no alert without external probing
  • Scan taking too long — a cluster with thousands of resources can cause Popeye scans to time out; a slow-responding scan endpoint indicates a backlog or resource exhaustion in the scan pod
  • RBAC permission revocation — Popeye needs broad read access to the cluster; if a ClusterRoleBinding is removed, subsequent scans fail silently with empty reports

External monitoring from Vigilmon watches Popeye's HTTP endpoints from outside the cluster, giving you an independent signal that scan results are being produced and delivered.

What you'll need

  • Popeye accessible via a Kubernetes Service and Ingress, or run as a CronJob with HTTP result delivery
  • A free Vigilmon account

Step 1: Deploy Popeye and expose its scan endpoint

Run Popeye as a CronJob that publishes scan results to an HTTP endpoint, or use its built-in spinach server mode. The simplest pattern is a CronJob that runs a scan and pushes results to a status endpoint you control:

# popeye-cronjob.yaml
apiVersion: batch/v1
kind: CronJob
metadata:
  name: popeye-scan
  namespace: monitoring
spec:
  schedule: "*/15 * * * *"
  jobTemplate:
    spec:
      template:
        spec:
          serviceAccountName: popeye
          containers:
            - name: popeye
              image: derailed/popeye:latest
              args:
                - --save
                - --output-file=/tmp/report.json
                - --out=json
              resources:
                requests:
                  cpu: 200m
                  memory: 256Mi
                limits:
                  cpu: 500m
                  memory: 512Mi
          restartPolicy: Never

For continuous HTTP exposure, deploy Popeye with its spinach configuration:

# popeye-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: popeye
  namespace: monitoring
spec:
  replicas: 1
  selector:
    matchLabels:
      app: popeye
  template:
    metadata:
      labels:
        app: popeye
    spec:
      serviceAccountName: popeye
      containers:
        - name: popeye
          image: derailed/popeye:latest
          args: ["--spinach", "/etc/popeye/spinach.yml"]
          ports:
            - containerPort: 9090
          readinessProbe:
            httpGet:
              path: /healthz
              port: 9090
            initialDelaySeconds: 5
            periodSeconds: 10

Expose via Ingress:

# popeye-ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: popeye-ingress
  namespace: monitoring
spec:
  ingressClassName: nginx
  rules:
    - host: cluster-health.yourdomain.com
      http:
        paths:
          - path: /popeye
            pathType: Prefix
            backend:
              service:
                name: popeye
                port:
                  number: 9090

kubectl apply -f popeye-deployment.yaml
kubectl apply -f popeye-ingress.yaml

# Verify endpoints
curl https://cluster-health.yourdomain.com/popeye/healthz
# OK

curl https://cluster-health.yourdomain.com/popeye/sanitize | head -20

Step 2: Understand Popeye's health indicators

Popeye exposes several indicators you can probe:

  • /healthz — returns OK when the server is running; failure means the scan process has crashed
  • /sanitize — returns the latest scan report as JSON; a slow response indicates scan backlog; a 5xx response indicates scan failure
  • /metrics — Prometheus metrics including scan duration and issue counts per severity level

A healthy /sanitize response includes a score field and an issues map:

{
  "score": 82,
  "issues": {
    "default": [
      {"group": "__root__", "gvr": "v1/pods", "level": 2, "message": "CPU limit missing"}
    ]
  }
}

A score drop over time indicates accumulating misconfigurations. A missing or malformed response indicates Popeye itself has failed.

Step 3: Set up HTTP monitoring in Vigilmon

With Popeye endpoints accessible, add them to Vigilmon:

  1. Log in to vigilmon.online and go to Monitors → New Monitor
  2. Choose HTTP / HTTPS
  3. Add a monitor for each endpoint:

| Monitor name | URL | Expected status | |---|---|---| | Popeye health | https://cluster-health.yourdomain.com/popeye/healthz | 200 | | Popeye scan report | https://cluster-health.yourdomain.com/popeye/sanitize | 200 | | Popeye metrics | https://cluster-health.yourdomain.com/popeye/metrics | 200 |

  1. Set the check interval to 1 minute
  2. For the health monitor, under Expected response, match OK in the response body
  3. For the scan report monitor, match score in the response body to confirm a valid report is being returned
  4. For the metrics monitor, optionally match popeye_ in the response body to confirm Prometheus metrics are present
  5. Save each monitor

Vigilmon probes from multiple geographic regions, giving you an external perspective on Popeye availability that complements internal cluster alerting.

Step 4: Monitor the Popeye TCP port

Add a TCP-layer monitor to catch port-level failures independently of HTTP content:

  1. Go to Monitors → New Monitor
  2. Choose TCP Port
  3. Enter your Popeye hostname and port 9090
  4. Save the monitor

A TCP failure when the Popeye pod is Running typically indicates a Service selector mismatch, a NetworkPolicy blocking external access, or a crash that the readiness probe has not yet detected.

Step 5: Configure alert channels

Popeye failures mean your cluster sanitization coverage has lapsed. Configure alerts so you know immediately when scan results stop being produced.

Email alerts

  1. Go to Alert Channels → Add Channel → Email
  2. Enter your platform engineering or SRE on-call address
  3. Assign the channel to all Popeye monitors

Webhook alerts for incident routing

  1. Go to Alert Channels → Add Channel → Webhook
  2. Enter your incident management webhook URL (PagerDuty, Opsgenie, Slack)
  3. The payload Vigilmon sends:
{
  "monitor_name": "Popeye scan report",
  "status": "down",
  "url": "https://cluster-health.yourdomain.com/popeye/sanitize",
  "started_at": "2024-01-15T10:23:00Z",
  "duration_seconds": 300
}

Wire this alert to a runbook that checks CronJob history, verifies the Popeye service account RBAC bindings, and restarts the scan deployment if needed.

Step 6: Correlate Vigilmon alerts with Popeye diagnostics

When you receive a Popeye downtime alert, run these checks:

# 1. Check Popeye pod status
kubectl get pods -n monitoring -l app=popeye

# 2. Check pod logs for RBAC or scan errors
kubectl logs -n monitoring -l app=popeye --tail=100

# 3. Verify the service account has necessary ClusterRole binding
kubectl get clusterrolebinding | grep popeye
kubectl describe clusterrolebinding popeye

# 4. Check recent CronJob runs if using CronJob mode
kubectl get jobs -n monitoring | grep popeye
kubectl describe job <latest-popeye-job> -n monitoring

# 5. Check for OOM kills or resource exhaustion
kubectl describe pod -n monitoring -l app=popeye | grep -A 5 "OOMKilled\|Reason\|Exit Code"

# 6. Run a manual scan to verify RBAC works
kubectl run popeye-debug --rm -it \
  --image=derailed/popeye:latest \
  --restart=Never \
  --serviceaccount=popeye \
  -n monitoring \
  -- popeye --out=json -n default 2>&1 | head -30

# 7. Check if API server rate limiting is affecting scans
kubectl logs -n monitoring -l app=popeye | grep -i "rate limit\|too many requests\|429"

If Vigilmon shows the health endpoint down but the pod is Running, the Popeye HTTP server has crashed internally — check for a nil pointer panic in the logs. If /sanitize returns 200 but the body lacks a score field, the scan itself failed while the HTTP server survived.

Step 7: Create a status page for cluster configuration health

Make Popeye's availability visible to engineering leadership without requiring cluster access:

  1. Go to Status Pages → New Status Page
  2. Name it: "Cluster Configuration Health"
  3. Add your monitors: Popeye health, scan report, metrics
  4. Share the URL with platform engineering, SRE, and engineering management

Summary

| What you set up | What it catches | |---|---| | HTTP monitor on /healthz | Popeye server crash, pod failure | | HTTP monitor on /sanitize | Scan failure, RBAC revocation, report generation errors | | HTTP monitor on /metrics | Prometheus metrics unavailable | | TCP monitor on port 9090 | NetworkPolicy blocks, Service selector drift | | Email + webhook alert channels | Immediate notification when scan coverage lapses | | Status page | Visibility into cluster configuration health monitoring |

Popeye continuously scans your cluster for configuration issues that accumulate silently over time. External monitoring ensures that Popeye itself is always running and producing reports so your sanitization coverage never lapses without your knowledge.

Get started at vigilmon.online — free tier, no credit card required.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles