How to Monitor Reloader with Vigilmon

Reloader is a Kubernetes controller that watches ConfigMaps and Secrets for changes and automatically triggers rolling restarts on Deployments, DaemonSets, and StatefulSets that use them. Instead of manually rolling pods after a config update or secret rotation, teams annotate their workloads with reloader.stakater.com/auto: "true" and Reloader handles the rest.

But Reloader is operational infrastructure: when it fails, config changes and secret rotations are silently ignored. Pods keep running with stale configuration until someone notices something is wrong — which often happens during an incident, not during normal operations. In this tutorial you'll set up uptime and response-time monitoring for your Reloader deployment using Vigilmon — free tier, no credit card required.

Why Reloader needs external monitoring

Reloader's failure modes are invisible until it's too late:

• Controller pod crashes — Reloader stops watching ConfigMaps and Secrets; the next config update or secret rotation lands silently with no pod restart
• RBAC permission revoked — a cluster policy change removes Reloader's permission to watch resources or patch Deployments; updates are silently dropped
• Leader election lost — in high-availability deployments, all replicas fail to elect a leader and no rolling restarts are triggered
• Webhook annotations not propagated — Reloader relies on resource annotations; if an operator or GitOps tool overwrites annotations, entire workloads fall out of the auto-reload watch list without any warning
• Secret rotation happens but no restart triggers — a secret rotates successfully in Vault or Kubernetes, but Reloader fails to detect the change; applications keep using the expired credential

External monitoring from Vigilmon catches these before a config update sits undelivered for hours.

What you'll need

• Reloader installed (helm upgrade --install reloader stakater/reloader --namespace reloader)
• A Deployment annotated with reloader.stakater.com/auto: "true" for end-to-end testing
• A free Vigilmon account

Step 1: Expose a health endpoint for Reloader

Reloader exposes a health endpoint on port 9090 by default. Expose it externally so Vigilmon can probe it:

# Verify Reloader is running
kubectl get pods -n reloader -l app.kubernetes.io/name=reloader-reloader
# NAME                                   READY   STATUS    RESTARTS
# reloader-reloader-5f6d8b9c4-hqk8j      1/1     Running   0

# Check the health endpoint
kubectl port-forward -n reloader deploy/reloader-reloader 9090:9090 &
curl http://localhost:9090/metrics | grep reloader
# reloader_reload_executed_total{...} 42

Expose via a LoadBalancer Service:

# reloader-health-service.yaml
apiVersion: v1
kind: Service
metadata:
  name: reloader-health
  namespace: reloader
spec:
  type: LoadBalancer
  selector:
    app.kubernetes.io/name: reloader-reloader
  ports:
    - name: http
      port: 9090
      targetPort: 9090

kubectl apply -f reloader-health-service.yaml
kubectl get svc reloader-health -n reloader
# NAME               TYPE           EXTERNAL-IP     PORT(S)
# reloader-health    LoadBalancer   203.0.113.65    9090:31800/TCP

Reloader's metrics endpoint also serves as its health check — if it returns any response, the controller is alive. You can create a dedicated /healthz endpoint via a sidecar if you prefer a clean 200:

# In the Reloader deployment, add a sidecar
- name: healthz
  image: node:20-alpine
  command:
    - node
    - -e
    - |
      const http = require('http');
      const https = require('https');
      http.createServer((req, res) => {
        if (req.url !== '/healthz') { res.writeHead(404); res.end(); return; }
        http.get('http://localhost:9090/metrics', r => {
          const ok = r.statusCode === 200;
          res.writeHead(ok ? 200 : 503, {'Content-Type':'application/json'});
          res.end(JSON.stringify({ status: ok ? 'ok' : 'error', reloader: r.statusCode }));
        }).on('error', e => {
          res.writeHead(503, {'Content-Type':'application/json'});
          res.end(JSON.stringify({ status: 'error', message: e.message }));
        });
      }).listen(8080);
  ports:
    - containerPort: 8080

Step 2: Add an end-to-end reload verification probe

The health endpoint tells you Reloader is running — but not that it's actually reloading. Add a canary check that updates a ConfigMap and verifies a deployment restarted:

# canary-configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: reloader-canary
  namespace: monitoring
data:
  version: "1"
  updated_at: "2024-01-01T00:00:00Z"
---
# canary-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: reloader-canary-app
  namespace: monitoring
  annotations:
    reloader.stakater.com/auto: "true"
spec:
  replicas: 1
  selector:
    matchLabels:
      app: reloader-canary-app
  template:
    metadata:
      labels:
        app: reloader-canary-app
    spec:
      containers:
        - name: app
          image: node:20-alpine
          command:
            - node
            - -e
            - |
              const http = require('http');
              const fs = require('fs');
              http.createServer((req, res) => {
                if (req.url !== '/health') { res.writeHead(404); res.end(); return; }
                const version = fs.readFileSync('/config/version', 'utf8').trim();
                res.writeHead(200, {'Content-Type':'application/json'});
                res.end(JSON.stringify({
                  status: 'ok',
                  config_version: version,
                  started_at: process.env.START_TIME
                }));
              }).listen(8080);
          env:
            - name: START_TIME
              value: ""  # Reloader will restart this pod; track via pod start time
          volumeMounts:
            - name: config
              mountPath: /config
          ports:
            - containerPort: 8080
      volumes:
        - name: config
          configMap:
            name: reloader-canary

kubectl apply -f canary-configmap.yaml canary-deployment.yaml

# Expose the canary app
kubectl expose deploy reloader-canary-app -n monitoring \
  --type=LoadBalancer --port=8080
kubectl get svc reloader-canary-app -n monitoring
# EXTERNAL-IP: 203.0.113.66

# Test the canary app health
curl http://203.0.113.66:8080/health
# {"status":"ok","config_version":"1","started_at":""}

# Update the ConfigMap — Reloader should restart the pod
kubectl patch configmap reloader-canary -n monitoring \
  --patch '{"data":{"version":"2","updated_at":"2024-01-15T10:00:00Z"}}'

# Within 30 seconds, the pod should have restarted
kubectl get pods -n monitoring -l app=reloader-canary-app -w

Vigilmon monitors the canary app's /health endpoint — if it goes down briefly, it indicates a Reloader-triggered restart occurred. If the config version in the response never updates despite ConfigMap changes, Reloader has silently stopped working.

Step 3: Set up HTTP monitoring in Vigilmon

With your endpoints ready, add them to Vigilmon:

1. Log in to vigilmon.online and go to Monitors → New Monitor
2. Choose HTTP / HTTPS
3. Add a monitor for each endpoint:

| Monitor name | URL | Expected status | |---|---|---| | Reloader Controller Health | http://203.0.113.65:9090/metrics | 200 | | Reloader Canary App | http://203.0.113.66:8080/health | 200 | | Reloader Healthz Sidecar | http://203.0.113.65:8080/healthz | 200 |

4. Set the check interval to 1 minute
5. Under Expected response, set status code 200
6. For the canary app, optionally match "status":"ok" in the response body
7. Save each monitor

Vigilmon will immediately surface when the controller crashes. The canary app monitor provides a second signal — if it goes persistently unreachable (not just a brief restart), something deeper is wrong.

Step 4: Monitor the TCP layer

If Reloader is deployed with leader election or a webhook admission controller, add a TCP monitor:

1. Go to Monitors → New Monitor
2. Choose TCP Port
3. Enter 203.0.113.65 and port 9090
4. Save the monitor

A TCP check catches cases where the Reloader process is stuck and not responding to HTTP but the port is still bound — distinct from a pod restart.

Step 5: Configure alert channels

When Reloader goes down, config and secret updates silently stop reaching running pods. This is the kind of failure that only surfaces during an incident, when you realize the config change you pushed three hours ago was never applied.

Email alerts

1. Go to Alert Channels → Add Channel → Email
2. Enter your platform engineering team's on-call address
3. Assign the channel to all Reloader monitors

Webhook alerts for GitOps pipelines

1. Go to Alert Channels → Add Channel → Webhook
2. Use a webhook URL that routes to Slack or PagerDuty
3. The payload Vigilmon sends:

{
  "monitor_name": "Reloader Controller Health",
  "status": "down",
  "url": "http://203.0.113.65:9090/metrics",
  "started_at": "2024-01-15T18:30:00Z",
  "duration_seconds": 90
}

Wire this into an alert that warns your GitOps pipeline not to push config-only changes while Reloader is down — since those changes will land but no pods will restart to pick them up.

Step 6: Correlate Vigilmon alerts with Reloader diagnostics

When you receive a Reloader downtime alert, check these in order:

# 1. Check the Reloader pod
kubectl get pods -n reloader -l app.kubernetes.io/name=reloader-reloader
kubectl describe pod -n reloader -l app.kubernetes.io/name=reloader-reloader

# 2. Check Reloader logs for watch or restart errors
kubectl logs -n reloader deploy/reloader-reloader --tail=100 \
  | grep -i "error\|fail\|watch\|reload"

# 3. Verify Reloader has the required RBAC permissions
kubectl auth can-i watch configmaps --as=system:serviceaccount:reloader:reloader-reloader -n default
kubectl auth can-i patch deployments --as=system:serviceaccount:reloader:reloader-reloader -n default

# 4. Check if Reloader is tracking your annotated workloads
kubectl get deployments --all-namespaces \
  -o jsonpath='{range .items[?(@.metadata.annotations.reloader\.stakater\.com/auto=="true")]}{.metadata.namespace}/{.metadata.name}{"\n"}{end}'

# 5. Check recent reload events in the cluster
kubectl get events --all-namespaces --field-selector reason=SuccessfulCreate \
  | grep reloader

# 6. Verify leader election state (if HA deployment)
kubectl get lease -n reloader
kubectl describe lease reloader -n reloader

# 7. Check Reloader metrics for reload count trends
curl http://203.0.113.65:9090/metrics | grep reloader_reload
# reloader_reload_executed_total{...} 42
# reloader_reload_failed_total{...} 0

If the Reloader pod shows Running but the reload count isn't incrementing after ConfigMap changes, check RBAC permissions and annotation coverage on your workloads.

Step 7: Create a status page for your configuration delivery layer

Config and secret delivery is as critical as the CI/CD pipeline that produces those configs. A status page makes the delivery layer visible:

1. Go to Status Pages → New Status Page
2. Name it: "Config Delivery - Reloader"
3. Add your monitors: Reloader Controller Health, Reloader Canary App
4. Share the URL with your platform and application teams

When a config change isn't taking effect, teams can check the status page to determine whether Reloader is healthy before assuming an application bug.

Summary

| What you set up | What it catches | |---|---| | HTTP monitor on Reloader metrics | Controller crashes, pod restarts | | HTTP monitor on canary app | End-to-end reload verification | | TCP monitor on metrics port | Port-level failures, stuck processes | | Email + webhook alert channels | Immediate notification before stale configs cause incidents | | Status page | Team self-service when config changes don't take effect |

The goal is to make your configuration delivery layer as observable as the applications it updates. When Reloader stops reloading, you want an alert before a config change silently fails to reach your running pods — not hours later during an incident.

Get started at vigilmon.online — free tier, no credit card required.