Monitoring Kube-green with Vigilmon: Suspension Health, Webhook Availability, Metrics Endpoint & SSL Certificate Alerts

Kube-green is the Kubernetes operator that suspends non-production workloads on a schedule — shutting down dev, staging, and preview environments outside business hours and waking them up again when engineers arrive. When the kube-green controller crashes, suspension schedules stop executing silently: development environments keep running overnight, burning cloud spend without anyone knowing. When the validating webhook becomes unavailable, new SleepInfo resource deployments fail and engineers get cryptic API server errors. Vigilmon gives you external visibility into the availability of kube-green's metrics endpoint, its webhook service, and the underlying controller health so you know the moment your cost-saving schedules stop working.

What You'll Build

• A monitor on kube-green's metrics endpoint to detect controller process failures
• A TCP check on the webhook service port to confirm admission webhooks are reachable
• SSL certificate monitoring for your Kubernetes API server and webhook TLS
• An alert runbook that maps Vigilmon findings to specific kubectl remediation commands

Prerequisites

• A kube-green installation (v0.5.0+) deployed via Helm or manifest in the kube-green namespace
• The kube-green metrics endpoint exposed on port 8080 (default)
• A Kubernetes cluster with an accessible API endpoint
• A free account at vigilmon.online

Step 1: Understand Kube-green's Observability Surface

Kube-green exposes several surfaces that indicate whether the controller is healthy and processing SleepInfo schedules:

# Check that the controller pod is running
kubectl get pods -n kube-green

# Inspect the controller logs for schedule execution
kubectl logs -n kube-green deployment/kube-green-controller-manager --tail=50

# List all SleepInfo resources and their current status
kubectl get sleepinfo -A

# Check the metrics endpoint (if port-forwarded or exposed)
kubectl port-forward -n kube-green svc/kube-green-controller-manager-metrics-service 8080:8080
curl http://localhost:8080/metrics | grep kube_green

The key metric to watch is kube_green_sleepinfo_scheduled_total — it increments each time a suspension or wake-up is executed. A stale counter means the controller has stopped processing schedules.

Kube-green also registers a validating webhook at /validate-kube-green-sigs-k8s-io-v1alpha1-sleepinfo. If this webhook service is unreachable, any attempt to create or update a SleepInfo resource produces an API server timeout.

Step 2: Expose the Kube-green Metrics Endpoint

By default, kube-green's metrics are only reachable within the cluster. To enable external monitoring via Vigilmon, expose the service through an Ingress or a NodePort:

# kube-green-metrics-ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: kube-green-metrics
  namespace: kube-green
  annotations:
    nginx.ingress.kubernetes.io/auth-type: basic
    nginx.ingress.kubernetes.io/auth-secret: kube-green-metrics-auth
spec:
  rules:
  - host: kube-green-metrics.example.com
    http:
      paths:
      - path: /metrics
        pathType: Prefix
        backend:
          service:
            name: kube-green-controller-manager-metrics-service
            port:
              number: 8080
  tls:
  - hosts:
    - kube-green-metrics.example.com
    secretName: kube-green-metrics-tls

Apply the Ingress and verify the endpoint returns Prometheus-format metrics:

kubectl apply -f kube-green-metrics-ingress.yaml
curl https://kube-green-metrics.example.com/metrics | grep controller_runtime

A healthy response includes controller_runtime_reconcile_total and process uptime metrics from the controller-runtime framework.

Step 3: Create a Vigilmon HTTP Monitor for the Metrics Endpoint

1. Log in to Vigilmon → Add Monitor → HTTP.
2. URL: https://kube-green-metrics.example.com/metrics.
3. Check interval: 2 minutes.
4. Response timeout: 15 seconds.
5. Expected status: 200.
6. Keyword: controller_runtime_reconcile_total (present in all healthy controller-runtime-based operators).
7. Label: kube-green controller metrics.
8. Click Save.

This monitor catches:

• Kube-green controller pod crashes or OOM kills that stop schedule processing
• Controller-manager deployment rollout failures after Helm upgrades
• Namespace-level network policy changes that isolate the controller
• Kubernetes API server disruptions that cause the controller to back off indefinitely
• Memory or CPU pressure that causes the kube-green pod to be evicted

Alert sensitivity: Set to trigger after 2 consecutive failures to avoid false positives during cluster maintenance windows.

Step 4: Monitor the Validating Webhook Service via TCP

Kube-green registers a validating webhook with the Kubernetes API server. The webhook service runs on port 9443 inside the cluster and is exposed on the same node as the controller. When the webhook service is down, any kubectl apply for a SleepInfo resource hangs for 30 seconds before failing with a timeout error.

If your webhook service is externally reachable (common in managed Kubernetes environments where you expose services via a load balancer):

# Verify the webhook service is accessible
kubectl get svc -n kube-green
# Look for kube-green-webhook-service on port 443

To expose the webhook port for external TCP monitoring:

# kube-green-webhook-lb.yaml
apiVersion: v1
kind: Service
metadata:
  name: kube-green-webhook-external
  namespace: kube-green
spec:
  type: LoadBalancer
  selector:
    control-plane: controller-manager
  ports:
  - port: 9443
    targetPort: 9443
    protocol: TCP

Then in Vigilmon:

1. Add Monitor → TCP.
2. Host: <load-balancer-ip-or-hostname>.
3. Port: 9443.
4. Check interval: 2 minutes.
5. Label: kube-green webhook service.
6. Click Save.

Why the webhook matters: Admission webhooks run synchronously in the Kubernetes API server request path. A failed webhook means kubectl apply -f sleepinfo.yaml blocks all engineers who try to update suspension schedules, even while the kube-green controller itself continues processing existing schedules.

Step 5: Monitor SSL Certificates

Kube-green uses TLS for its webhook endpoint, managed either by cert-manager or by a self-signed certificate injected via caBundle in the ValidatingWebhookConfiguration. The metrics Ingress you created in Step 2 also needs a valid certificate:

# Check the webhook certificate expiry
kubectl get secret -n kube-green kube-green-serving-cert -o jsonpath='{.data.tls\.crt}' | \
  base64 -d | openssl x509 -noout -dates

# Check the metrics Ingress certificate
openssl s_client -connect kube-green-metrics.example.com:443 2>/dev/null | \
  openssl x509 -noout -dates

In Vigilmon:

1. Add Monitor → SSL Certificate.
2. Domain: kube-green-metrics.example.com.
3. Alert when expiry is within: 30 days.
4. Alert again: 14 days, 7 days, 3 days, 1 day.
5. Click Save.

cert-manager rotation: If you use cert-manager to manage kube-green's webhook certificate, set the Certificate resource's renewBefore to at least 720h (30 days). cert-manager renewal failures are silent until the certificate actually expires — at which point all SleepInfo admission calls fail and the Kubernetes API server logs x509: certificate has expired errors.

Step 6: Verify Suspension Schedules Are Executing

The metrics endpoint alone confirms the controller process is alive — it does not confirm schedules are actually running. To verify end-to-end execution, create a canary SleepInfo resource in a dedicated test namespace and monitor its status field:

# canary-sleepinfo.yaml
apiVersion: kube-green.com/v1alpha1
kind: SleepInfo
metadata:
  name: canary-sleep
  namespace: kube-green-monitoring
spec:
  weekdays: "1-5"
  sleepAt: "23:00"
  wakeUpAt: "07:00"
  timeZone: "UTC"

Check the status after the scheduled time:

kubectl get sleepinfo -n kube-green-monitoring canary-sleep -o yaml
# Look for: status.lastScheduleTime matching the expected suspension time

You can also monitor a lightweight HTTP service in the test namespace that reports whether its deployment replica count is zero (suspended) or non-zero (awake) — this creates a behavioral check that proves kube-green's schedule is executing, not just that the controller pod is running.

Step 7: Configure Alerting

In Vigilmon under Settings → Notifications, configure your alert channels:

| Monitor | Trigger | Action | |---|---|---| | Metrics endpoint | Non-200 or keyword missing | Check kube-green pod: kubectl get pods -n kube-green; review logs for reconcile errors | | Webhook TCP | Connection refused | Webhook service down; kubectl get svc -n kube-green; check pod selector labels | | SSL certificate | < 30 days to expiry | Renew cert; check cert-manager: kubectl get certificate -n kube-green |

Escalation policy: Kube-green failures are cost-impacting but rarely user-facing. Route alerts to the platform/infrastructure team's Slack channel rather than a paging rotation. Set a 5-minute alert threshold to suppress transient failures during routine Kubernetes node drains.

Common Kube-green Failure Modes and What Vigilmon Catches

| Scenario | Vigilmon monitor | |---|---| | Controller pod OOM killed | Metrics endpoint returns connection error; alert within 2 min | | Helm upgrade breaks controller deployment | Metrics endpoint down during rollout; recovers when rollout completes | | Webhook certificate expires | Webhook TCP port still accepts connections but TLS handshake fails; SSL monitor catches 30 days early | | Namespace quota prevents controller pod from scheduling | Metrics endpoint down; check kubectl describe namespace kube-green | | Network policy blocks metrics scrape | Metrics endpoint returns 403 or connection refused; verify Ingress network policy | | SleepInfo schedules stop executing silently | Metrics endpoint is green; requires behavioral canary check (Step 6) | | cert-manager fails to renew webhook cert | Webhook service TLS fails; SSL monitor alerts at 30-day threshold | | Controller cannot reach Kubernetes API | Reconcile errors in logs; metrics endpoint stays up but schedule count stops incrementing |

Kube-green saves real infrastructure spend — but only when it's actually running. A crashed controller silently bills you for idle development environments all night, and the failure mode is invisible until someone notices the cloud invoice. Vigilmon closes this blind spot by checking the metrics endpoint, the webhook service availability, and the SSL certificates that keep both working, so your suspension schedules execute reliably and your cost optimization runs without manual oversight.

Start monitoring kube-green in under 5 minutes — register free at vigilmon.online.