Monitoring Robusta with Vigilmon: Runner Health API, Relay Server Availability, Webhook Endpoint & SSL Certificate Alerts

Robusta is the Kubernetes observability and automated remediation platform that enriches Prometheus alerts with live cluster context — attaching pod logs, resource graphs, and suggested fixes to every firing alert before it reaches your on-call engineer. When the Robusta runner goes down, Prometheus alerts start arriving as raw, context-free notifications: no logs, no graph snapshots, no automated triage. When the Robusta relay server (the cloud component that routes notifications to Slack, Teams, and mobile apps) is unreachable, alert delivery stops entirely. Vigilmon gives you external visibility into the Robusta stack: the runner health endpoint, the relay server availability, the Alertmanager webhook that Robusta registers, and the SSL certificates protecting each layer.

What You'll Build

• An HTTP monitor on Robusta's runner health endpoint to detect process failures
• A TCP check on the Alertmanager webhook port to confirm alert routing is functional
• An availability monitor on the Robusta relay server for notification delivery
• SSL certificate monitoring for all Robusta HTTPS endpoints
• An escalation runbook that maps Vigilmon alerts to specific remediation steps

Prerequisites

• Robusta v0.10.0+ deployed via Helm in the robusta namespace
• Prometheus and Alertmanager configured to send webhooks to Robusta
• A Robusta account (free tier available) for relay server access
• A free account at vigilmon.online

Step 1: Understand Robusta's Architecture and Health Surfaces

Robusta consists of two main components in your cluster:

1. Robusta Runner: The in-cluster pod that receives Alertmanager webhooks, enriches alerts with cluster context, and triggers automated playbooks (remediation actions).
2. Robusta Relay: An optional cloud service that routes enriched alerts to notification targets (Slack, Teams, PagerDuty, mobile app).

# Check Robusta runner pod health
kubectl get pods -n robusta

# View runner logs for webhook processing
kubectl logs -n robusta deployment/robusta-runner --tail=100

# Check Robusta's registered playbooks
kubectl get robustadocument -A

# Verify Alertmanager is sending to Robusta
kubectl get alertmanagerconfig -A

Robusta exposes a health endpoint you can probe to determine whether the runner process is alive and processing webhooks:

# Port-forward to check the runner health endpoint
kubectl port-forward -n robusta svc/robusta-runner 9090:9090
curl http://localhost:9090/healthz
# Returns: {"status": "ok"}

Step 2: Expose and Monitor the Robusta Runner Health Endpoint

To enable external monitoring with Vigilmon, expose the runner health endpoint via an Ingress or LoadBalancer service:

# robusta-health-ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: robusta-runner-health
  namespace: robusta
  annotations:
    nginx.ingress.kubernetes.io/auth-type: basic
    nginx.ingress.kubernetes.io/auth-secret: robusta-health-auth
spec:
  rules:
  - host: robusta-health.example.com
    http:
      paths:
      - path: /healthz
        pathType: Exact
        backend:
          service:
            name: robusta-runner
            port:
              number: 9090
  tls:
  - hosts:
    - robusta-health.example.com
    secretName: robusta-health-tls

Apply and verify:

kubectl apply -f robusta-health-ingress.yaml
curl https://robusta-health.example.com/healthz
# Returns: {"status": "ok"}

In Vigilmon:

1. Log in to Vigilmon → Add Monitor → HTTP.
2. URL: https://robusta-health.example.com/healthz.
3. Check interval: 60 seconds.
4. Response timeout: 15 seconds.
5. Expected status: 200.
6. Keyword: ok.
7. Label: Robusta runner health.
8. Click Save.

This monitor catches:

• Robusta runner pod OOM kills from memory-intensive playbook execution
• Kubernetes API connectivity failures that prevent runner from enriching alerts
• Helm upgrade failures during Robusta version updates
• Pod eviction events from node resource pressure
• Namespace quota limits preventing the runner from restarting after a crash

Alert sensitivity: Set to trigger after 1 consecutive failure. When Robusta's runner is down, on-call engineers receive context-free Prometheus alerts — they're flying blind during incidents.

Step 3: Monitor the Alertmanager Webhook Port via TCP

Robusta registers an Alertmanager webhook receiver that Prometheus sends all firing alerts to. This webhook port is the entry point for all automated remediation. If the port is unreachable (due to a Kubernetes NetworkPolicy change, a Service selector mismatch, or a runner crash), Alertmanager starts logging webhook delivery failures:

# Check Robusta's webhook service
kubectl get svc -n robusta robusta-runner
# Look for port 9090 (or your configured webhook port)

# Verify Alertmanager is configured to send to Robusta
kubectl get secret -n monitoring alertmanager-prometheus-kube-prometheus-alertmanager -o jsonpath='{.data.alertmanager\.yaml}' | base64 -d

The typical Alertmanager configuration for Robusta:

# alertmanager.yaml (excerpt)
receivers:
- name: robusta
  webhook_configs:
  - url: http://robusta-runner.robusta.svc.cluster.local:9090/api/alerts
    send_resolved: true

If the runner service is externally accessible for TCP monitoring:

1. Add Monitor → TCP.
2. Host: robusta-webhook.example.com (or the LoadBalancer IP).
3. Port: 9090.
4. Check interval: 2 minutes.
5. Label: Robusta webhook port.
6. Click Save.

Why TCP monitoring matters here: Alertmanager buffers failed webhook deliveries and retries — but only for a limited window. If Robusta's webhook port is unreachable for more than 30 minutes, Alertmanager drops the alerts and they never reach Robusta, even after the runner recovers. TCP monitoring catches this failure before the retry window closes.

Step 4: Monitor the Robusta Relay Server

The Robusta relay server routes enriched alert notifications from your cluster to external targets (Slack, Teams, PagerDuty, mobile). If the relay is unreachable, Robusta runner can still process alerts internally but cannot deliver them to your team:

# Verify Robusta can reach the relay server
kubectl logs -n robusta deployment/robusta-runner | grep "relay"
# Look for: "Successfully sent to relay" or "Failed to connect to relay"

The Robusta relay endpoint used by the runner is configured in your generated_values.yaml:

# Robusta configuration
globalConfig:
  robusta_ui_token: <your-token>
  # relay_url is typically: https://relay.robusta.dev

Monitor the relay server availability:

1. Add Monitor → HTTP.
2. URL: https://relay.robusta.dev (check Robusta docs for the current relay URL, or use your self-hosted relay).
3. Check interval: 2 minutes.
4. Response timeout: 15 seconds.
5. Expected status: 200 (or any non-5xx response).
6. Label: Robusta relay server.
7. Click Save.

If you self-host the Robusta relay:

# Self-hosted relay health check
curl https://relay.example.com/healthz

Use your self-hosted relay URL instead.

Step 5: Configure Robusta Playbooks for Self-Monitoring

Robusta can monitor itself through its own playbook system — creating a meta-monitoring loop that detects when Robusta's core components are degraded:

# robusta-self-monitoring.yaml
apiVersion: robusta.dev/v1
kind: PlaybookRepository
metadata:
  name: self-monitoring
  namespace: robusta
---
apiVersion: robusta.dev/v1
kind: RobustaDocument
metadata:
  name: monitor-robusta-runner
  namespace: robusta
spec:
  triggers:
  - on_pod_crash_loop:
      name_prefix: robusta-runner
      namespace: robusta
  actions:
  - create_finding:
      title: "Robusta Runner Crash Loop Detected"
      description: "Robusta runner is in a crash loop — alert enrichment is degraded"
      severity: HIGH

This playbook uses Robusta to monitor Robusta itself — which works during crash loops and restarts because the on_pod_crash_loop trigger fires from a different component than the runner pod itself.

Step 6: Monitor SSL Certificates

All HTTPS endpoints in the Robusta stack need certificate monitoring:

# Check runner health Ingress certificate
openssl s_client -connect robusta-health.example.com:443 2>/dev/null | openssl x509 -noout -dates

# If you have a self-hosted relay
openssl s_client -connect relay.example.com:443 2>/dev/null | openssl x509 -noout -dates

In Vigilmon:

1. Add Monitor → SSL Certificate.
2. Domain: robusta-health.example.com.
3. Alert when expiry is within: 30 days.
4. Alert again: 14 days, 7 days, 3 days, 1 day.
5. Click Save.

Relay TLS and alert delivery: If you self-host the Robusta relay and its TLS certificate expires, the Robusta runner starts logging TLS errors on every alert enrichment attempt. No notifications reach Slack, Teams, or mobile — and on-call engineers don't know until they notice alerts are missing. A 30-day Vigilmon SSL alert gives your team time to renew before delivery breaks.

Step 7: Configure Alerting

In Vigilmon under Settings → Notifications, configure your alert channels:

| Monitor | Trigger | Action | |---|---|---| | Runner health | Non-200 or ok missing | Robusta runner down; kubectl get pods -n robusta; check OOM or eviction events | | Webhook TCP port | Connection refused | Alertmanager cannot deliver to Robusta; check Service and NetworkPolicy | | Relay server | Non-200 or connection error | Notifications not reaching team; check relay credentials; notify on-call manually | | SSL certificate | < 30 days to expiry | Renew cert; Alertmanager webhook and relay connections will fail at expiry |

Escalation path: Because Robusta is your alert enrichment layer, a Robusta failure means your existing Prometheus/Alertmanager setup continues alerting — but without Kubernetes context attached. Configure Vigilmon to alert your platform team's Slack channel immediately when the runner health check fails, so they can investigate before an incident requires enriched alerts.

Common Robusta Failure Modes and What Vigilmon Catches

| Scenario | Vigilmon monitor | |---|---| | Runner pod OOM killed during high-alert volume | Health endpoint goes down; alert within 60 s | | Helm upgrade fails during Robusta version update | Health endpoint unavailable during rollout; may auto-recover | | Alertmanager webhook port becomes unreachable | TCP check fails; Alertmanager queues deliveries up to retry window | | Relay server connectivity lost | Relay HTTP monitor fails; alerts are enriched but not delivered | | Runner cannot reach Kubernetes API | Health endpoint stays up but playbooks fail silently | | SSL certificate expires on self-hosted relay | SSL monitor alerts at 30 days; runner TLS errors begin at expiry | | NetworkPolicy change blocks Alertmanager → Robusta traffic | Webhook TCP check fails; Alertmanager delivery errors in logs | | Robusta playbook causes runner crash loop | Health endpoint bounces; runner logs show playbook exception | | Relay authentication token expires | Relay returns 401; no delivery errors visible from outside without log access |

Robusta gives your on-call team the context they need to resolve Kubernetes incidents faster — but that capability disappears silently when the runner crashes or the relay loses connectivity. Instead of discovering that alert enrichment has been broken for hours mid-incident, Vigilmon monitors the runner health endpoint, the webhook port, and the relay server availability continuously, so you know the moment Robusta's automated remediation layer needs attention and can restore it before the next production alert fires.

Start monitoring Robusta in under 5 minutes — register free at vigilmon.online.