Monitoring K6 Operator with Vigilmon: Test API Health, Results Backend Availability, Prometheus Endpoint & SSL Alerts

K6 Operator is the Kubernetes-native controller that turns K6 distributed load tests into first-class Kubernetes objects — letting platform and QA teams run TestRun CRDs that scale thousands of virtual users across a Kubernetes cluster without managing test infrastructure manually. When the K6 Operator's controller-manager crashes, no new test runs can be dispatched. When the results backend (InfluxDB, Grafana Cloud, or a Prometheus remote-write endpoint) becomes unavailable, test data is silently lost. When the Prometheus metrics scrape endpoint goes dark, you lose real-time visibility into test execution health. Vigilmon gives you external monitoring for the infrastructure that K6 Operator depends on.

What You'll Build

• A monitor on the K6 Operator controller-manager health endpoint
• An availability check on your test results backend (InfluxDB or Grafana Cloud)
• A Prometheus metrics endpoint monitor to confirm scraping is active
• SSL certificate monitoring for your results backend domain
• An alerting setup that catches each failure tier before test runs are silently discarded

Prerequisites

• K6 Operator installed in your cluster (typically via Helm or kubectl apply)
• A results backend: InfluxDB, TimescaleDB, or Grafana Cloud
• Prometheus scraping enabled for the K6 Operator pod
• A free account at vigilmon.online

Step 1: Understand K6 Operator's Health Endpoints

K6 Operator's controller-manager (built with controller-runtime) exposes health probes on port 8081 by default:

# Liveness probe
curl http://<operator-pod-ip>:8081/healthz
# Returns: ok

# Readiness probe
curl http://<operator-pod-ip>:8081/readyz
# Returns: ok

Because these endpoints are on the pod's ClusterIP, you cannot reach them directly from outside the cluster. The standard approach for external monitoring is to expose the health endpoint via a Kubernetes Service of type NodePort or LoadBalancer, or to use an Ingress with a dedicated path.

apiVersion: v1
kind: Service
metadata:
  name: k6-operator-health
  namespace: k6-operator-system
spec:
  selector:
    control-plane: controller-manager
  ports:
    - name: health
      port: 8081
      targetPort: 8081
  type: LoadBalancer

Apply this service and note the external IP or hostname Kubernetes assigns.

Step 2: Create a Vigilmon HTTP Monitor for the Controller-Manager Health

1. Log in to Vigilmon → Add Monitor → HTTP.
2. URL: http://<external-ip>:8081/healthz (or the Ingress path you configured).
3. Check interval: 60 seconds.
4. Response timeout: 10 seconds.
5. Expected status: 200.
6. Keyword: ok.
7. Label: K6 Operator controller health.
8. Click Save.

This monitor catches:

• Controller-manager pod crashes or OOM kills
• CRD reconciliation loop panics that restart the operator
• Kubernetes API server connectivity failures that prevent the operator from watching TestRun objects
• Cluster resource pressure that evicts the controller-manager pod

Alert sensitivity: Set to trigger after 1 consecutive failure. When the K6 Operator is down, new TestRun CRDs sit in a Pending state indefinitely, with no error surfaced to the engineer who submitted the test.

Step 3: Monitor Your Results Backend

K6 Operator forwards test metrics to a results backend. The exact endpoint depends on your backend:

InfluxDB v2:

curl https://influxdb.example.com/health
# Returns: {"name":"influxdb","message":"ready for queries and writes","status":"pass",...}

Grafana Cloud Prometheus remote-write (check endpoint reachability):

curl -I https://prometheus-prod-01.grafana.net/api/prom/push
# Returns: 400 Bad Request (endpoint is live; the 400 means no payload was sent)

For InfluxDB:

1. Add Monitor → HTTP.
2. URL: https://influxdb.example.com/health.
3. Check interval: 2 minutes.
4. Response timeout: 10 seconds.
5. Expected status: 200.
6. Keyword: pass (InfluxDB health response status field).
7. Label: K6 results backend (InfluxDB).
8. Click Save.

For Grafana Cloud remote-write endpoint:

1. Add Monitor → HTTP.
2. URL: https://prometheus-prod-01.grafana.net/api/prom/push.
3. Check interval: 5 minutes.
4. Expected status: 400 (the endpoint is live but rejects unauthenticated empty POSTs).
5. Label: K6 results backend (Grafana Cloud).
6. Click Save.

Why monitor 400 as success? Remote-write endpoints are authenticated and reject unauthenticated requests. A 400 Bad Request confirms the endpoint is alive and processing requests. A 502 or 503 means the backend is down — which is what you want to catch.

Step 4: Monitor the Prometheus Metrics Endpoint

K6 Operator exposes its own Prometheus metrics on port 8080 for controller-runtime metrics. These include reconciliation counts, error rates, and work queue depths that tell you whether the operator is processing TestRun objects:

curl http://<operator-pod-ip>:8080/metrics | head -20
# Returns: Prometheus exposition format lines beginning with #

Expose this with a Service or Ingress in the same way as the health endpoint:

apiVersion: v1
kind: Service
metadata:
  name: k6-operator-metrics
  namespace: k6-operator-system
spec:
  selector:
    control-plane: controller-manager
  ports:
    - name: metrics
      port: 8080
      targetPort: 8080
  type: LoadBalancer

Then add a Vigilmon monitor:

1. Add Monitor → HTTP.
2. URL: http://<metrics-service-external-ip>:8080/metrics.
3. Check interval: 2 minutes.
4. Response timeout: 10 seconds.
5. Expected status: 200.
6. Keyword: controller_runtime (present in all controller-runtime Prometheus metrics).
7. Label: K6 Operator Prometheus metrics.
8. Click Save.

Step 5: Monitor SSL Certificates for Your Results Backend

If your InfluxDB instance or Grafana Cloud endpoint uses a custom domain with a TLS certificate, monitor it for expiry:

openssl s_client -connect influxdb.example.com:443 2>/dev/null | openssl x509 -noout -dates

1. Add Monitor → SSL Certificate.
2. Domain: influxdb.example.com (or your results backend domain).
3. Alert when expiry is within: 30 days.
4. Alert again: 14 days, 7 days, 3 days, 1 day.
5. Click Save.

When the results backend certificate expires, K6 Operator's metric forwarding fails with a TLS handshake error. The controller logs show write failures, but the test still runs — you lose all data silently. A 30-day warning gives you time to renew before any K6 test results are discarded.

Step 6: Configure Alerting

In Vigilmon under Settings → Notifications, configure your alert channels:

| Monitor | Trigger | Action | |---|---|---| | Controller-manager health | Non-200 or ok missing | Check operator pod: kubectl get pods -n k6-operator-system | | Results backend | Non-200 / unexpected status | Check backend logs; test runs may be silently discarding metrics | | Prometheus metrics | Non-200 or keyword missing | Metrics scraping broken; check Service and pod health | | SSL certificate | < 30 days to expiry | Renew backend certificate; TLS write failures will follow |

Alert thresholds: Use 1 consecutive failure for the controller health monitor. Use 2 consecutive failures for results backend and metrics monitors to absorb transient network disruptions.

Common K6 Operator Failure Modes and What Vigilmon Catches

| Scenario | Vigilmon monitor | |---|---| | Controller-manager pod OOM killed | Health monitor fires within 60 seconds | | RBAC misconfiguration prevents CRD watching | Controller panics; health monitor fires | | Results backend down during active test run | Backend availability monitor fires; data loss begins | | Prometheus scrape endpoint port blocked | Metrics monitor fires; dashboards go stale | | InfluxDB disk full | Health endpoint may still return 200; write failures appear in K6 logs | | Results backend SSL certificate expires | SSL monitor fires at 30-day threshold; metric writes fail with TLS error | | Kubernetes API server degraded | Controller cannot reconcile TestRun CRDs; health probe may still pass | | Network policy blocks operator-to-backend traffic | Backend monitor fires; controller health stays green | | Helm upgrade replaces Service with wrong selector | All monitors fire simultaneously |

K6 Operator turns load testing into a Kubernetes-native workflow — but when the controller crashes or the results backend goes down, test runs either stall silently or discard data without any notification. Vigilmon gives you external visibility into every layer: the operator's health, your results backend, Prometheus metric collection, and certificate health, so you know immediately when a failure would compromise your load test data or prevent new tests from running.

Start monitoring K6 Operator in under 5 minutes — register free at vigilmon.online.