litmuschaos_health.py

Prerequisites

• LitmusChaos 3.x installed (Helm or operator-based)
• kubectl with cluster-admin access
• A free account at vigilmon.online

Part 1: Understanding what to monitor in LitmusChaos

LitmusChaos has three monitoring surfaces:

1. ChaosCenter — the backend API server and frontend dashboard; losing this makes experiments unmanageable
2. Chaos operators and runners — pods that execute individual fault injections; a stuck runner leaves a fault running indefinitely
3. ChaosEngine status — the CRD that tracks experiment state; Completed is healthy, Stopped or Running for too long is a signal

All three need monitoring. The most dangerous failure is a runner pod that loses contact with ChaosCenter and continues injecting a fault with no one watching.

Part 2: Build a LitmusChaos HTTP health endpoint

Create litmuschaos_health.py

import os
import subprocess
import json
import time
import requests
from fastapi import FastAPI, Response

app = FastAPI()

CHAOSCENTER_URL = os.environ.get("CHAOSCENTER_URL", "http://localhost:9091")
NAMESPACE = os.environ.get("LITMUS_NAMESPACE", "litmus")
STALLED_EXPERIMENT_MINUTES = int(os.environ.get("STALLED_EXPERIMENT_MINUTES", "60"))


def kubectl_json(*args):
    result = subprocess.run(
        ["kubectl", *args, "-o", "json"],
        capture_output=True, text=True, timeout=15,
    )
    if result.returncode != 0:
        raise RuntimeError(result.stderr.strip()[:300])
    return json.loads(result.stdout)


@app.get("/health")
def health(response: Response):
    checks = {}

    # ChaosCenter API reachability
    try:
        resp = requests.get(f"{CHAOSCENTER_URL}/status", timeout=8)
        if resp.status_code == 200:
            checks["chaoscenter_api"] = "ok"
        else:
            checks["chaoscenter_api"] = f"error: HTTP {resp.status_code}"
    except Exception as e:
        checks["chaoscenter_api"] = f"error: {e}"

    # Chaos operator pod readiness
    try:
        pods = kubectl_json(
            "get", "pods", "-n", NAMESPACE,
            "-l", "app.kubernetes.io/component=operator",
        )
        items = pods.get("items", [])
        not_ready = [
            p["metadata"]["name"]
            for p in items
            if not all(
                c.get("status") == "True"
                for c in p.get("status", {}).get("conditions", [])
                if c.get("type") == "Ready"
            )
        ]
        if not_ready:
            checks["chaos_operator"] = f"degraded: {', '.join(not_ready)}"
        else:
            checks["chaos_operator"] = f"ok ({len(items)} ready)"
    except Exception as e:
        checks["chaos_operator"] = f"error: {e}"

    # Stalled ChaosEngine detection
    try:
        engines = kubectl_json("get", "chaosengines", "-A")
        stalled = []
        for eng in engines.get("items", []):
            status_phase = eng.get("status", {}).get("engineStatus", "")
            if status_phase not in ("completed", "stopped", ""):
                # Engine is actively running; check start time
                start = eng.get("metadata", {}).get("creationTimestamp", "")
                if start:
                    import datetime
                    created = datetime.datetime.fromisoformat(
                        start.replace("Z", "+00:00")
                    )
                    age_min = (
                        datetime.datetime.now(datetime.timezone.utc) - created
                    ).total_seconds() / 60
                    if age_min > STALLED_EXPERIMENT_MINUTES:
                        stalled.append(
                            f"{eng['metadata']['namespace']}/{eng['metadata']['name']} "
                            f"({int(age_min)}m)"
                        )
        if stalled:
            checks["stalled_engines"] = f"warning: {'; '.join(stalled)}"
        else:
            checks["stalled_engines"] = "ok"
    except Exception as e:
        checks["stalled_engines"] = f"error: {e}"

    api_ok = checks.get("chaoscenter_api") == "ok"
    operator_ok = "ok" in str(checks.get("chaos_operator", ""))
    healthy = api_ok and operator_ok

    response.status_code = 200 if healthy else 503
    return {"status": "ok" if healthy else "degraded", "checks": checks}

Run the health service

pip install fastapi uvicorn requests
uvicorn litmuschaos_health:app --host 0.0.0.0 --port 8096

Deploy in-cluster

# litmuschaos-health-deploy.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: litmuschaos-health
  namespace: litmus
spec:
  replicas: 1
  selector:
    matchLabels:
      app: litmuschaos-health
  template:
    metadata:
      labels:
        app: litmuschaos-health
    spec:
      serviceAccountName: litmuschaos-health-sa
      containers:
        - name: health
          image: python:3.12-slim
          command:
            - sh
            - -c
            - |
              pip install -q fastapi uvicorn requests &&
              uvicorn litmuschaos_health:app --host 0.0.0.0 --port 8096
          ports:
            - containerPort: 8096
          env:
            - name: CHAOSCENTER_URL
              value: http://litmusportal-server-service:9091
            - name: LITMUS_NAMESPACE
              value: litmus
            - name: STALLED_EXPERIMENT_MINUTES
              value: "60"
---
apiVersion: v1
kind: Service
metadata:
  name: litmuschaos-health
  namespace: litmus
spec:
  selector:
    app: litmuschaos-health
  ports:
    - port: 8096
      targetPort: 8096
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: litmuschaos-health-sa
  namespace: litmus
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: litmuschaos-health-reader
rules:
  - apiGroups: [""]
    resources: ["pods"]
    verbs: ["get", "list"]
  - apiGroups: ["litmuschaos.io"]
    resources: ["chaosengines", "chaosexperiments", "chaosresults"]
    verbs: ["get", "list"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: litmuschaos-health-reader
subjects:
  - kind: ServiceAccount
    name: litmuschaos-health-sa
    namespace: litmus
roleRef:
  kind: ClusterRole
  name: litmuschaos-health-reader
  apiGroup: rbac.authorization.k8s.io

kubectl apply -f litmuschaos-health-deploy.yaml

Verify the endpoint

kubectl port-forward svc/litmuschaos-health 8096:8096 -n litmus
curl http://localhost:8096/health

Expected healthy response:

{
  "status": "ok",
  "checks": {
    "chaoscenter_api": "ok",
    "chaos_operator": "ok (1 ready)",
    "stalled_engines": "ok"
  }
}

Part 3: Set up HTTP monitoring in Vigilmon

Monitor ChaosCenter API and operator health

1. Log in to vigilmon.online and click Add Monitor.
2. Choose HTTP(S) monitor.
3. Enter: http://your-ingress-host:8096/health
4. Set interval to 2 minutes.
5. Add a keyword check: must contain "status":"ok".
6. Add your alert channel.
7. Click Save.

The keyword check fires when the operator is degraded or a stalled experiment is detected, even if the HTTP status is 200.

Monitor ChaosCenter dashboard directly

Add a second Vigilmon monitor pointed at the ChaosCenter web UI:

1. Add Monitor → HTTP(S) monitor.
2. Enter: http://your-chaoscenter-host:9091/
3. Set interval to 5 minutes.
4. Keyword: ChaosCenter or another string from the homepage.

This catches cases where the API is up but the frontend build is broken or the service is misconfigured.

Part 4: Monitor experiment outcomes via ChaosResult

LitmusChaos writes a ChaosResult CRD after each experiment. A failed experiment that goes unnoticed may indicate a real production weakness.

# chaosresult_health.py
import subprocess
import json
from fastapi import FastAPI, Response

app = FastAPI()


@app.get("/health/results")
def results_health(response: Response):
    try:
        result = subprocess.run(
            ["kubectl", "get", "chaosresults", "-A", "-o", "json"],
            capture_output=True, text=True, timeout=15,
        )
        if result.returncode != 0:
            response.status_code = 503
            return {"status": "error", "detail": result.stderr.strip()[:200]}

        items = json.loads(result.stdout).get("items", [])
        failed = [
            f"{r['metadata']['namespace']}/{r['metadata']['name']}: "
            f"{r.get('status', {}).get('experimentStatus', {}).get('verdict', 'unknown')}"
            for r in items
            if r.get("status", {}).get("experimentStatus", {}).get("verdict", "") == "Fail"
        ]

        if failed:
            response.status_code = 503
            return {
                "status": "degraded",
                "failed_experiments": failed,
                "total_results": len(items),
            }

        return {"status": "ok", "total_results": len(items)}

    except Exception as e:
        response.status_code = 503
        return {"status": "error", "detail": str(e)}

Add a Vigilmon monitor for /health/results with a 10-minute interval to track experiment outcomes over time.

Part 5: Docker Compose for local testing

# docker-compose.yml
version: "3.8"

services:
  litmuschaos-health:
    image: python:3.12-slim
    command: >
      sh -c "pip install -q fastapi uvicorn requests &&
             uvicorn litmuschaos_health:app --host 0.0.0.0 --port 8096"
    ports:
      - "8096:8096"
    volumes:
      - ./litmuschaos_health.py:/app/litmuschaos_health.py
      - ~/.kube:/root/.kube:ro
    working_dir: /app
    environment:
      CHAOSCENTER_URL: http://host.docker.internal:9091
      LITMUS_NAMESPACE: litmus

Part 6: Webhook alerts for chaos control plane loss

A ChaosCenter outage during a running experiment is a critical situation — engineers cannot abort faults or review blast radius.

// webhook-receiver.ts
import express from 'express';

const app = express();
app.use(express.json());

app.post('/webhook/vigilmon', async (req, res) => {
  const { monitor_name, status, url, checked_at } = req.body;

  if (status === 'down') {
    console.error('[CHAOS] LitmusChaos control plane DOWN', {
      monitor: monitor_name,
      at: checked_at,
    });

    await alertChaosTeam({
      title: `LitmusChaos DOWN: ${monitor_name}`,
      severity: 'critical',
      impact: 'Running chaos experiments cannot be monitored or aborted',
      since: checked_at,
      runbook: 'https://wiki.example.com/runbooks/litmuschaos-recovery',
    });

    // Attempt emergency experiment abort via kubectl
    await triggerEmergencyAbort();

  } else if (status === 'up') {
    console.info('[CHAOS] LitmusChaos control plane recovered', {
      monitor: monitor_name,
    });
    await resolveAlert(monitor_name);
  }

  res.sendStatus(204);
});

Vigilmon sends this payload:

{
  "monitor_id": "mon_litmus",
  "monitor_name": "LitmusChaos — ChaosCenter API",
  "status": "down",
  "url": "http://cluster.internal:8096/health",
  "checked_at": "2026-07-02T14:00:00Z",
  "response_code": 503,
  "response_time_ms": 8002
}

Part 7: SSL monitoring for ChaosCenter

If ChaosCenter is served over HTTPS with a TLS-terminating ingress:

1. In Vigilmon, click Add Monitor.
2. Choose SSL monitor.
3. Enter your ChaosCenter domain: chaos.example.com.
4. Set alert threshold to 14 days before expiry.
5. Add your alert channel.

An expired certificate on the ChaosCenter ingress blocks engineers from accessing the dashboard and aborting experiments — exactly the wrong time to be locked out.

Summary

Your LitmusChaos deployment now has layered monitoring:

1. ChaosCenter API health endpoint — confirms the control plane is reachable, polled every 2 minutes by Vigilmon.
2. Chaos operator pod monitoring — detects operator crashes that would leave running experiments without lifecycle management.
3. Stalled experiment detection — flags ChaosEngines that have been Running beyond the expected duration.
4. ChaosResult outcome tracking — surfaces failed experiments that may indicate real resilience gaps.
5. Webhook alerts with emergency abort — triggers team escalation and automated experiment termination when the control plane goes down.
6. SSL monitors — alerts before certificate expiry locks engineers out of the dashboard during a live experiment.

Vigilmon handles check scheduling, multi-region polling, and alert routing. When LitmusChaos loses control of a running experiment, your chaos engineering team knows within 2 minutes.

Monitor your LitmusChaos chaos engineering platform free at vigilmon.online

#litmuschaos #chaosengineering #kubernetes #sre #resilience #monitoring