Litmus Chaos is an open-source chaos engineering platform that helps SREs and DevOps teams validate the resilience of cloud-native applications on Kubernetes. It provides a rich library of pre-built chaos experiments — pod delete, node CPU hog, network loss, disk I/O stress — that you can run against your workloads to find weaknesses before your users do.
But a chaos experiment only tells you something useful if you have an independent observer watching from outside your infrastructure. Vigilmon provides exactly that: continuous, external uptime monitoring that records the exact moment your service becomes unavailable and the moment it recovers, giving you an objective SLO measurement for every chaos run.
What Litmus Chaos and Vigilmon do differently
Litmus Chaos injects faults and measures the result using internal probes (HTTP, command, Prometheus). Vigilmon monitors your service from outside the cluster, from the perspective of a real end user.
| Capability | Litmus Chaos | Vigilmon | |-----------|-------------|---------| | Inject pod failures | Yes | No | | HTTP probe during experiment | Yes (internal) | Yes (external, continuous) | | 24/7 uptime monitoring | No | Yes | | Alert on service down | No | Yes | | Public status page | No | Yes | | SSL certificate monitoring | No | Yes | | Multi-region checks | No | Yes |
Together, they give you a complete picture: Litmus controls the fault injection, Vigilmon provides the external ground truth.
What you'll need
- A Kubernetes cluster (EKS, GKE, AKS, or a local cluster like kind)
- Litmus Chaos installed (ChaosCenter or the Kubernetes operator)
- At least one HTTP service exposed with an Ingress or LoadBalancer
- A free Vigilmon account — no credit card required
Step 1: Prepare your application health endpoint
Add a /health route to your application that reflects the health of its dependencies. This is what Vigilmon will probe every minute.
Python / FastAPI:
from fastapi import FastAPI
from contextlib import asynccontextmanager
import asyncpg
import redis.asyncio as aioredis
@app.get("/health")
async def health():
checks = {}
try:
conn = await asyncpg.connect(DATABASE_URL)
await conn.fetchval("SELECT 1")
await conn.close()
checks["database"] = "ok"
except Exception as e:
checks["database"] = f"error: {str(e)}"
try:
r = aioredis.from_url(REDIS_URL)
await r.ping()
checks["cache"] = "ok"
except Exception:
checks["cache"] = "error"
healthy = all(v == "ok" for v in checks.values())
status_code = 200 if healthy else 503
return JSONResponse(
content={"status": "ok" if healthy else "degraded", "checks": checks},
status_code=status_code,
)
Java / Spring Boot:
@RestController
@RequestMapping("/health")
public class HealthController {
@GetMapping
public ResponseEntity<HealthResponse> health() {
boolean dbOk = checkDatabase();
boolean cacheOk = checkRedis();
boolean healthy = dbOk && cacheOk;
return ResponseEntity
.status(healthy ? HttpStatus.OK : HttpStatus.SERVICE_UNAVAILABLE)
.body(new HealthResponse(
healthy ? "ok" : "degraded",
Map.of("database", dbOk ? "ok" : "error",
"cache", cacheOk ? "ok" : "error")
));
}
}
Deploy with Kubernetes readiness and liveness probes:
containers:
- name: api
image: your-org/api:latest
ports:
- containerPort: 8080
readinessProbe:
httpGet:
path: /health
port: 8080
initialDelaySeconds: 10
periodSeconds: 5
failureThreshold: 3
livenessProbe:
httpGet:
path: /health
port: 8080
initialDelaySeconds: 30
periodSeconds: 10
failureThreshold: 3
Step 2: Install Litmus Chaos
Using Helm:
helm repo add litmuschaos https://litmuschaos.github.io/litmus-helm/
helm repo update
helm install chaos litmuschaos/litmus \
--namespace litmus \
--create-namespace \
--set portal.frontend.service.type=NodePort
Install ChaosExperiments from the Hub:
kubectl apply -f https://hub.litmuschaos.io/api/chaos/master?file=charts/generic/pod-delete/experiment.yaml -n production
kubectl apply -f https://hub.litmuschaos.io/api/chaos/master?file=charts/generic/pod-cpu-hog/experiment.yaml -n production
kubectl apply -f https://hub.litmuschaos.io/api/chaos/master?file=charts/generic/network-loss/experiment.yaml -n production
Annotate target deployments:
kubectl annotate deployment api-server litmuschaos.io/chaos="true" -n production
Step 3: Set up Vigilmon monitors
Before running any experiments, configure your monitors and let them run for at least 30 minutes to establish a healthy baseline.
- Log in to vigilmon.online
- Monitors → New Monitor → HTTP / HTTPS
- URL:
https://api.your-domain.com/health - Check interval: 1 minute
- Expected status code:
200 - Response body contains:
"status":"ok" - Save
Add monitors for every service in your chaos scope:
| Service | Monitor URL | Expected Status |
|---------|-------------|----------------|
| API Server | https://api.your-domain.com/health | 200 |
| Frontend App | https://app.your-domain.com | 200 |
| Admin Panel | https://admin.your-domain.com/health | 200 |
Also add TCP monitors for any databases or queues that Litmus might target:
- Monitors → New Monitor → TCP Port
- Host:
your-db.internal.example.com - Port:
5432 - Save
Step 4: Configure Vigilmon alerts
Real-time alerting during chaos experiments is essential:
Slack:
- Alert Channels → Add Channel → Webhook
- Paste your Slack incoming webhook URL
- Assign to all monitors
PagerDuty:
- Alert Channels → Add Channel → Webhook
- Use your PagerDuty events API endpoint
- Assign to production-critical monitors only
When Vigilmon detects a failure, it sends:
{
"monitor_name": "API Server Health",
"status": "down",
"url": "https://api.your-domain.com/health",
"started_at": "2024-07-01T10:00:00Z",
"duration_seconds": 0
}
Note the started_at timestamp — you'll use this to correlate with when Litmus injected the fault.
Step 5: Run a pod delete experiment
The pod delete experiment is the most common starting point for chaos engineering. It deletes one pod from a deployment and verifies the service remains available:
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
name: api-server-pod-delete
namespace: production
spec:
appinfo:
appns: production
applabel: 'app=api-server'
appkind: deployment
chaosServiceAccount: litmus-admin
monitoring: false
jobCleanUpPolicy: retain
experiments:
- name: pod-delete
spec:
components:
env:
- name: TOTAL_CHAOS_DURATION
value: '120' # 2 minutes
- name: CHAOS_INTERVAL
value: '30' # delete a pod every 30 seconds
- name: FORCE
value: 'false'
probe:
- name: healthcheck-probe
type: httpProbe
mode: Continuous
runProperties:
probeTimeout: 5
retry: 3
interval: 10
httpProbe/inputs:
url: https://api.your-domain.com/health
insecureSkipVerify: false
method:
get:
criteria: ==
responseCode: "200"
Apply it:
kubectl apply -f pod-delete-engine.yaml -n production
What to watch:
- Litmus's internal HTTP probe runs every 10 seconds
- Vigilmon's external probe runs every 60 seconds
- If Vigilmon fires an incident during the experiment, you have a real resilience gap
Step 6: Network loss experiment
Simulate packet loss on your API service's network:
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
name: api-network-loss
namespace: production
spec:
appinfo:
appns: production
applabel: 'app=api-server'
appkind: deployment
chaosServiceAccount: litmus-admin
experiments:
- name: pod-network-loss
spec:
components:
env:
- name: TOTAL_CHAOS_DURATION
value: '300' # 5 minutes
- name: NETWORK_INTERFACE
value: 'eth0'
- name: NETWORK_PACKET_LOSS_PERCENTAGE
value: '100' # 100% packet loss (partition)
- name: CONTAINER_RUNTIME
value: 'containerd'
- name: SOCKET_PATH
value: '/run/containerd/containerd.sock'
100% packet loss effectively partitions the pod from the network. Vigilmon will fire an incident if this causes the external health endpoint to become unreachable.
Step 7: CPU hog experiment
Stress CPU resources to verify your service stays responsive under compute pressure:
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
name: api-cpu-hog
namespace: production
spec:
appinfo:
appns: production
applabel: 'app=api-server'
appkind: deployment
chaosServiceAccount: litmus-admin
experiments:
- name: pod-cpu-hog
spec:
components:
env:
- name: TOTAL_CHAOS_DURATION
value: '300'
- name: CPU_CORES
value: '2'
- name: PODS_AFFECTED_PERC
value: '50' # affect 50% of pods
During this experiment, check Vigilmon's response time data. If your health endpoint starts timing out (Vigilmon marks it down), you've found the CPU threshold at which the service becomes functionally unavailable — even before Kubernetes marks it as unhealthy.
Step 8: Build a chaos experiment runbook with Vigilmon validation
Here's a repeatable workflow for every chaos experiment:
#!/bin/bash
# chaos-experiment.sh
EXPERIMENT_NAME=$1
DURATION=${2:-300} # default 5 minutes
echo "[$(date -u)] Starting chaos experiment: $EXPERIMENT_NAME"
echo "[$(date -u)] Vigilmon baseline: check https://vigilmon.online for incident history"
# Apply chaos engine
kubectl apply -f "experiments/${EXPERIMENT_NAME}.yaml" -n production
# Wait for experiment duration
echo "[$(date -u)] Chaos running for ${DURATION}s. Watching Vigilmon..."
sleep $DURATION
# Check chaos result
VERDICT=$(kubectl get chaosresult "${EXPERIMENT_NAME}-result" -n production \
-o jsonpath='{.status.experimentStatus.verdict}')
echo "[$(date -u)] Litmus verdict: $VERDICT"
echo "[$(date -u)] Experiment complete. Check Vigilmon for external incident history."
# Clean up
kubectl delete chaosengine "${EXPERIMENT_NAME}" -n production
After each experiment, check the Vigilmon incident log:
- Zero incidents + Litmus Pass: service is resilient
- Incident during experiment + Litmus Pass: internal probes passed but external users experienced downtime — your internal probes may be too lenient
- Incident beyond experiment end: service didn't recover fully — investigate auto-restart or circuit breaker configuration
Step 9: Use Litmus probes alongside Vigilmon
Litmus supports HTTP probes that run during experiments. Use them together with Vigilmon for defense-in-depth verification:
- Litmus HTTP probe: runs every 10 seconds, internal to the cluster, verifies the endpoint from within Kubernetes networking
- Vigilmon probe: runs every 60 seconds, external to the cluster, verifies the endpoint from the public internet
When both agree (both see failure), you have high confidence the service is genuinely down. When only Vigilmon sees failure, the issue may be in your ingress controller, load balancer, or TLS termination — not in the pod itself.
Step 10: Status page during chaos experiments
Give your team a shared view of service health during experiments:
- Status Pages → New Status Page
- Name it "Chaos Test Status" or "Engineering Status"
- Add all monitored services
- Publish and share with the chaos testing team
Before each experiment, post a maintenance notice on the status page. During experiments, the page updates in real time. After each experiment, the page history gives you a timeline artifact.
Chaos experiment outcomes reference
| Litmus Result | Vigilmon Result | Interpretation | |--------------|----------------|----------------| | Pass | No incident | Service is resilient ✓ | | Pass | Incident (< 1 min) | Brief blip — investigate pod restart timing | | Pass | Incident (> 1 min) | External availability failed despite passing internal probes | | Fail | No incident | Internal degradation not externally visible — good graceful degradation | | Fail | Incident | Service fully unavailable — priority fix | | Fail | Incident longer than experiment | Recovery automation not working |
What's next
- Litmus Chaos Workflows: chain experiments and gate each step on Vigilmon's monitor status using the Vigilmon API
- SSL certificate monitoring: add expiry alerts for all domains in your chaos test scope
- Heartbeat monitoring: ensure your chaos experiment scheduler (cron jobs or Litmus Schedules) continues to fire — Vigilmon alerts you when a scheduled chaos run stops reporting in
Get started free at vigilmon.online — monitors go live in under a minute, no credit card required.