Chaos Mesh is an open-source chaos engineering platform for Kubernetes. It lets you inject faults — pod failures, network partitions, disk latency, CPU stress — into your running services to verify they degrade gracefully and recover automatically.
But chaos engineering without external monitoring is like running a fire drill without anyone watching the exits. You need an independent observer that can tell you, from the outside world, exactly when a service became unavailable and when it recovered. That's exactly what Vigilmon provides.
In this tutorial you'll set up Vigilmon monitors for your Kubernetes services, run Chaos Mesh experiments, and use the real-time incident data to validate your service's resilience against your SLOs.
Why external monitoring is essential for chaos engineering
Chaos Mesh can tell you that it injected a pod failure at 14:00:00. Your internal metrics (Prometheus, Datadog) can tell you that error rates spiked. But only an external monitor can tell you:
- Did real users experience an outage?
- At what exact timestamp did the service become unreachable from outside the cluster?
- How long before the service recovered to a 200 response?
- Did the recovery path function correctly (auto-restart, load balancer failover)?
Vigilmon probes your services from outside your cluster — the same vantage point your users have. This makes it the authoritative source of truth for chaos experiment outcomes.
What you'll need
- A Kubernetes cluster (kind, minikube, EKS, GKE, AKS)
- Chaos Mesh installed in your cluster
- One or more HTTP services exposed via Ingress or LoadBalancer
- A free Vigilmon account
Step 1: Expose a health endpoint from your Kubernetes service
Your service needs a /health route that Vigilmon can probe. It should return 200 when the service and its dependencies are healthy, and a non-200 status when degraded.
Example: Node.js / Express:
app.get('/health', async (req, res) => {
const checks = {
database: await checkDatabase(),
cache: await checkRedis(),
};
const healthy = Object.values(checks).every(Boolean);
res.status(healthy ? 200 : 503).json({
status: healthy ? 'ok' : 'degraded',
checks,
timestamp: new Date().toISOString(),
});
});
Kubernetes deployment and service:
apiVersion: apps/v1
kind: Deployment
metadata:
name: api-server
namespace: production
spec:
replicas: 3
selector:
matchLabels:
app: api-server
template:
metadata:
labels:
app: api-server
spec:
containers:
- name: api
image: your-org/api:latest
ports:
- containerPort: 3000
livenessProbe:
httpGet:
path: /health
port: 3000
initialDelaySeconds: 15
periodSeconds: 10
readinessProbe:
httpGet:
path: /health
port: 3000
initialDelaySeconds: 5
periodSeconds: 5
---
apiVersion: v1
kind: Service
metadata:
name: api-server
namespace: production
spec:
selector:
app: api-server
ports:
- port: 80
targetPort: 3000
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: api-server
namespace: production
spec:
rules:
- host: api.your-domain.com
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: api-server
servicePort: number: 80
Step 2: Install Chaos Mesh
Install Chaos Mesh using Helm:
helm repo add chaos-mesh https://charts.chaos-mesh.org
helm repo update
kubectl create namespace chaos-mesh
helm install chaos-mesh chaos-mesh/chaos-mesh \
--namespace chaos-mesh \
--set chaosDaemon.runtime=containerd \
--set chaosDaemon.socketPath=/run/containerd/containerd.sock \
--version 2.6.3
Verify the installation:
kubectl get pods -n chaos-mesh
# NAME READY STATUS RESTARTS AGE
# chaos-controller-manager-xxx 3/3 Running 0 2m
# chaos-daemon-xxx 1/1 Running 0 2m
# chaos-dashboard-xxx 1/1 Running 0 2m
Access the Chaos Mesh dashboard:
kubectl port-forward svc/chaos-dashboard -n chaos-mesh 2333:2333
Open http://localhost:2333 in your browser.
Step 3: Set up Vigilmon monitors for your services
Before running any chaos experiments, configure Vigilmon monitors so you have a baseline of normal operation.
- Log in to vigilmon.online
- Monitors → New Monitor → HTTP / HTTPS
- URL:
https://api.your-domain.com/health - Check interval: 1 minute
- Expected status code:
200 - Save
Add monitors for each service you plan to run experiments against:
| Service | Monitor URL | Type |
|---------|-------------|------|
| API Server | https://api.your-domain.com/health | HTTP |
| Frontend | https://app.your-domain.com | HTTP |
| PostgreSQL (if exposed) | your-db.example.com:5432 | TCP |
| Redis (if exposed) | your-redis.example.com:6379 | TCP |
Let Vigilmon run for at least 30 minutes before your first experiment. This establishes a baseline uptime and confirms your health endpoints are working.
Step 4: Configure Vigilmon alerts
Set up alert channels so your team receives instant notification when an experiment causes an external outage:
- Alert Channels → Add Channel → Webhook
- Enter your Slack or PagerDuty webhook URL
- Assign to all your service monitors
This is critical for chaos experiments: you want to know the exact second an outage begins and ends, not discover it 10 minutes later by refreshing a dashboard.
Step 5: Run a pod failure experiment
Start with the simplest chaos experiment: killing one pod and verifying the service remains available through your other replicas.
Create pod-failure.yaml:
apiVersion: chaos-mesh.org/v1alpha1
kind: PodChaos
metadata:
name: api-pod-failure
namespace: production
spec:
action: pod-failure
mode: one
selector:
namespaces:
- production
labelSelectors:
app: api-server
duration: '2m'
scheduler:
cron: '@none'
Apply it:
kubectl apply -f pod-failure.yaml
What to observe in Vigilmon:
- If Vigilmon fires an incident, your deployment is not resilient enough — traffic isn't load balanced to healthy pods fast enough
- If Vigilmon stays green throughout the 2-minute experiment, your readiness probes and load balancer are working correctly
- Check the incident history after the experiment for the exact downtime window (should be 0 if your setup is correct)
Step 6: Network partition experiment
Simulate a network partition between your API and database:
apiVersion: chaos-mesh.org/v1alpha1
kind: NetworkChaos
metadata:
name: api-db-partition
namespace: production
spec:
action: partition
mode: all
selector:
namespaces:
- production
labelSelectors:
app: api-server
direction: to
target:
selector:
namespaces:
- production
labelSelectors:
app: postgresql
mode: all
duration: '5m'
kubectl apply -f network-partition.yaml
Expected behavior:
- Your API server loses connectivity to the database
- Health endpoint returns 503 (if it checks DB connectivity)
- Vigilmon fires an incident immediately
- Your incident response runbook triggers
What you learn: Does your app fail gracefully? Does it serve cached data or return meaningful error messages? Does it recover automatically when the partition heals?
Step 7: CPU stress experiment
Simulate CPU exhaustion on your pods to test behavior under resource pressure:
apiVersion: chaos-mesh.org/v1alpha1
kind: StressChaos
metadata:
name: api-cpu-stress
namespace: production
spec:
mode: all
selector:
namespaces:
- production
labelSelectors:
app: api-server
stressors:
cpu:
workers: 4
load: 90
duration: '5m'
kubectl apply -f cpu-stress.yaml
Watch Vigilmon during this experiment. If response times exceed your Vigilmon timeout threshold (default 30 seconds), Vigilmon will fire an incident even though the pod is technically running. This reveals that high CPU load makes your service functionally unavailable before Kubernetes declares it unhealthy.
Step 8: Scheduled chaos experiments
Run chaos experiments on a schedule to continuously validate resilience:
apiVersion: chaos-mesh.org/v1alpha1
kind: Schedule
metadata:
name: weekly-pod-failure
namespace: production
spec:
schedule: '0 2 * * 2' # every Tuesday at 2am
type: PodChaos
podChaos:
action: pod-failure
mode: one
selector:
namespaces:
- production
labelSelectors:
app: api-server
duration: '5m'
Vigilmon runs continuously and will capture any incident caused by these scheduled experiments automatically. Review the incident log every Tuesday morning to verify resilience over time.
Step 9: Create a public status page
During chaos experiments, give your users and stakeholders visibility:
- Status Pages → New Status Page
- Name it "System Status"
- Add all your service monitors
- Publish
When experiments go as planned, the status page stays green. When an experiment reveals a real resilience gap, the status page reflects it immediately — giving your team a shared source of truth for incident response drills.
Interpreting Vigilmon data from chaos experiments
| Vigilmon outcome | What it means | |-----------------|---------------| | No incident during pod failure | Kubernetes load balancing is working; replicas absorb traffic | | Incident for exactly the pod restart duration | Single point of failure — increase replica count | | Incident longer than experiment duration | Service didn't recover automatically after chaos ended — investigate | | No incident during network partition | App handles DB loss gracefully (circuit breaker, read-through cache) | | Incident during network partition | No graceful degradation — implement circuit breakers or fallbacks |
What's next
- Chaos Workflows: chain multiple experiments in sequence and use Vigilmon data to gate progression — only move to the next experiment if the previous one shows green recovery
- Vigilmon SSL monitoring: add certificate expiry alerts for all domains in your chaos test scope
- Heartbeat monitors: verify your Chaos Mesh schedule controller is running — Vigilmon alerts you if scheduled experiments stop firing
Get started free at vigilmon.online — monitors go live in under a minute, no credit card required.