tutorial

How to Monitor ChaosBlade Chaos Engineering Experiments with Vigilmon

ChaosBlade is a chaos engineering platform for cloud-native infrastructure. It injects failures into Kubernetes pods, nodes, containers, and network layers t...

ChaosBlade is a chaos engineering platform for cloud-native infrastructure. It injects failures into Kubernetes pods, nodes, containers, and network layers to help you validate that your systems behave correctly under real-world fault conditions. But chaos engineering is only useful if you can observe the impact accurately — and for that you need external monitoring that runs independently of the system under test.

This tutorial shows how to use Vigilmon alongside ChaosBlade experiments to measure real-world availability impact, validate your recovery mechanisms, and build confidence that your chaos library is teaching you true things about your system.

The monitoring gap in chaos engineering

ChaosBlade excels at injecting faults — CPU pressure, network latency, pod kill, DNS failure, and dozens of other scenarios. What it doesn't tell you is:

  • What users experienced — was the service actually unreachable from outside the cluster, or did Kubernetes recover fast enough to be invisible?
  • How long the impact lasted — ChaosBlade records when the experiment ran, not when external connectivity degraded and recovered
  • Whether your SLO was breached — only external monitoring with time-stamped incident windows can answer this
  • Whether your mitigation actually worked — killing a pod isn't meaningful unless you verify that traffic recovered within your SLA window

Vigilmon runs continuously from multiple geographic regions, independent of your cluster's internal state. When a ChaosBlade experiment hits your service, Vigilmon shows you the exact external impact in real time.

What you'll need

  • A Kubernetes cluster with ChaosBlade installed
  • At least one service with external HTTP or TCP accessibility
  • A free Vigilmon account

Step 1: Baseline your services before running experiments

Before injecting any chaos, establish a clean baseline in Vigilmon:

  1. Log in to vigilmon.online and go to Monitors → New Monitor
  2. Choose HTTP / HTTPS
  3. Set the URL to your primary service endpoint:
    • https://api.yourdomain.com/health
  4. Check interval: 30 seconds (use a shorter interval during chaos experiments for finer resolution)
  5. Expected status: 200
  6. Save

Run the monitor for at least 24 hours before your first chaos experiment. This gives you a baseline response-time histogram and confirms zero false positives in your environment.

Services to monitor

# List all externally accessible services
kubectl get svc -A --field-selector spec.type!=ClusterIP

# Get Ingress endpoints
kubectl get ingress -A

# Get external IPs for LoadBalancer services
kubectl get svc -A --field-selector spec.type=LoadBalancer \
  -o jsonpath='{range .items[*]}{.metadata.namespace}/{.metadata.name}: {.status.loadBalancer.ingress[0].ip}{"\n"}{end}'

Create an HTTP monitor for each critical service you plan to test with ChaosBlade.

Step 2: Add TCP monitoring for network-layer experiments

ChaosBlade supports network-layer experiments (latency injection, packet loss, DNS failure). TCP monitoring in Vigilmon catches these failures at the transport layer — independently of what the application reports:

  1. Go to Monitors → New Monitor → TCP Port
  2. Host: your service domain or IP
  3. Port: 443 for HTTPS, 80 for HTTP, or your application port
  4. Interval: 30 seconds
  5. Save

When you run a ChaosBlade network latency experiment, Vigilmon's TCP monitor will show increased response times before the HTTP health check starts failing — giving you a leading indicator.

Step 3: Run a ChaosBlade experiment with active monitoring

Here's a workflow for running a controlled chaos experiment while Vigilmon monitors impact:

Experiment 1: Pod kill resilience

# chaosblade-pod-kill.yaml
apiVersion: chaosblade.io/v1alpha1
kind: ChaosBlade
metadata:
  name: pod-kill-experiment
spec:
  experiments:
    - scope: pod
      target: pod
      action: delete
      desc: "Kill one pod from my-api deployment"
      matchers:
        - name: labels
          value:
            - "app=my-api"
        - name: namespace
          value:
            - "production"
        - name: evict-count
          value:
            - "1"

Apply the experiment and watch Vigilmon:

# Apply the experiment
kubectl apply -f chaosblade-pod-kill.yaml

# Watch experiment status
kubectl get cb pod-kill-experiment -w

# Watch Vigilmon (open the dashboard in your browser)
# You should see: brief latency spike → HTTP 200 resumes → incident window closes

# After verifying recovery, destroy the experiment
kubectl delete -f chaosblade-pod-kill.yaml

Vigilmon's incident history gives you the exact impact window. If the HTTP monitor went red for 45 seconds, that's 45 seconds of external user impact.

Experiment 2: Network latency injection

# chaosblade-network-delay.yaml
apiVersion: chaosblade.io/v1alpha1
kind: ChaosBlade
metadata:
  name: network-delay-experiment
spec:
  experiments:
    - scope: pod
      target: network
      action: delay
      desc: "Inject 200ms latency on egress traffic"
      matchers:
        - name: labels
          value:
            - "app=my-api"
        - name: namespace
          value:
            - "production"
        - name: interface
          value:
            - "eth0"
        - name: time
          value:
            - "200"
        - name: offset
          value:
            - "50"

With latency injection active, Vigilmon's response time chart will show the exact latency added. Compare before, during, and after:

kubectl apply -f chaosblade-network-delay.yaml
# Observe response time increase in Vigilmon dashboard
# (visible under Monitor → Response Time History)
kubectl delete -f chaosblade-network-delay.yaml
# Confirm response times return to baseline

Experiment 3: CPU resource pressure

# chaosblade-cpu-burn.yaml
apiVersion: chaosblade.io/v1alpha1
kind: ChaosBlade
metadata:
  name: cpu-burn-experiment
spec:
  experiments:
    - scope: pod
      target: cpu
      action: fullload
      desc: "Saturate one CPU core on api pods"
      matchers:
        - name: labels
          value:
            - "app=my-api"
        - name: namespace
          value:
            - "production"
        - name: cpu-count
          value:
            - "1"

Vigilmon's HTTP response time monitoring shows whether CPU saturation degrades user-facing latency and by how much.

Step 4: Automate experiment-to-monitoring correlation

Use Vigilmon's maintenance window feature to mark periods when chaos experiments are running, so planned degradations don't trigger on-call pages:

# Create a maintenance window via Vigilmon API before starting an experiment
VIGILMON_API_KEY="your-api-key"
START=$(date -u +%Y-%m-%dT%H:%M:%SZ)
END=$(date -u -d '+30 minutes' +%Y-%m-%dT%H:%M:%SZ)

curl -X POST "https://vigilmon.online/api/v1/maintenance" \
  -H "Authorization: Bearer $VIGILMON_API_KEY" \
  -H "Content-Type: application/json" \
  -d "{
    \"name\": \"ChaosBlade pod-kill experiment\",
    \"start\": \"$START\",
    \"end\": \"$END\",
    \"monitor_ids\": [\"mon_abc123\"]
  }"

# Run the experiment
kubectl apply -f chaosblade-pod-kill.yaml
sleep 300  # Let it run for 5 minutes

# Collect results
kubectl get cb pod-kill-experiment -o json | jq '.status'

# Destroy experiment
kubectl delete -f chaosblade-pod-kill.yaml

# Close the maintenance window early if recovery was faster than expected

This pattern gives you:

  • A clean incident history (planned experiments don't create false outage records)
  • Accurate SLO calculations (exclude experiment windows from uptime percentage)
  • Auditable chaos logs (maintenance window names document what was tested and when)

Step 5: Heartbeat monitoring for experiment orchestration

ChaosBlade experiments are typically orchestrated by a controller or CI/CD pipeline. Vigilmon heartbeat monitors verify that your chaos orchestration is running correctly:

  1. Go to Monitors → New Monitor → Heartbeat
  2. Name: chaosblade-controller
  3. Expected interval: 60 minutes (if your chaos scheduler runs hourly)
  4. Copy the heartbeat URL

In your chaos orchestration script:

#!/bin/bash
# chaos-orchestrator.sh — runs scheduled chaos experiments

echo "[$(date)] Starting scheduled chaos run"

# Run experiment
kubectl apply -f experiments/pod-kill-canary.yaml
sleep 120  # 2 minute experiment window

# Collect and log results
kubectl get cb pod-kill-canary -o json > /var/log/chaos/$(date +%Y%m%d-%H%M%S).json

# Destroy experiment
kubectl delete -f experiments/pod-kill-canary.yaml

# Signal Vigilmon that orchestration completed successfully
curl -sf https://vigilmon.online/heartbeat/YOUR_TOKEN || \
  echo "WARNING: Failed to ping Vigilmon heartbeat"

echo "[$(date)] Chaos run complete"

If your chaos orchestrator crashes or the experiment controller fails, Vigilmon alerts you — preventing silent "chaos theater" where experiments stop running without anyone noticing.

Step 6: Configure alert channels for chaos experiments

Even during planned experiments, you want real-time visibility into impact:

  1. Go to Alert Channels → Add Channel → Webhook
  2. Use a dedicated Slack channel for chaos notifications (not your production alerts channel)
{
  "monitor_name": "api.yourdomain.com health",
  "status": "down",
  "url": "https://api.yourdomain.com/health",
  "started_at": "2026-01-15T14:30:00Z",
  "duration_seconds": 45
}

Route experiment-window alerts to a #chaos-lab channel and production alerts to #incidents:

def route_alert(alert):
    if is_maintenance_window_active():
        post_to_slack("#chaos-lab", f"[EXPECTED] {alert['monitor_name']} down for {alert['duration_seconds']}s")
    else:
        post_to_slack("#incidents", f"[UNEXPECTED] {alert['monitor_name']} down — page on-call")
        trigger_pagerduty(alert)

Step 7: Build a chaos experiment status page

Track your chaos experiment outcomes and service resilience on a shared status page:

  1. Go to Status Pages → New Status Page
  2. Name: Chaos Engineering Dashboard
  3. Add monitors: all production service HTTP and TCP monitors
  4. Enable incident history (shows the impact windows from past experiments)
  5. Publish (internal URL, not public)

Your team can review the incident history to see which experiments caused user impact, how long recovery took, and whether recent changes improved resilience.

Chaos experiment monitoring matrix

| Experiment | Monitor type | What to watch | |-----------|-------------|----------------| | Pod kill | HTTP 30s interval | Incident duration = time to reschedule + pass readiness probe | | Network delay | HTTP + TCP | Response time increase, threshold breach | | CPU fullload | HTTP 30s interval | Latency degradation, eventual timeout | | DNS failure | HTTP | Total outage until DNS caches expire | | Node failure | HTTP + TCP per node | Which services are single-node and vulnerable | | Memory pressure | HTTP | OOMKill events triggering pod restarts |

Validating your chaos hypotheses

A good chaos experiment has a stated hypothesis and a measurable outcome. Vigilmon provides the measurement:

Hypothesis: "Killing one of three API pods causes no user-visible downtime due to Kubernetes rescheduling."

Measurement from Vigilmon:

  • HTTP monitor shows 200 continuously → hypothesis confirmed
  • HTTP monitor shows a 12-second gap → hypothesis partially confirmed (12s of impact), improve readiness probe timing
  • HTTP monitor shows a 90-second gap → hypothesis refuted, investigate HPA and rescheduling behavior

Without external monitoring, you're running chaos experiments in the dark.

What's next

  • Automated chaos pipelines — integrate Vigilmon incident window data into your CI/CD pipeline to automatically fail a release if a canary chaos experiment causes more than N seconds of external impact
  • SLO burn rate — use Vigilmon's uptime history to calculate SLO consumption per chaos experiment, and pause experimentation when you're close to burning through your error budget
  • Steady-state hypothesis testing — run Vigilmon's response time trend reports before and after experiments to verify that repeated chaos isn't slowly degrading your baseline performance

Get started free at vigilmon.online — no credit card, monitors are live in under a minute.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles