How to Monitor Load Test Results with Locust and Vigilmon

Load testing and uptime monitoring solve different problems but need to work together. Locust is a Python-based open-source load testing framework that simulates thousands of concurrent users hitting your application. Vigilmon is an external uptime monitoring service that watches your endpoints from the outside world. Together, they answer two complementary questions: "How does my service behave under pressure?" (Locust) and "Is my service actually reachable and within SLA?" (Vigilmon).

This tutorial walks you through setting up Locust load tests, running them against a staging or production environment, and using Vigilmon to observe the impact from an external perspective — the same vantage point your real users have.

Why load testing needs external monitoring

Most load testing setups measure from the inside: Locust reports the response times and failure rates it observes as the load generator. But that's not the full picture:

• CDN caching — your load test might hit your origin while real users hit a cache; Vigilmon checks the same endpoint your users hit
• Firewall and rate limiting — your load test traffic might be allowlisted; external monitoring isn't
• Geographic variation — Vigilmon monitors from multiple regions; Locust typically runs from one location
• Post-load recovery — Locust stops and goes silent; Vigilmon continues watching to see if your service recovers cleanly

Running both in parallel gives you a complete performance profile.

What you'll need

• Python 3.8+ and pip
• A target application with HTTP endpoints
• A free Vigilmon account
• At least 2 GB RAM for the Locust worker (more for high concurrency)

Step 1: Install and configure Locust

pip install locust

Create a basic load test file locustfile.py:

from locust import HttpUser, task, between, events
from locust.runners import MasterRunner
import requests
import os

VIGILMON_API_KEY = os.getenv("VIGILMON_API_KEY", "")
VIGILMON_MONITOR_ID = os.getenv("VIGILMON_MONITOR_ID", "")

class WebsiteUser(HttpUser):
    wait_time = between(1, 3)  # Random wait between requests

    @task(3)
    def view_homepage(self):
        self.client.get("/")

    @task(2)
    def view_products(self):
        self.client.get("/products")

    @task(1)
    def health_check(self):
        with self.client.get("/health", catch_response=True) as response:
            if response.status_code != 200:
                response.failure(f"Health check failed: {response.status_code}")
            else:
                response.success()

    @task(1)
    def api_users(self):
        with self.client.get("/api/users", catch_response=True) as response:
            if response.elapsed.total_seconds() > 2.0:
                response.failure(f"Response too slow: {response.elapsed.total_seconds():.2f}s")
            else:
                response.success()

The @task weight values control how often each endpoint is hit — a weight of 3 means it's called 3x as often as a weight of 1. Adjust these to match your actual traffic distribution.

Step 2: Run a baseline load test

Before stressing your system, establish a baseline at low concurrency:

locust --headless \
  --users 10 \
  --spawn-rate 2 \
  --run-time 2m \
  --host https://your-app.example.com \
  --csv baseline \
  -f locustfile.py

Key flags:

• --users 10 — maximum concurrent users
• --spawn-rate 2 — users added per second during ramp-up
• --run-time 2m — test duration
• --csv baseline — write results to CSV files

Review baseline_stats.csv to establish your baseline:

Type,Name,Request Count,Failure Count,Median (ms),Average (ms),Min (ms),Max (ms),Average Content Size,Requests/s,...
GET,/,1203,0,45,48,22,312,2048,10.02,...
GET,/health,401,0,12,14,8,89,32,3.34,...
GET,/api/users,398,0,23,27,15,201,1024,3.31,...

Note your baseline median response times and failure rate (should be 0%). These are your SLOs.

Step 3: Set up Vigilmon monitors before load testing

Set up external monitors to observe your service throughout the load test:

Monitor 1: Primary health endpoint

1. Log in to vigilmon.online → Monitors → New Monitor
2. Type: HTTP / HTTPS
3. URL: https://your-app.example.com/health
4. Expected status: 200
5. Interval: 1 minute (or lowest available on your plan)
6. Enable response time tracking
7. Save and note the monitor ID from the URL

Monitor 2: Key API endpoint

Type: HTTP / HTTPS
URL: https://your-app.example.com/api/users
Expected status: 200
Interval: 1 minute

Monitor 3: TCP port

Type: TCP Port
Host: your-app.example.com
Port: 443
Interval: 1 minute

Let the monitors run for at least 5 minutes before starting any load test to collect a clean baseline in Vigilmon's charts.

Step 4: Integrate Locust with Vigilmon via the API

Use Locust's event hooks to annotate Vigilmon when load tests start and stop. This makes it trivial to correlate response time changes in Vigilmon with the test timeline:

# locustfile.py — with Vigilmon integration

from locust import HttpUser, task, between, events
from locust.runners import MasterRunner
import requests
import os
import time

VIGILMON_API_KEY = os.getenv("VIGILMON_API_KEY", "")
VIGILMON_BASE = "https://vigilmon.online/api/v1"

def annotate_vigilmon(title, message, monitor_ids):
    if not VIGILMON_API_KEY:
        return
    try:
        requests.post(
            f"{VIGILMON_BASE}/events",
            headers={"Authorization": f"Bearer {VIGILMON_API_KEY}"},
            json={
                "title": title,
                "description": message,
                "monitor_ids": monitor_ids,
                "type": "info"
            },
            timeout=5
        )
    except Exception:
        pass  # Don't let annotation failures break the load test

MONITOR_IDS = os.getenv("VIGILMON_MONITOR_IDS", "").split(",")

@events.test_start.add_listener
def on_test_start(environment, **kwargs):
    user_count = environment.parsed_options.num_users if environment.parsed_options else "?"
    annotate_vigilmon(
        title=f"Locust load test started — {user_count} users",
        message=f"Load test beginning at {time.strftime('%Y-%m-%dT%H:%M:%SZ', time.gmtime())}",
        monitor_ids=MONITOR_IDS
    )

@events.test_stop.add_listener
def on_test_stop(environment, **kwargs):
    stats = environment.stats
    total_reqs = stats.total.num_requests
    total_fails = stats.total.num_failures
    fail_rate = (total_fails / total_reqs * 100) if total_reqs > 0 else 0
    annotate_vigilmon(
        title=f"Locust load test ended — {fail_rate:.1f}% failure rate",
        message=f"Completed {total_reqs} requests, {total_fails} failures",
        monitor_ids=MONITOR_IDS
    )

class WebsiteUser(HttpUser):
    wait_time = between(1, 3)

    @task(3)
    def view_homepage(self):
        self.client.get("/")

    @task(2)
    def api_users(self):
        with self.client.get("/api/users", catch_response=True) as r:
            if r.elapsed.total_seconds() > 2.0:
                r.failure(f"Slow: {r.elapsed.total_seconds():.2f}s")

Run with:

VIGILMON_API_KEY=your-key \
VIGILMON_MONITOR_IDS=mon_health,mon_api \
locust --headless -u 100 -r 10 -t 5m --host https://your-app.example.com

Step 5: Run a ramp-up load test

The most revealing load test gradually increases concurrency until you find the breaking point:

# ramp_up_locustfile.py
from locust import HttpUser, task, between
from locust.shape import LoadTestShape

class StagesShape(LoadTestShape):
    """
    Ramp up to 500 users over 10 minutes, hold for 5 minutes, then ramp down.
    """
    stages = [
        {"duration": 60,  "users": 50,  "spawn_rate": 5},
        {"duration": 180, "users": 150, "spawn_rate": 5},
        {"duration": 300, "users": 300, "spawn_rate": 10},
        {"duration": 600, "users": 500, "spawn_rate": 20},
        {"duration": 900, "users": 500, "spawn_rate": 1},  # Hold at peak
        {"duration": 960, "users": 0,   "spawn_rate": 50}, # Ramp down
    ]

    def tick(self):
        run_time = self.get_run_time()
        for stage in self.stages:
            if run_time < stage["duration"]:
                return stage["users"], stage["spawn_rate"]
        return None

class AppUser(HttpUser):
    wait_time = between(1, 5)

    @task
    def get_homepage(self):
        self.client.get("/")

    @task
    def get_health(self):
        self.client.get("/health")

Run with:

locust -f ramp_up_locustfile.py --headless --host https://your-app.example.com --csv ramp_test

Watch Vigilmon's response time graph simultaneously. You'll typically see:

1. Normal range (0-150 users): Response times match baseline
2. Degradation onset (150-300 users): Medians start climbing, P95 spikes
3. SLA breach (300+ users): P99 > 2s, first Vigilmon alert may trigger
4. Recovery (after ramp-down): Response times return to baseline

The exact numbers depend on your infrastructure, but the shape is consistent.

Step 6: Interpret Locust + Vigilmon results together

Response time comparison

| Metric | Locust P50 | Locust P95 | Vigilmon avg | Assessment | |--------|-----------|-----------|--------------|------------| | Baseline | 45ms | 120ms | 52ms | Normal | | 100 users | 67ms | 189ms | 71ms | Acceptable | | 300 users | 312ms | 1,240ms | 380ms | Degraded | | 500 users | 2,100ms | 8,400ms | Incident triggered | SLA breached |

The Vigilmon external check lags slightly behind Locust (it checks once per minute rather than continuously) but is a better proxy for user experience because it's independent of the load generator.

Failure analysis

Check Locust's failure breakdown:

cat ramp_test_failures.csv

Common failure patterns:

• ConnectionError — server overwhelmed, refusing new connections
• ReadTimeout — server accepted the connection but didn't respond in time
• HTTPError 500 — server is up but application errors are occurring
• HTTPError 429 — rate limiting is kicking in (this is good — it means rate limiting works)

Cross-reference Vigilmon incidents for the same time window. If Locust sees 500s but Vigilmon doesn't report an incident, your load test traffic is taking a different code path from Vigilmon's probe traffic.

Step 7: Set up Locust in distributed mode for high concurrency

For very high concurrency (1,000+ users), run Locust in distributed mode with multiple workers:

# On the master node
locust -f locustfile.py --master --host https://your-app.example.com

# On each worker node (run this on multiple machines or containers)
locust -f locustfile.py --worker --master-host master-node-ip

Or use Docker Compose:

version: "3"
services:
  master:
    image: locustio/locust
    ports:
      - "8089:8089"
    volumes:
      - ./locustfile.py:/home/locust/locustfile.py
    command: -f /home/locust/locustfile.py --master

  worker:
    image: locustio/locust
    volumes:
      - ./locustfile.py:/home/locust/locustfile.py
    command: -f /home/locust/locustfile.py --worker --master-host master
    deploy:
      replicas: 4  # 4 workers, scale as needed

docker-compose up --scale worker=4

Access the Locust web UI at http://master-node:8089 to monitor the distributed test in real time.

Step 8: Automate load tests in CI/CD

Integrate Locust into your CI pipeline to catch performance regressions before deployment:

# .github/workflows/load-test.yml
name: Load Test

on:
  push:
    branches: [main]

jobs:
  load-test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      
      - name: Install Locust
        run: pip install locust
      
      - name: Run load test
        env:
          VIGILMON_API_KEY: ${{ secrets.VIGILMON_API_KEY }}
          VIGILMON_MONITOR_IDS: ${{ secrets.VIGILMON_MONITOR_IDS }}
        run: |
          locust --headless \
            -u 50 -r 5 -t 2m \
            --host ${{ secrets.STAGING_URL }} \
            --csv ci_results \
            --exit-code-on-error 1
      
      - name: Check failure rate
        run: |
          python3 - << 'EOF'
          import csv
          with open("ci_results_stats.csv") as f:
              reader = csv.DictReader(f)
              for row in reader:
                  if row["Name"] == "Aggregated":
                      reqs = int(row["Request Count"])
                      fails = int(row["Failure Count"])
                      fail_rate = fails / reqs if reqs > 0 else 0
                      if fail_rate > 0.01:  # Fail CI if > 1% error rate
                          print(f"FAIL: {fail_rate:.1%} error rate exceeds 1% threshold")
                          exit(1)
                      avg_ms = float(row["Average (ms)"])
                      if avg_ms > 500:  # Fail CI if avg > 500ms
                          print(f"FAIL: {avg_ms:.0f}ms average exceeds 500ms SLO")
                          exit(1)
                      print(f"PASS: {fail_rate:.2%} errors, {avg_ms:.0f}ms avg")
          EOF
      
      - name: Upload results
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: locust-results
          path: ci_results_*.csv

Common Locust issues

WebSocket or SSE connections:

• Use locust.contrib.fasthttp.FastHttpUser for WebSocket load testing
• For SSE, use the sseclient library in your task

High memory usage on workers:

• Locust holds connection state per user; 10,000 users requires significant RAM
• Use --processes to fork multiple OS processes per worker node

Results look too good (Locust running on same host as target):

• Always run Locust on a separate machine from the target
• Shared resources (CPU, network) artificially help the target

SSL certificate errors:

• Add --insecure flag for staging environments with self-signed certs

Summary

You've deployed Locust for graduated load testing, integrated Vigilmon as an independent external observer, scripted Vigilmon annotations to mark test boundaries, and connected the two in a CI pipeline that fails builds when performance regressions are detected.

Locust tells you how your system behaves from the load generator's perspective. Vigilmon tells you how your users experience the same events from outside. Together they give you a complete picture of performance under pressure — not just peak throughput numbers, but real user experience data at every load level.

Start with a free Vigilmon account and run your first monitored load test in minutes.