Monitoring Your APIs with Schemathesis and Vigilmon

Schemathesis finds bugs by generating hundreds of test cases from your OpenAPI spec and throwing them at your live API. But running it once in CI isn't enough. APIs drift. Deployments break things. Edge cases appear only in production data shapes.

By the end of this tutorial you'll have:

• Schemathesis running on a schedule against your live API
• Vigilmon alerts when your API fails spec conformance
• Heartbeat monitoring so you know when Schemathesis runs stop completing
• HTTP uptime monitoring for the API itself

Setup takes under 20 minutes.

Step 1: Install Schemathesis and run a baseline check

Schemathesis can test any API with an OpenAPI (Swagger) or GraphQL schema. Install it with pip:

pip install schemathesis

Run a quick test against your API:

# From a spec file
st run ./openapi.yaml --url http://api.yourdomain.com

# From a URL
st run https://api.yourdomain.com/openapi.json

# With authentication
st run https://api.yourdomain.com/openapi.json \
  --header "Authorization: Bearer $API_TOKEN"

Schemathesis generates hundreds of test cases per endpoint and reports any that produce:

• 5xx responses (server errors)
• Schema validation failures (response doesn't match spec)
• Unhandled exceptions

A successful run exits 0. Any conformance failure exits non-zero.

Step 2: Run Schemathesis as a scheduled monitoring job

The most powerful use of Schemathesis isn't one-shot CI — it's scheduled conformance monitoring against your staging or production API.

With GitHub Actions (scheduled):

# .github/workflows/schemathesis-monitor.yml
name: Schemathesis API Conformance Monitor

on:
  schedule:
    - cron: '0 */4 * * *'  # Every 4 hours
  workflow_dispatch:

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: '3.12'

      - name: Install Schemathesis
        run: pip install schemathesis

      - name: Run conformance checks
        run: |
          st run https://api.yourdomain.com/openapi.json \
            --header "Authorization: Bearer ${{ secrets.API_TOKEN }}" \
            --checks all \
            --max-examples 50 \
            --junit-xml schemathesis-results.xml
        continue-on-error: true
        id: schemathesis

      - name: Publish test results
        uses: mikepenz/action-junit-report@v4
        if: always()
        with:
          report_paths: schemathesis-results.xml

      - name: Ping Vigilmon heartbeat on success
        if: steps.schemathesis.outcome == 'success'
        run: curl -s "${{ secrets.VIGILMON_SCHEMATHESIS_HEARTBEAT }}"

      - name: Ping Vigilmon failure webhook on failure
        if: steps.schemathesis.outcome == 'failure'
        run: |
          curl -s -X POST "${{ secrets.VIGILMON_FAILURE_WEBHOOK }}" \
            -H "Content-Type: application/json" \
            -d '{"message": "Schemathesis conformance check failed"}'

With a cron script (self-hosted):

#!/bin/bash
# /usr/local/bin/schemathesis-check.sh

API_URL="${SCHEMATHESIS_API_URL}"
API_TOKEN="${API_TOKEN}"
HEARTBEAT_URL="${VIGILMON_SCHEMATHESIS_HEARTBEAT}"
RESULTS_DIR="/var/log/schemathesis"

mkdir -p "$RESULTS_DIR"
TIMESTAMP=$(date +%Y%m%d_%H%M%S)
RESULTS_FILE="${RESULTS_DIR}/run_${TIMESTAMP}.xml"

st run "$API_URL/openapi.json" \
  --header "Authorization: Bearer $API_TOKEN" \
  --checks all \
  --max-examples 30 \
  --junit-xml "$RESULTS_FILE"

EXIT_CODE=$?

if [ $EXIT_CODE -eq 0 ]; then
  curl -s "$HEARTBEAT_URL" > /dev/null
  echo "$(date): Schemathesis run passed"
else
  echo "$(date): Schemathesis run FAILED (exit $EXIT_CODE)"
fi

# Keep last 30 days of results
find "$RESULTS_DIR" -name "run_*.xml" -mtime +30 -delete

Add to cron:

0 */4 * * * /usr/local/bin/schemathesis-check.sh >> /var/log/schemathesis-cron.log 2>&1

Step 3: Set up HTTP monitoring for your API

Schemathesis tests spec conformance. But if your API is completely down, Schemathesis can't even start. Add a basic uptime monitor alongside the conformance monitoring.

In Vigilmon:

1. Sign up at vigilmon.online (free, no card required)
2. Click New Monitor → HTTP
3. Enter https://api.yourdomain.com/health (or your API root)
4. Set check interval: 1 minute (paid) or 5 minutes (free)
5. Under Expected status code, set 200
6. Save

Also monitor your OpenAPI spec URL directly:

https://api.yourdomain.com/openapi.json   → spec reachable (Schemathesis needs this)
https://api.yourdomain.com/health         → API alive

If the spec URL goes down, Schemathesis can't run — and you want to know that separately from an API outage.

Step 4: Set up Heartbeat monitoring for Schemathesis runs

The heartbeat monitor ensures your Schemathesis runs are completing on schedule. If GitHub Actions is disabled, the cron job crashes, or Schemathesis itself hangs, you want an alert — not just silence.

In Vigilmon:

1. Click New Monitor → Heartbeat
2. Name it Schemathesis conformance monitor
3. Set expected interval: 5 hours (slightly longer than your 4-hour schedule)
4. Copy the ping URL into VIGILMON_SCHEMATHESIS_HEARTBEAT

This heartbeat catches three failure modes:

• Schemathesis itself fails to run (CI disabled, cron broken)
• Schemathesis finds a conformance violation (exit non-zero, heartbeat not sent)
• The run hangs indefinitely and never completes

All three produce a missed heartbeat → alert.

Step 5: Focus Schemathesis on high-risk endpoints

Running all checks on all endpoints can take a long time and generate noise on complex APIs. Use Schemathesis filtering to focus on the endpoints that matter most:

# Only test write endpoints (POST, PUT, PATCH, DELETE)
st run https://api.yourdomain.com/openapi.json \
  --method POST PUT PATCH DELETE \
  --checks all \
  --max-examples 100

# Only test a specific path prefix
st run https://api.yourdomain.com/openapi.json \
  --endpoint "/v1/payments" \
  --checks all \
  --max-examples 200

# Use stateful testing for endpoint sequences
st run https://api.yourdomain.com/openapi.json \
  --stateful links \
  --checks all

Run quick sanity checks (low --max-examples) every 4 hours, and deep checks (high --max-examples, stateful) nightly. Use separate Vigilmon heartbeat monitors for each:

VIGILMON_SCHEMATHESIS_QUICK_HEARTBEAT   → interval 5h
VIGILMON_SCHEMATHESIS_DEEP_HEARTBEAT    → interval 25h

Step 6: Slack alerts and status page

Go to Notifications → New Channel → Slack in Vigilmon and add your webhook.

When a conformance check fails (heartbeat missed):

🔴 MISSED HEARTBEAT: Schemathesis conformance monitor
Last ping: 6 hours ago (expected every 5 hours)

This tells you: either the job isn't running, or the job ran and found violations.

Check the CI run or cron log to distinguish:

tail -n 50 /var/log/schemathesis-cron.log

For a public or team-visible status page:

1. Status Pages → New Status Page
2. Add: API health, spec URL check, Schemathesis heartbeat monitors
3. Share with your engineering team so they can self-diagnose before filing issues

What you've built

| What | How | |------|-----| | Spec conformance monitoring | Schemathesis on 4-hour schedule | | API uptime monitoring | Vigilmon HTTP monitor → /health | | Spec reachability check | Vigilmon HTTP monitor → /openapi.json | | Conformance run heartbeat | Vigilmon heartbeat — only pinged on success | | Deep check heartbeat | Separate heartbeat for nightly full run | | Slack alerts | Vigilmon Slack notification channel | | Team status page | Vigilmon public status page |

Schemathesis tells you your API is wrong. Vigilmon tells you Schemathesis stopped checking. Together they give you continuous confidence that your API matches its contract.

Next steps

• Add --checks not_a_server_error response_conformance to your quick checks for speed, and save all for the nightly run
• Store Schemathesis results in S3 or a shared location so the team can review failures without re-running
• Set up a separate Schemathesis run against your staging environment pre-deploy to catch regressions before they hit production

Get started free at vigilmon.online.