Monitoring Toxiproxy with Vigilmon: API Health & CI/CD Integration

Toxiproxy is a TCP proxy designed to simulate network failures in automated tests. You configure proxies and inject "toxics" (latency, connection drops, bandwidth limits) to verify your application handles real-world network conditions correctly. When Toxiproxy is part of your CI/CD pipeline, a failed or unreachable Toxiproxy instance causes confusing test failures that look like application bugs. Vigilmon lets you confirm that Toxiproxy itself is healthy before your test suite runs. This tutorial wires Toxiproxy into Vigilmon for uptime monitoring and alerts in about 15 minutes.

What You'll Build

A Vigilmon HTTP monitor on Toxiproxy's /version endpoint
A Vigilmon HTTP monitor on Toxiproxy's /proxies endpoint
A CI/CD pre-flight check that polls Vigilmon before running tests
Email and webhook alerts when Toxiproxy is unreachable

Prerequisites

Toxiproxy 2.x (github.com/Shopify/toxiproxy)
A CI/CD environment (GitHub Actions, GitLab CI, or any pipeline)
A free Vigilmon account

Step 1: Understand Toxiproxy's Built-in API

Toxiproxy exposes an HTTP management API (default port 8474) with several built-in endpoints you can use as health check targets:

| Endpoint | Description | |---|---| | GET /version | Returns the running Toxiproxy version string | | GET /proxies | Returns all configured proxies and their status | | POST /proxies | Create a proxy | | GET /proxies/{name} | Get a specific proxy | | DELETE /proxies/{name} | Delete a proxy | | POST /reset | Reset all toxics |

The /version endpoint is the lightest possible health check — no side effects, no authentication required:

curl -s http://localhost:8474/version
# "2.9.0"

The /proxies endpoint gives you richer status including whether each proxy is enabled:

curl -s http://localhost:8474/proxies | python -m json.tool
# {
#   "myapp_redis": {
#     "name": "myapp_redis",
#     "listen": "0.0.0.0:6380",
#     "upstream": "localhost:6379",
#     "enabled": true,
#     "toxics": []
#   }
# }

Both endpoints return 200 OK when Toxiproxy is healthy. Any connection failure or non-2xx response means Toxiproxy is down.

Step 2: Create a Vigilmon HTTP Monitor for `/version`

| Field | Value | |---|---| | URL | http://toxiproxy.internal:8474/version | | Method | GET | | Check interval | 60 seconds | | Expected status | 200 | | Timeout | 5 seconds | | Regions | Use your nearest region |

Note: If your Toxiproxy runs on an internal/private network, use Vigilmon's private location agent installed in your network, or expose the management port through a VPN-accessible IP.

Click Save. Vigilmon immediately starts polling. Within one minute you'll see the first green tick.

Why /version and not /proxies? /version has no payload parsing overhead and is always safe to call — it's the Toxiproxy equivalent of GET /ping. Use /proxies as a second monitor only if you want to verify that specific proxy configurations are present.

Step 3: Create a Second Monitor for `/proxies` (Optional)

If your CI/CD pipeline depends on specific proxies being pre-configured (e.g. in a shared Toxiproxy instance used across teams), add a second monitor for /proxies:

| Field | Value | |---|---| | URL | http://toxiproxy.internal:8474/proxies | | Method | GET | | Check interval | 120 seconds | | Expected status | 200 | | Response body contains | "enabled": true |

The Response body contains assertion fails if all proxies are disabled or the expected proxy name is missing, catching misconfiguration in addition to connectivity failures.

Step 4: CI/CD Pre-flight Check

The most valuable Toxiproxy monitoring in a CI/CD context is a pre-flight check at the start of each test job. This fails fast with a clear error message rather than letting tests run with a broken network proxy and producing cryptic failures.

GitHub Actions

# .github/workflows/integration-tests.yml
name: Integration Tests

on: [push, pull_request]

jobs:
  test:
    runs-on: ubuntu-latest

    services:
      toxiproxy:
        image: ghcr.io/shopify/toxiproxy:2.9.0
        ports:
          - 8474:8474
          - 6380:6380  # example proxied Redis port

    steps:
      - uses: actions/checkout@v4

      - name: Wait for Toxiproxy to be ready
        run: |
          echo "Waiting for Toxiproxy management API..."
          for i in $(seq 1 30); do
            if curl -sf http://localhost:8474/version > /dev/null 2>&1; then
              echo "Toxiproxy is ready (version: $(curl -s http://localhost:8474/version))"
              break
            fi
            echo "  attempt $i/30 — sleeping 2s"
            sleep 2
          done
          # Fail the job if Toxiproxy never came up
          curl -sf http://localhost:8474/version || (echo "Toxiproxy failed to start" && exit 1)

      - name: Configure Toxiproxy proxies
        run: |
          curl -s -X POST http://localhost:8474/proxies \
            -H "Content-Type: application/json" \
            -d '{"name":"redis","listen":"0.0.0.0:6380","upstream":"localhost:6379","enabled":true}'

      - name: Run integration tests
        run: go test ./... -tags=integration
        env:
          REDIS_URL: redis://localhost:6380

GitLab CI

# .gitlab-ci.yml
integration-tests:
  image: golang:1.21
  services:
    - name: ghcr.io/shopify/toxiproxy:2.9.0
      alias: toxiproxy
  script:
    - |
      echo "Waiting for Toxiproxy..."
      for i in $(seq 1 30); do
        curl -sf http://toxiproxy:8474/version && break || sleep 2
      done
    - curl -sf http://toxiproxy:8474/version || exit 1
    - go test ./... -tags=integration
  variables:
    TOXIPROXY_URL: http://toxiproxy:8474

Step 5: Heartbeat for Long-Running Toxiproxy Instances

For shared or persistent Toxiproxy instances (e.g. in a staging environment that runs 24/7), add a Vigilmon heartbeat monitor so you're alerted if the process dies between CI runs.

Get your Heartbeat URL from Vigilmon (Dashboard → Heartbeat Monitors → New):

https://vigilmon.online/api/heartbeats/YOUR-UUID/ping

Set the period to 5 minutes (for a polling interval of 3 minutes).

Send the heartbeat via a cron job on the host running Toxiproxy:

# /etc/cron.d/toxiproxy-heartbeat
*/3 * * * * root curl -sf http://localhost:8474/version && curl -sf https://vigilmon.online/api/heartbeats/YOUR-UUID/ping

This two-step command only pings Vigilmon if Toxiproxy's /version endpoint responds successfully — combining uptime verification and heartbeat in a single cron line. If Toxiproxy dies, the first curl fails, the second never runs, and Vigilmon alerts you within 5 minutes.

Step 6: Alert Routing

Email Alerts

Configured in Step 2. Vigilmon alerts when:

The /version or /proxies endpoint returns non-2xx
The endpoint is unreachable within the timeout
The heartbeat window expires (for persistent instances)

Webhook Alerts for CI/CD Systems

Instead of email, you can route Toxiproxy alerts to a Slack channel or a custom webhook that opens a ticket:

In Vigilmon → Alert Channels → Add Channel → Webhook
Paste your Slack incoming webhook URL (or custom endpoint)
Assign to your Toxiproxy monitors

Alert payload:

{
  "text": "🔴 *toxiproxy.internal:8474/version* is DOWN\nStatus: connection refused | Region: eu-west-1\nDuration: 1m 02s"
}

Step 7: Test the Full Loop

Simulate Toxiproxy down: stop the Toxiproxy process and curl /version — expect connection refused.
Verify Vigilmon detects it: within one check interval the dashboard goes red and an alert fires.
Test heartbeat silence (persistent setups): stop Toxiproxy and wait for the cron heartbeat to miss a beat — you receive an alert within the heartbeat period.
Recover: restart Toxiproxy. Vigilmon auto-recovers and sends a "back online" notification.

Production Checklist

[ ] Vigilmon HTTP monitor on /version (primary liveness check)
[ ] Optional: monitor on /proxies for proxy configuration validation
[ ] CI/CD pre-flight check waits for Toxiproxy before running tests
[ ] Heartbeat monitor configured for persistent Toxiproxy instances
[ ] Alert channels tested (email and/or Slack/webhook)
[ ] Toxiproxy management port firewalled — only accessible from CI runners and monitoring agents

Wrapping Up

You now have Toxiproxy covered from two angles:

Uptime monitoring that catches Toxiproxy API failures within 60 seconds, surfacing network proxy issues before they silently fail your tests
CI/CD pre-flight checks that fast-fail jobs with a clear error rather than cascading test failures
Heartbeat monitoring for persistent Toxiproxy instances that should stay up between CI runs

If you hit issues or have questions, drop a comment below. Happy monitoring!