tutorial

Monitoring Tracetest with Vigilmon: API Health, Web UI Availability, Test Runner Endpoint & SSL Certificate Alerts

How to monitor Tracetest with Vigilmon — API health checks, web UI availability, test trigger endpoint monitoring, and SSL certificate alerts for trace-based testing.

Tracetest is the distributed tracing-based testing tool that lets backend and SRE teams write assertions against OpenTelemetry trace data — validating that services produce the spans, attributes, and timing relationships they are supposed to produce rather than just checking HTTP response codes. When Tracetest's API server goes down, automated trace-based test runs stop triggering, and engineers lose the ability to validate that distributed transactions flow correctly across services. When the OTLP trace ingestion endpoint is unreachable, tests cannot collect the trace data they need to evaluate assertions. Vigilmon gives you external visibility into Tracetest's availability: the API server health endpoint, web UI, test runner API, trace ingestion endpoint, and SSL certificate expiry.

What You'll Build

  • A monitor on Tracetest's API health endpoint to detect server failures
  • A web UI availability check confirming the Tracetest dashboard is accessible
  • A test trigger API endpoint check to verify automated test runs can be initiated
  • An OTLP trace ingestion endpoint check to confirm traces can reach Tracetest
  • An SSL certificate monitor for your Tracetest domain

Prerequisites

  • Tracetest installed (Docker Compose, Kubernetes via Helm, or Tracetest Cloud)
  • Tracetest API server and web UI exposed externally with a DNS name
  • An OpenTelemetry Collector or direct OTLP instrumentation sending traces to Tracetest
  • A free account at vigilmon.online

Step 1: Understand Tracetest's Health Endpoints

Tracetest's API server exposes a health endpoint for liveness and readiness checks:

curl http://tracetest.example.com:11633/api/health
# Returns: {"status":"pass"}

For a TLS-enabled installation:

curl https://tracetest.example.com/api/health
# Returns HTTP 200 with {"status":"pass"}

A healthy Tracetest API server returns 200 with {"status":"pass"}. If the API server process is unresponsive or the underlying PostgreSQL database connection is broken, you receive a connection error, 502, or a response with "status":"fail".

Tracetest also exposes a readiness endpoint that confirms database connectivity:

curl https://tracetest.example.com/api/ready
# Returns: {"status":"ready"} when fully initialized

Step 2: Create a Vigilmon HTTP Monitor for the API Health Endpoint

  1. Log in to Vigilmon → Add Monitor → HTTP.
  2. URL: https://tracetest.example.com/api/health (use http://tracetest.example.com:11633/api/health if not behind TLS termination).
  3. Check interval: 60 seconds.
  4. Response timeout: 15 seconds.
  5. Expected status: 200.
  6. Keyword: pass (present in the response body when the server is healthy).
  7. Click Save.

This monitor catches:

  • Tracetest API server process crashes or out-of-memory kills
  • PostgreSQL database connection failures that prevent test result storage
  • Trace store (Jaeger, Tempo, OpenSearch) connectivity failures
  • Container restarts during configuration changes
  • Upgrade failures that leave Tracetest in a degraded state

Alert sensitivity: Set to trigger after 1 consecutive failure. When Tracetest's API is down, no automated test runs can be triggered and no trace assertions can be evaluated — breaking the entire trace-based testing pipeline.

Step 3: Monitor the Tracetest Web UI

The Tracetest web UI is where SRE and backend engineers define test cases, write trace assertions, review test run results, and inspect individual span data. A UI outage can prevent test configuration changes even when the backend API is functioning:

curl https://tracetest.example.com
# Returns HTML containing "Tracetest" in the page content
  1. Add Monitor → HTTP.
  2. URL: https://tracetest.example.com.
  3. Check interval: 2 minutes.
  4. Response timeout: 20 seconds.
  5. Expected status: 200.
  6. Keyword: Tracetest (appears in the web UI HTML and page title).
  7. Label: Tracetest web UI.
  8. Click Save.

When the web UI monitor fires but the /api/health monitor is green, the problem is in the React frontend serving layer — a static asset issue, a misconfigured Nginx reverse proxy, or an ingress rule change that broke the UI path while leaving the /api/* routes intact. Engineers can still run tests via the Tracetest CLI while the UI is being restored.

Step 4: Monitor the Test Trigger API

Tracetest exposes a REST API that CI/CD pipelines and the tracetest CLI use to initiate test runs programmatically. This is the endpoint that your GitHub Actions workflows, Jenkins jobs, or Tekton tasks call when running trace-based tests in a pipeline:

# Verify the tests API endpoint responds
curl -I https://tracetest.example.com/api/tests
# Returns HTTP 200 (list of tests) or 401 (unauthorized) — both confirm the endpoint exists
  1. Add Monitor → HTTP.
  2. URL: https://tracetest.example.com/api/tests.
  3. Check interval: 5 minutes.
  4. Response timeout: 15 seconds.
  5. Expected status: 200 (if using an API token in the header; note Vigilmon supports custom headers for authenticated endpoints).
  6. Label: Tracetest test trigger API.
  7. Click Save.

For Tracetest instances requiring API key authentication, configure the monitor with an Authorization: Bearer <token> header. Vigilmon supports custom HTTP headers in monitor configuration. Without authentication, a 401 response confirms the endpoint is reachable but requires credentials — configure the monitor to accept 401 as a valid response code if you are not using header-based authentication in monitoring.

Step 5: Monitor the OTLP Trace Ingestion Port

Tracetest receives traces from instrumented services via the OTLP (OpenTelemetry Protocol) endpoint. Tracetest can either receive traces directly (acting as a collector) or poll them from an external trace store. When configured for direct ingestion, Tracetest listens on standard OTLP ports:

  • OTLP gRPC: port 4317
  • OTLP HTTP: port 4318

TCP monitoring confirms these ports accept connections — critical for catching firewall changes or pod restarts that break trace collection:

# Test OTLP gRPC port
nc -zv tracetest.example.com 4317

# Test OTLP HTTP port
nc -zv tracetest.example.com 4318

For OTLP HTTP, you can also do an HTTP check:

curl -I http://tracetest.example.com:4318/v1/traces
# Returns 405 (Method Not Allowed — endpoint exists but expects POST with trace payload)
  1. Add Monitor → TCP.
  2. Host: tracetest.example.com.
  3. Port: 4317 (gRPC OTLP).
  4. Check interval: 2 minutes.
  5. Label: Tracetest OTLP gRPC port.
  6. Click Save.

Repeat for port 4318 if you also use OTLP/HTTP.

If Tracetest is configured to poll traces from an external store (Jaeger, Grafana Tempo, OpenSearch) rather than receiving them directly, skip the OTLP port monitors — in that architecture, Tracetest's availability depends on the external trace store being reachable, not on direct OTLP ingestion.

Step 6: Monitor SSL Certificates

Tracetest's HTTPS endpoint is used by the CLI, web UI, and CI/CD integrations. An expired certificate breaks all of these simultaneously, and because trace-based tests typically run in automated pipelines, certificate expiry can silently cause test suite failures that look like API errors in CI logs:

openssl s_client -connect tracetest.example.com:443 2>/dev/null | openssl x509 -noout -dates
  1. Add Monitor → SSL Certificate.
  2. Domain: tracetest.example.com.
  3. Alert when expiry is within: 30 days.
  4. Alert again: 14 days, 7 days, 3 days, 1 day.
  5. Click Save.

Step 7: Configure Alerting

In Vigilmon under Settings → Notifications, configure your alert channels:

| Monitor | Trigger | Action | |---|---|---| | /api/health | Non-200 or pass missing | Check Tracetest server container; verify PostgreSQL and trace store connectivity | | Web UI | Non-200 or keyword missing | Frontend serving issue; check Nginx or ingress routing for / path | | Test trigger API | Non-200/401 response | CI/CD test runs will fail; check Tracetest API and authentication config | | OTLP gRPC port | Connection refused | Traces cannot be ingested; instrumented services accumulate backpressure | | SSL certificate | < 30 days to expiry | Renew certificate; CLI and CI/CD pipelines will start rejecting TLS connections |

Alert after: 1 consecutive failure for the API health and OTLP port monitors. 2 consecutive failures for UI and test trigger monitors.

Integrating Vigilmon Heartbeat Pings into Tracetest Test Runs

You can use Tracetest's test lifecycle to send heartbeat pings to Vigilmon, confirming that your trace-based test suite is successfully completing on schedule. This is a "dead man's switch" pattern — if your CI/CD pipeline stops running tests (due to a broken pipeline, a misconfigured schedule, or a failed build), the heartbeat stops and Vigilmon alerts you:

# In your CI/CD pipeline, after a successful Tracetest test run:
TRACETEST_RESULT=$(tracetest run test -f ./test-definition.yaml --wait-for-result)
if echo "$TRACETEST_RESULT" | grep -q "PASS"; then
  curl -m 10 --retry 3 https://vigilmon.online/api/v1/heartbeat/YOUR_HEARTBEAT_ID
fi

Or with the Tracetest CLI's exit code:

tracetest run test -f ./test-definition.yaml --wait-for-result
if [ $? -eq 0 ]; then
  curl -m 10 --retry 3 https://vigilmon.online/api/v1/heartbeat/YOUR_HEARTBEAT_ID
fi

Create the heartbeat monitor in Vigilmon:

  1. Add Monitor → Heartbeat.
  2. Expected interval: Match your CI/CD schedule (e.g., 10 minutes for a test suite that runs every 10 minutes, or 1 hour for an hourly schedule).
  3. Grace period: 5 minutes.
  4. Label: Tracetest suite heartbeat.
  5. Click Save, then copy the unique heartbeat URL.

Common Tracetest Failure Modes and What Vigilmon Catches

| Scenario | Vigilmon monitor | |---|---| | Tracetest API server pod crash | /api/health returns connection error; alert within 60 s | | PostgreSQL database unavailable | Health check returns fail; all test results are lost in-flight | | Web UI static assets not served | UI keyword monitor fires; API health stays green | | OTLP port blocked by firewall | OTLP TCP monitor fires; all trace ingestion silently stops | | CI/CD pipeline stops running tests | Heartbeat monitor fires after grace period expires | | SSL certificate expired | SSL monitor alerts; CLI and CI/CD integrations reject TLS | | Tracetest upgrade breaks API | /api/health returns errors after version update | | External trace store (Jaeger/Tempo) unreachable | Tests run but assertions fail; Tracetest API stays healthy | | Ingress controller failure | All HTTP monitors fire simultaneously | | Test trigger API routing broken | Test trigger monitor fires; CLI runs succeed but REST API calls fail |

Tracetest is the bridge between your distributed tracing infrastructure and your testing pipeline — when it fails, your trace-based tests stop running silently and you lose the observability signal that confirms distributed transactions are behaving correctly. Vigilmon gives you external visibility into Tracetest's entire availability surface: the API health endpoint, web UI, test trigger API, OTLP trace ingestion port, and SSL certificate expiry, plus a heartbeat pattern to confirm your test suite is actually running on schedule.

Start monitoring Tracetest in under 5 minutes — register free at vigilmon.online.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles