tutorial

How to Monitor Braintrust with Vigilmon

Braintrust runs your LLM evals and trace logging — but it needs monitoring too. Learn how to track Braintrust API availability, alert on eval pipeline failures, and heartbeat your CI/CD LLM quality checks.

How to Monitor Braintrust with Vigilmon

Braintrust is where teams run LLM evaluations, log model traces, and track quality metrics over time. If your Braintrust logging pipeline goes down or an eval run silently fails, you lose the visibility that tells you when a model regression happened — exactly when you need it most.

Monitoring Braintrust means two things: confirming the Braintrust platform itself is reachable and functional, and ensuring your own eval scripts and logging integrations run on schedule. This guide covers both.

What can go wrong with a Braintrust integration

Braintrust API unavailability. Your application instruments LLM calls with the Braintrust SDK. If the Braintrust API is unreachable, trace logging either fails silently or blocks your application depending on how you've configured error handling.

Eval pipelines stop running. Automated eval jobs (via SDK, CI, or cron) can fail without alerting you — a bad deploy, a changed dependency, or an expired API key stops your quality pipeline while you ship changes you think are safe.

SDK logging silently drops events. The Braintrust SDK uses background threads or async queues for trace logging. These can silently fail under high load or network degradation, leaving gaps in your eval dataset without any indication something went wrong.

CI LLM quality gates stop blocking. If your pytest-integrated evals stop running, broken prompts merge to main. The absence of a failing test is not the same as a passing test.

Step 1: Check Braintrust API reachability from your application

Add a health check to your application that verifies the Braintrust API is reachable:

# app/health/braintrust_check.py
import os
import httpx

BRAINTRUST_API_URL = "https://api.braintrustdata.com"


async def check_braintrust_api() -> dict:
    """Verify Braintrust API is reachable and the API key is valid."""
    api_key = os.environ.get("BRAINTRUST_API_KEY", "")
    if not api_key:
        return {"status": "error", "detail": "BRAINTRUST_API_KEY not set"}

    try:
        async with httpx.AsyncClient(timeout=5) as client:
            response = await client.get(
                f"{BRAINTRUST_API_URL}/v1/project",
                headers={"Authorization": f"Bearer {api_key}"},
            )
        if response.status_code == 200:
            return {"status": "ok"}
        elif response.status_code == 401:
            return {"status": "error", "detail": "Invalid API key"}
        else:
            return {"status": "error", "detail": f"HTTP {response.status_code}"}
    except httpx.TimeoutException:
        return {"status": "error", "detail": "Braintrust API timeout"}
    except Exception as e:
        return {"status": "error", "detail": str(e)}

Wire it into your application's /health endpoint:

# app/health/routes.py
from fastapi import APIRouter, status
from fastapi.responses import JSONResponse
from app.health.braintrust_check import check_braintrust_api

router = APIRouter()


@router.get("/health")
async def health_check():
    braintrust = await check_braintrust_api()
    checks = {"braintrust": braintrust}

    all_ok = all(c["status"] == "ok" for c in checks.values())
    return JSONResponse(
        status_code=status.HTTP_200_OK if all_ok else status.HTTP_503_SERVICE_UNAVAILABLE,
        content={
            "status": "ok" if all_ok else "degraded",
            "checks": checks,
        },
    )

Test it:

curl http://localhost:8000/health
# {"status":"ok","checks":{"braintrust":{"status":"ok"}}}

Step 2: Monitor the Braintrust API with Vigilmon

Set up Vigilmon to watch the Braintrust API endpoint directly:

  1. Sign up at vigilmon.online — free tier, no credit card
  2. Click New Monitor → HTTP
  3. Enter https://api.braintrustdata.com
  4. Set the check interval (5 minutes on free tier)
  5. Save

Also monitor your own application's Braintrust-dependent health endpoint:

| Endpoint | What it catches | |---|---| | https://api.braintrustdata.com | Braintrust platform availability | | https://yourapp.com/health | Your app's Braintrust connectivity |

Vigilmon probes from multiple regions. If the Braintrust API goes down or your API key expires, you'll know before your eval data starts going missing.

Step 3: Heartbeat monitoring for eval pipelines

Eval pipelines run on a schedule — nightly, on every deploy, or after model updates. They fail silently more often than you'd expect: expired API keys, dependency updates that break imports, CI configuration drift. Heartbeat monitoring catches all of these.

The pattern: your eval script pings a unique Vigilmon URL on successful completion. If pings stop arriving within the expected interval, Vigilmon alerts you.

# evals/run_evals.py
import os
import httpx
import braintrust
from braintrust import Eval

# Define your eval
def exact_match(output: str, expected: str) -> float:
    return 1.0 if output.strip() == expected.strip() else 0.0


async def run_nightly_evals():
    """Run LLM quality evals and ping heartbeat on success."""
    try:
        result = await Eval(
            "Production Quality Check",
            data=lambda: [
                {"input": "Summarize this text: ...", "expected": "..."},
                {"input": "Translate to French: Hello", "expected": "Bonjour"},
            ],
            task=lambda input: your_llm_function(input),
            scores=[exact_match],
        )

        print(f"Eval complete: {result.summary}")

        # Only ping on success
        heartbeat_url = os.environ.get("EVAL_HEARTBEAT_URL")
        if heartbeat_url:
            async with httpx.AsyncClient() as client:
                await client.get(heartbeat_url, timeout=10)
            print("Heartbeat pinged")

    except Exception as e:
        print(f"Eval failed: {e}")
        # Do NOT ping — absence of ping IS the alert
        raise


def your_llm_function(input: str) -> str:
    # Replace with your actual LLM call instrumented with Braintrust
    import anthropic
    client = anthropic.Anthropic()
    response = client.messages.create(
        model="claude-haiku-4-5-20251001",
        max_tokens=256,
        messages=[{"role": "user", "content": input}],
    )
    return response.content[0].text

In Vigilmon:

  1. Click New Monitor → Heartbeat
  2. Set the expected interval (e.g. 25 hours for a nightly job)
  3. Copy the unique ping URL
  4. Set the environment variable:
EVAL_HEARTBEAT_URL=https://vigilmon.online/api/heartbeat/your-unique-token

Add heartbeats for each critical eval pipeline:

EVAL_HEARTBEAT_URL=https://vigilmon.online/api/heartbeat/token-1
REGRESSION_HEARTBEAT_URL=https://vigilmon.online/api/heartbeat/token-2
SAFETY_EVAL_HEARTBEAT_URL=https://vigilmon.online/api/heartbeat/token-3

Step 4: Monitor CI/CD eval gates

If you've integrated Braintrust evals into your CI pipeline using pytest or a custom runner, add a Vigilmon heartbeat to the CI job itself:

# .github/workflows/llm-quality.yml
name: LLM Quality Gate

on:
  push:
    branches: [main]

jobs:
  eval:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.11"

      - name: Install dependencies
        run: pip install braintrust pytest

      - name: Run LLM evals
        env:
          BRAINTRUST_API_KEY: ${{ secrets.BRAINTRUST_API_KEY }}
        run: pytest evals/ -v

      - name: Ping Vigilmon heartbeat on success
        if: success()
        run: curl -s "${{ secrets.CI_EVAL_HEARTBEAT_URL }}"

This heartbeat tells Vigilmon: "the CI eval job ran and passed." If a deploy goes through without triggering evals (job skipped, runner failure, config change), Vigilmon alerts you.

Step 5: Log SDK errors to detect silent drops

Add a Braintrust SDK error handler that surfaces silent failures:

# app/instrumentation.py
import logging
import braintrust
import os

logger = logging.getLogger(__name__)

_error_count = 0


def init_braintrust():
    """Initialize Braintrust SDK with error visibility."""
    api_key = os.environ.get("BRAINTRUST_API_KEY")
    if not api_key:
        logger.warning("BRAINTRUST_API_KEY not set — tracing disabled")
        return

    # The SDK uses background flush; wrap with error tracking
    original_flush = braintrust.flush

    def tracked_flush(*args, **kwargs):
        global _error_count
        try:
            return original_flush(*args, **kwargs)
        except Exception as e:
            _error_count += 1
            logger.error("Braintrust flush failed (total failures: %d): %s", _error_count, e)
            raise

    braintrust.flush = tracked_flush


def get_sdk_error_count() -> int:
    return _error_count

Expose the error count on your health endpoint so Vigilmon can detect degraded tracing:

from app.instrumentation import get_sdk_error_count

@router.get("/health")
async def health_check():
    sdk_errors = get_sdk_error_count()
    checks = {
        "braintrust": await check_braintrust_api(),
        "sdk_errors": {
            "status": "ok" if sdk_errors == 0 else "degraded",
            "count": sdk_errors,
        },
    }
    # ...

Step 6: Alerts and notifications

Configure alert routing in Vigilmon:

For Slack:

  1. Create an incoming webhook in Slack (Settings → Integrations → Webhooks)
  2. In Vigilmon go to Notifications → New Channel → Slack
  3. Paste the URL and enable it on your monitors

You'll receive:

🔴 DOWN: yourapp.com/health
Status: 503 (Braintrust API unreachable)
Started: 4 minutes ago

🔴 MISSED: nightly-eval-pipeline heartbeat
Expected every: 25 hours
Last ping: 31 hours ago

For eval pipelines, a missed heartbeat alert means: "your LLM quality data has a gap." That's the signal to investigate before shipping more model changes.

What you've built

| What | How | |---|---| | Braintrust API connectivity check | /health endpoint probing the Braintrust REST API | | Platform uptime monitoring | Vigilmon HTTP monitor on api.braintrustdata.com | | Eval pipeline monitoring | Heartbeat ping on successful completion | | CI quality gate monitoring | Heartbeat in GitHub Actions on eval pass | | SDK error visibility | Flush error counter exposed on health endpoint | | Instant alerts | Slack webhook notifications |

LLM quality measurement only works if the measurement pipeline runs. This setup ensures gaps in your eval data are caught immediately, not days later when you're trying to explain a regression.

Get started free at vigilmon.online — your first monitor is running in under a minute.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles