Monitoring OpenLIT with Vigilmon

OpenLIT is an OpenTelemetry-native observability platform purpose-built for LLM applications. It instruments your Python or TypeScript code to emit spans for every LLM call — capturing token counts, costs, prompt/completion content, model latency, and GPU utilization — then stores and visualizes those traces in its own UI backed by ClickHouse.

When OpenLIT itself is unhealthy, your entire LLM observability pipeline goes dark. You lose cost visibility, latency alerts, and the ability to debug unexpected model behavior. This guide shows you how to monitor OpenLIT with Vigilmon so you catch pipeline failures before they become blind spots.

What Can Go Wrong with OpenLIT

OpenLIT has several components that can fail independently:

| Component | Failure mode | Impact | |-----------|-------------|--------| | OpenLIT API server | Down or unreachable | Traces can't be ingested or queried | | ClickHouse backend | Full disk or OOM | Trace storage fails silently | | OTLP collector endpoint | Port unreachable | SDK instrumentation drops spans | | GPU utilization collector | Missing DCGM exporter | GPU metrics disappear from dashboard | | Cost computation pipeline | Stale pricing config | Cost analytics show wrong values |

Most of these failures are silent — your application keeps running, your LLM calls complete, but the observability data never arrives.

Setting Up OpenLIT

If you haven't deployed OpenLIT yet, the quickest path is Docker Compose:

# docker-compose.yml
version: "3.8"
services:
  clickhouse:
    image: clickhouse/clickhouse-server:24.3
    ports:
      - "8123:8123"
      - "9000:9000"
    volumes:
      - clickhouse_data:/var/lib/clickhouse
    environment:
      CLICKHOUSE_DB: openlit
      CLICKHOUSE_USER: default
      CLICKHOUSE_PASSWORD: openlit

  openlit:
    image: ghcr.io/openlit/openlit:latest
    ports:
      - "3000:3000"
    environment:
      INIT_DB_HOST: clickhouse
      INIT_DB_PORT: 9000
      INIT_DB_DATABASE: openlit
      INIT_DB_USERNAME: default
      INIT_DB_PASSWORD: openlit
      TELEMETRY_ENABLED: "false"
    depends_on:
      - clickhouse

volumes:
  clickhouse_data:

docker compose up -d
# OpenLIT UI → http://localhost:3000
# OTLP endpoint → http://localhost:3000/api/otlp

To instrument your Python application:

# Install the SDK
# pip install openlit

import openlit
import openai

openlit.init(
    otlp_endpoint="http://localhost:3000/api/otlp",
    application_name="my-llm-app",
    environment="production",
)

client = openai.OpenAI()

# OpenLIT auto-instruments this call — traces token usage, cost, latency
response = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[{"role": "user", "content": "Summarize this document."}],
)

OpenLIT wraps the OpenAI, Anthropic, Cohere, and other LLM SDKs automatically via monkey-patching — no manual span creation required.

Health Endpoints to Monitor

OpenLIT Application Health

OpenLIT exposes a health endpoint at /api/health:

GET http://your-openlit-host:3000/api/health

Expected response:

{"status": "ok"}

This endpoint checks that the application server is running and can reach the database.

ClickHouse HTTP Interface

ClickHouse exposes its own HTTP health check:

GET http://your-clickhouse-host:8123/ping

Expected response:

Ok.

If ClickHouse is down, OpenLIT will fail to write or query any trace data — even if the OpenLIT application server responds as healthy.

OTLP Ingestion Endpoint

The OTLP/HTTP endpoint used by the SDK to submit spans:

POST http://your-openlit-host:3000/api/otlp/v1/traces

You can probe this with a GET to confirm the server is accepting connections (it will return 405 Method Not Allowed — but the key point is that it's reachable, not timing out).

Vigilmon Monitor Setup

Monitor 1: OpenLIT Application Health

1. Log in to Vigilmon → Monitors → New Monitor
2. Type: HTTP
3. Method: GET
4. URL: http://your-openlit-host:3000/api/health
5. Interval: 2 minutes
6. Keyword check: "status":"ok"
7. Response timeout: 10 seconds
8. Save

Monitor 2: ClickHouse Storage Layer

1. Type: HTTP
2. Method: GET
3. URL: http://your-clickhouse-host:8123/ping
4. Interval: 2 minutes
5. Keyword check: Ok.
6. Response timeout: 5 seconds
7. Save

Monitor 3: OTLP Ingestion Reachability

1. Type: HTTP
2. Method: GET
3. URL: http://your-openlit-host:3000/api/otlp/v1/traces
4. Interval: 5 minutes
5. Expected status code: Any (405 is acceptable — proves the path exists)
6. Response timeout: 10 seconds
7. Save

Tip: If your OpenLIT instance is behind a reverse proxy or firewall, test from outside your internal network to ensure you're probing the same path your SDK instruments hit.

Heartbeat Monitoring for the Trace Pipeline

The most important thing to monitor is whether traces are actually flowing — not just whether the server is up. A synthetic heartbeat check verifies the full pipeline end to end.

# monitoring/openlit_heartbeat.py
import time
import httpx
import openlit
import openai

def send_probe_trace() -> dict:
    """
    Makes a minimal LLM call that OpenLIT should instrument and forward to ClickHouse.
    Returns latency and success status.
    """
    openlit.init(
        otlp_endpoint="http://your-openlit-host:3000/api/otlp",
        application_name="vigilmon-probe",
        environment="production",
    )

    client = openai.OpenAI()
    start = time.time()

    try:
        response = client.chat.completions.create(
            model="gpt-4o-mini",
            max_tokens=5,
            messages=[{"role": "user", "content": "Reply: ok"}],
        )
        latency_ms = int((time.time() - start) * 1000)
        return {"ok": True, "latency_ms": latency_ms}
    except Exception as exc:
        return {"ok": False, "error": str(exc), "latency_ms": int((time.time() - start) * 1000)}


def check_trace_arrived(trace_app: str, within_seconds: int = 60) -> bool:
    """
    Queries the OpenLIT API to verify that a trace from the probe app arrived recently.
    """
    try:
        resp = httpx.get(
            "http://your-openlit-host:3000/api/metrics/llm/usage",
            params={"timeLimit": within_seconds, "applications": trace_app},
            timeout=10,
        )
        data = resp.json()
        return data.get("totalRequests", 0) > 0
    except Exception:
        return False

Expose this as an HTTP health endpoint and configure Vigilmon to call it every 10 minutes:

# app/routes/health.py (FastAPI example)
from fastapi import APIRouter
from monitoring.openlit_heartbeat import send_probe_trace, check_trace_arrived

router = APIRouter()

@router.get("/health/openlit-pipeline")
async def openlit_pipeline_health():
    result = send_probe_trace()
    if not result["ok"]:
        return {"status": "error", "detail": result["error"]}, 503

    arrived = check_trace_arrived("vigilmon-probe", within_seconds=120)
    if not arrived:
        return {"status": "degraded", "detail": "trace not arriving in ClickHouse"}, 503

    return {"status": "ok", "latency_ms": result["latency_ms"]}

Vigilmon setup for this heartbeat endpoint:

1. URL: https://your-app.com/health/openlit-pipeline
2. Interval: 10 minutes
3. Keyword check: "status":"ok"
4. Timeout: 30 seconds (the LLM call takes a few seconds)

Monitoring Token Usage and Cost Tracking Accuracy

If the cost tracking pipeline breaks (stale pricing config, schema migration), your cost dashboards silently show wrong values. Add a sanity check:

# monitoring/cost_sanity.py
import httpx

EXPECTED_COST_PER_1K_INPUT_TOKENS = {
    "gpt-4o-mini": 0.00015,  # $0.15 per 1M input tokens
    "gpt-4o": 0.0025,
}

def check_cost_accuracy(model: str, reported_cost: float, token_count: int) -> bool:
    expected = EXPECTED_COST_PER_1K_INPUT_TOKENS.get(model)
    if not expected:
        return True  # Unknown model, skip check

    expected_cost = expected * (token_count / 1000)
    # Allow 20% variance for rounding differences
    return abs(reported_cost - expected_cost) / max(expected_cost, 0.000001) < 0.2

Alert if cost reports diverge significantly from expected values — it's a signal that the pricing config has drifted.

GPU Utilization Monitoring

If your deployment uses GPU inference, OpenLIT collects GPU metrics via DCGM. Verify the collector is running:

# Check DCGM exporter is up
curl -s http://your-gpu-host:9400/metrics | grep dcgm_gpu_utilization

Add a Vigilmon HTTP monitor:

• URL: http://your-gpu-host:9400/metrics
• Keyword check: dcgm_gpu_utilization
• Interval: 1 minute

If the keyword is absent, GPU metrics have stopped flowing — you'll see a flat line in OpenLIT's GPU dashboard without realizing it.

Alerting Configuration

| Monitor | Recommended interval | Alert on | |---------|---------------------|----------| | OpenLIT /api/health | 2 min | Failure or keyword mismatch | | ClickHouse /ping | 2 min | Failure | | OTLP endpoint reachability | 5 min | Timeout or connection refused | | Pipeline heartbeat | 10 min | Non-200 or keyword mismatch | | GPU metrics endpoint | 1 min | Keyword dcgm_gpu_utilization absent |

Set 2-failure confirmation on all monitors to avoid false positives from transient network blips. For the pipeline heartbeat, a single failure is worth alerting on immediately — trace gaps are hard to backfill.

Latency Percentile Baselines

OpenLIT's value comes partly from tracking model latency percentiles (p50, p95, p99) across your LLM calls. If the collection pipeline is degraded, these percentiles become misleading.

Set Vigilmon latency thresholds on your health endpoint:

1. Monitor → Advanced Settings → Response time threshold
2. Warning: 15,000ms (the health check calls OpenAI, so allow time)
3. Critical: 30,000ms

If the health endpoint consistently exceeds these thresholds, the upstream LLM provider may be experiencing latency issues that OpenLIT should be surfacing to you — check the OpenLIT dashboard for p95 spikes.

Summary

| What to monitor | Endpoint | Check type | |----------------|----------|-----------| | OpenLIT server health | :3000/api/health | HTTP keyword ok | | ClickHouse storage | :8123/ping | HTTP keyword Ok. | | OTLP ingestion path | :3000/api/otlp/v1/traces | HTTP reachability | | End-to-end pipeline | Your /health/openlit-pipeline | HTTP keyword ok | | GPU metrics collector | :9400/metrics | Keyword dcgm_gpu_utilization |

With these monitors in Vigilmon, you'll catch OpenLIT infrastructure failures before they create silent gaps in your LLM observability data. Monitoring the observer is just as important as the observer itself.

Further Reading

• Monitoring AI/LLM API Dependencies with Vigilmon
• Monitoring Helicone with Vigilmon
• Monitoring Phoenix (Arize) with Vigilmon