Monitoring Helicone with Vigilmon

Helicone is an open-source LLM observability platform and proxy (Y Combinator W23). It sits between your application and LLM providers like OpenAI, Anthropic, and Azure OpenAI — logging every request and response, computing costs, enforcing rate limits, and providing a dashboard for your usage analytics.

Because Helicone is a proxy, it's on the critical path of every LLM call your application makes. If Helicone goes down, your AI features go with it. If the logging pipeline degrades, you lose cost visibility and request history without knowing it.

This guide shows you how to monitor Helicone with Vigilmon to catch proxy outages, pipeline degradation, and service health issues before they affect your users.

Helicone Architecture Overview

Understanding what can fail helps you monitor the right things:

| Layer | Role | Failure impact | |-------|------|----------------| | Proxy endpoint | Routes LLM requests to providers | All LLM calls fail | | Worker/edge nodes | Process and forward requests | Increased latency or dropped requests | | Logging pipeline (Kafka/queue) | Buffers request logs asynchronously | Logs are lost, cost analytics break | | Database (Supabase/Postgres) | Stores requests, users, costs | Dashboard queries fail | | Cache layer | Serves cached LLM responses | Cache misses only — LLM calls still succeed | | Rate limiting service | Enforces per-user/per-key rate limits | Rate limits may not apply, leading to cost overruns |

Deploying Helicone (Self-Hosted)

For self-hosted deployments via Docker Compose:

# docker-compose.yml (abbreviated)
version: "3.8"
services:
  helicone-worker:
    image: helicone/worker:latest
    ports:
      - "8787:8787"
    environment:
      SUPABASE_URL: http://supabase:5432
      SUPABASE_SERVICE_ROLE_KEY: your-service-role-key
      OPENAI_API_BASE: https://api.openai.com

  helicone-web:
    image: helicone/web:latest
    ports:
      - "3000:3000"
    environment:
      NEXT_PUBLIC_SUPABASE_URL: http://supabase:5432

Point your LLM SDK at the Helicone proxy instead of the provider directly:

# Python — route OpenAI calls through Helicone
from openai import OpenAI

client = OpenAI(
    api_key="sk-your-openai-key",
    base_url="https://oai.helicone.ai/v1",
    default_headers={
        "Helicone-Auth": "Bearer sk-helicone-your-key",
    },
)

response = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[{"role": "user", "content": "Hello"}],
)

// TypeScript — route through Helicone
import OpenAI from "openai";

const openai = new OpenAI({
  apiKey: process.env.OPENAI_API_KEY,
  baseURL: "https://oai.helicone.ai/v1",
  defaultHeaders: {
    "Helicone-Auth": `Bearer ${process.env.HELICONE_API_KEY}`,
  },
});

Health Endpoints

Helicone Proxy Health

Helicone's worker exposes a health route:

GET https://oai.helicone.ai/health

For self-hosted instances:

GET http://your-helicone-host:8787/health

Expected response:

{"status": "healthy", "version": "2.x.x"}

Helicone Web Dashboard

GET https://helicone.ai

Or for self-hosted:

GET http://your-helicone-host:3000

Rate Limiting Service Health

If Helicone is configured with rate limiting, you can verify the rate limit enforcement endpoint is reachable:

GET http://your-helicone-host:8787/v1/rate-limits/check

Vigilmon Monitor Setup

Monitor 1: Helicone Proxy Uptime

1. Log in to Vigilmon → Monitors → New Monitor
2. Type: HTTP
3. Method: GET
4. URL: https://oai.helicone.ai/health (or your self-hosted URL)
5. Interval: 1 minute
6. Keyword check: healthy
7. Response timeout: 10 seconds
8. Save

This is your most critical monitor — a failure here means every LLM call in your application is broken.

Monitor 2: Proxy Latency Check

1. Type: HTTP
2. Method: GET
3. URL: https://oai.helicone.ai/health
4. Interval: 2 minutes
5. Response time threshold (warning): 2000ms
6. Response time threshold (critical): 5000ms
7. Save

A slow health endpoint often signals worker overload — which will affect all proxied LLM call latencies.

Monitor 3: Web Dashboard Availability

1. Type: HTTP
2. Method: GET
3. URL: https://helicone.ai (or your self-hosted UI URL)
4. Interval: 5 minutes
5. Keyword check: Helicone
6. Response timeout: 10 seconds
7. Save

Verifying the Logging Pipeline

The logging pipeline is the piece most likely to fail silently. Requests can succeed through the proxy while the async logging worker drops spans.

# monitoring/helicone_pipeline_check.py
import time
import httpx
from openai import OpenAI

HELICONE_API_KEY = "sk-helicone-your-key"
OPENAI_API_KEY = "sk-your-openai-key"
HELICONE_BASE_URL = "https://oai.helicone.ai/v1"

def send_probe_request() -> dict:
    """Send a minimal request through Helicone and capture the request ID."""
    client = OpenAI(
        api_key=OPENAI_API_KEY,
        base_url=HELICONE_BASE_URL,
        default_headers={
            "Helicone-Auth": f"Bearer {HELICONE_API_KEY}",
            "Helicone-Property-Purpose": "monitoring-probe",
        },
    )

    start = time.time()
    try:
        response = client.chat.completions.create(
            model="gpt-4o-mini",
            max_tokens=5,
            messages=[{"role": "user", "content": "Reply: ok"}],
        )
        latency_ms = int((time.time() - start) * 1000)
        # Helicone injects a request ID in the response headers (via SDK)
        return {"ok": True, "latency_ms": latency_ms}
    except Exception as exc:
        return {"ok": False, "error": str(exc)}


def verify_request_logged(purpose_filter: str, within_minutes: int = 5) -> bool:
    """Query Helicone API to verify that the probe request was logged."""
    try:
        resp = httpx.post(
            "https://api.helicone.ai/v1/request/query",
            headers={"authorization": f"Bearer {HELICONE_API_KEY}"},
            json={
                "filter": {
                    "property": {
                        "purpose": {"equals": purpose_filter},
                    },
                    "request": {
                        "createdAt": {
                            "gte": int((time.time() - within_minutes * 60) * 1000),
                        },
                    },
                },
                "limit": 1,
            },
            timeout=10,
        )
        data = resp.json()
        return len(data.get("data", [])) > 0
    except Exception:
        return False

Expose this as an HTTP health endpoint:

# FastAPI health route
from fastapi import APIRouter
from fastapi.responses import JSONResponse
from monitoring.helicone_pipeline_check import send_probe_request, verify_request_logged

router = APIRouter()

@router.get("/health/helicone-pipeline")
async def helicone_pipeline_health():
    result = send_probe_request()

    if not result["ok"]:
        return JSONResponse(
            {"status": "error", "detail": result.get("error")},
            status_code=503,
        )

    logged = verify_request_logged("monitoring-probe", within_minutes=5)
    if not logged:
        return JSONResponse(
            {"status": "degraded", "detail": "request not appearing in Helicone logs"},
            status_code=503,
        )

    return {"status": "ok", "proxy_latency_ms": result["latency_ms"]}

Vigilmon setup:

1. URL: https://your-app.com/health/helicone-pipeline
2. Interval: 10 minutes
3. Keyword check: "status":"ok"
4. Timeout: 30 seconds

Monitoring Cache Hit Rate

Helicone offers LLM response caching. A drop in cache hit rate can signal that your cache configuration has broken or been accidentally disabled — leading to unnecessary LLM API costs.

# monitoring/helicone_cache_check.py
import httpx

def get_cache_hit_rate(api_key: str, hours: int = 1) -> float | None:
    """Returns cache hit rate (0.0–1.0) over the last N hours."""
    import time
    gte = int((time.time() - hours * 3600) * 1000)

    try:
        resp = httpx.post(
            "https://api.helicone.ai/v1/request/query",
            headers={"authorization": f"Bearer {api_key}"},
            json={
                "filter": {
                    "request": {"createdAt": {"gte": gte}},
                },
            },
            timeout=15,
        )
        data = resp.json()
        requests = data.get("data", [])
        if not requests:
            return None

        cached = sum(1 for r in requests if r.get("cacheHit") is True)
        return cached / len(requests)
    except Exception:
        return None

If cache hit rate drops from an expected ~40% to near 0%, it may indicate that the Helicone-Cache-Enabled: true header is no longer being sent, or the cache TTL was misconfigured.

Cost Analytics Accuracy Check

Verify that Helicone's cost computation is tracking correctly by comparing its reported cost against your own calculation:

# monitoring/cost_verification.py
import httpx

EXPECTED_COST_PER_1K = {
    "gpt-4o-mini": {"input": 0.00015, "output": 0.0006},
    "gpt-4o": {"input": 0.0025, "output": 0.01},
}

def verify_cost_accuracy(api_key: str, tolerance: float = 0.25) -> dict:
    """
    Fetch the most recent logged request and compare Helicone's
    reported cost against our expected cost.
    """
    resp = httpx.post(
        "https://api.helicone.ai/v1/request/query",
        headers={"authorization": f"Bearer {api_key}"},
        json={"limit": 1, "sort": {"createdAt": "desc"}},
        timeout=10,
    )
    data = resp.json()
    requests = data.get("data", [])
    if not requests:
        return {"ok": True, "reason": "no requests to check"}

    req = requests[0]
    model = req.get("model", "")
    reported_cost = req.get("totalCost", 0)
    input_tokens = req.get("promptTokens", 0)
    output_tokens = req.get("completionTokens", 0)

    pricing = EXPECTED_COST_PER_1K.get(model)
    if not pricing:
        return {"ok": True, "reason": f"no pricing baseline for {model}"}

    expected_cost = (
        pricing["input"] * input_tokens / 1000
        + pricing["output"] * output_tokens / 1000
    )

    if expected_cost == 0:
        return {"ok": True}

    variance = abs(reported_cost - expected_cost) / expected_cost
    return {
        "ok": variance < tolerance,
        "variance": round(variance, 4),
        "reported": reported_cost,
        "expected": expected_cost,
    }

Rate Limiting Service Health

If rate limiting is enabled, ensure it's functioning by testing a known rate-limited key:

# Test that rate limits are being enforced
import httpx

def check_rate_limit_enforcement(helicone_api_key: str) -> bool:
    """
    Verifies that Helicone's rate limit API is responsive and returning data.
    """
    try:
        resp = httpx.get(
            "https://api.helicone.ai/v1/user-limits",
            headers={"authorization": f"Bearer {helicone_api_key}"},
            timeout=10,
        )
        return resp.status_code in (200, 404)  # 404 = no limits set, which is fine
    except Exception:
        return False

Alerting Strategy

| Monitor | Interval | Alert when | |---------|----------|-----------| | Proxy health endpoint | 1 min | Any failure — immediate escalation | | Proxy latency | 2 min | >2000ms warning, >5000ms critical | | Web dashboard | 5 min | Keyword mismatch | | Pipeline heartbeat | 10 min | Not logging — degraded mode | | Cache hit rate | Hourly | Drop >50% from baseline |

The proxy health monitor should page your on-call immediately — Helicone proxy downtime means broken LLM features for all users. The pipeline and cache monitors are high priority but not wake-the-team urgency; route them to Slack.

Summary

Helicone is on the critical path of your LLM stack. Monitoring it at multiple layers gives you:

• Proxy uptime — the most critical signal
• Logging pipeline health — catches silent data loss
• Cache hit rate — guards against cost overruns from misconfiguration
• Cost accuracy — ensures analytics remain trustworthy

With Vigilmon watching each layer, you'll know about Helicone issues before they surface as user complaints or unexpected LLM bills.

Further Reading

• Monitoring OpenLIT with Vigilmon
• Monitoring Phoenix (Arize) with Vigilmon
• Monitoring AI/LLM API Dependencies with Vigilmon