Monitoring Traceloop with Vigilmon

Traceloop (now merged with OpenLLMetry) is an OpenTelemetry-based LLM tracing library and platform. It instruments your Python, TypeScript, Go, Ruby, and Java applications to emit standardized spans for every LLM call, chain step, and agent workflow — then ships those spans to any OTLP-compatible backend (Grafana, Datadog, New Relic, Jaeger, or Traceloop's own cloud).

Because Traceloop is wired into your application at the SDK level, its health directly affects the completeness of your LLM observability data. If the trace collector is dropping spans, prompt content isn't being stored, or workflow tracing has lost context, you're accumulating gaps you may not notice until you try to debug a production incident.

This guide shows you how to monitor Traceloop with Vigilmon to detect collector failures, span completeness issues, and pipeline degradation in real time.

Traceloop Architecture

Traceloop consists of:

| Component | Role | Impact when degraded | |-----------|------|---------------------| | OpenLLMetry SDK | Instruments LLM SDKs in your app | Spans not emitted | | OTLP collector (local or remote) | Receives spans from the SDK | Spans lost before forwarding | | Backend exporter | Ships spans to your tracing backend | Spans not arriving in dashboards | | Traceloop Cloud API | (If using managed service) Stores and displays traces | Dashboard dark | | Workflow context propagator | Threads trace context through multi-step workflows | Workflow traces become disconnected spans |

Setting Up Traceloop

Install the SDK for your language:

# Python
pip install traceloop-sdk openai

# TypeScript
npm install @traceloop/node-server-sdk

Initialize in your application:

# Python
from traceloop.sdk import Traceloop
from traceloop.sdk.decorators import workflow, task
import openai

# Initialize — auto-instruments OpenAI, Anthropic, LangChain, etc.
Traceloop.init(
    app_name="my-llm-app",
    api_key="your-traceloop-api-key",  # or omit for self-hosted OTLP backend
    # For self-hosted OTLP:
    # exporter_type="otlp",
    # otlp_endpoint="http://your-collector:4317",
)

client = openai.OpenAI()

@workflow(name="answer-user-question")
def answer_question(question: str) -> str:
    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": question}],
    )
    return response.choices[0].message.content

// TypeScript
import * as traceloop from "@traceloop/node-server-sdk";
import OpenAI from "openai";

traceloop.initialize({
  appName: "my-llm-app",
  apiKey: process.env.TRACELOOP_API_KEY,
});

const openai = new OpenAI();

// All OpenAI calls are now auto-traced
const response = await openai.chat.completions.create({
  model: "gpt-4o-mini",
  messages: [{ role: "user", content: "Hello" }],
});

Health Signals to Monitor

Traceloop Cloud API Health

If using Traceloop's managed cloud service:

GET https://api.traceloop.com/health

Expected response:

{"status": "ok"}

OTLP Collector Health (Self-Hosted)

If running your own OTLP collector (OpenTelemetry Collector):

GET http://your-collector-host:13133/

The collector's health check extension returns:

{"status": "Server available", "upSince": "...", "uptime": "..."}

OTLP Receiver Endpoint

http://your-collector-host:4317   (gRPC)
http://your-collector-host:4318   (HTTP)

Probe the HTTP endpoint with a GET to confirm it's accepting connections:

GET http://your-collector-host:4318/v1/traces

Returns 405 Method Not Allowed — confirms the path is live.

Vigilmon Monitor Setup

Monitor 1: Traceloop Cloud API Health

1. Log in to Vigilmon → Monitors → New Monitor
2. Type: HTTP
3. Method: GET
4. URL: https://api.traceloop.com/health
5. Interval: 2 minutes
6. Keyword check: ok
7. Response timeout: 10 seconds
8. Save

Monitor 2: OTLP Collector Health (Self-Hosted)

1. Type: HTTP
2. Method: GET
3. URL: http://your-collector-host:13133/
4. Interval: 2 minutes
5. Keyword check: Server available
6. Response timeout: 5 seconds
7. Save

Monitor 3: OTLP HTTP Receiver Reachability

1. Type: HTTP
2. Method: GET
3. URL: http://your-collector-host:4318/v1/traces
4. Interval: 5 minutes
5. Expected status: Any (405 is acceptable)
6. Response timeout: 10 seconds
7. Save

LLM Span Completeness Check

Traceloop emits rich spans with semantic attributes like gen_ai.usage.input_tokens, gen_ai.system, gen_ai.request.model, and gen_ai.completion. If these attributes are missing from spans, downstream analytics are broken.

# monitoring/span_completeness_check.py
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import SimpleSpanProcessor
from opentelemetry.sdk.trace.export.in_memory_span_exporter import InMemorySpanExporter
from opentelemetry import trace
import openai

REQUIRED_SPAN_ATTRIBUTES = [
    "gen_ai.system",
    "gen_ai.request.model",
    "gen_ai.usage.input_tokens",
    "gen_ai.usage.output_tokens",
]

def check_span_completeness() -> dict:
    """
    Sends a probe LLM call and verifies that the resulting span
    contains all required semantic attributes.
    """
    exporter = InMemorySpanExporter()
    provider = TracerProvider()
    provider.add_span_processor(SimpleSpanProcessor(exporter))
    trace.set_tracer_provider(provider)

    from traceloop.sdk import Traceloop
    Traceloop.init(app_name="span-completeness-probe", disable_batch=True)

    client = openai.OpenAI()
    client.chat.completions.create(
        model="gpt-4o-mini",
        max_tokens=5,
        messages=[{"role": "user", "content": "Reply: ok"}],
    )

    spans = exporter.get_finished_spans()
    if not spans:
        return {"ok": False, "reason": "no spans emitted"}

    llm_span = next(
        (s for s in spans if s.name.startswith("openai")),
        None,
    )
    if not llm_span:
        return {"ok": False, "reason": "no LLM span found"}

    attrs = dict(llm_span.attributes or {})
    missing = [k for k in REQUIRED_SPAN_ATTRIBUTES if k not in attrs]

    return {
        "ok": len(missing) == 0,
        "missing_attributes": missing,
        "span_name": llm_span.name,
    }

Expose this as a health endpoint and monitor it every 15 minutes with Vigilmon — keyword check "ok":true.

Prompt and Completion Storage Verification

Traceloop can capture prompt content and model completions in span attributes. If this is disabled (e.g., by a privacy flag or SDK version mismatch), you lose prompt-level debugging capability.

# monitoring/prompt_storage_check.py
import os
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export.in_memory_span_exporter import InMemorySpanExporter
from opentelemetry.sdk.trace.export import SimpleSpanProcessor
from opentelemetry import trace
import openai

def check_prompt_capture_enabled() -> dict:
    """
    Verifies that Traceloop is capturing prompt and completion content.
    Requires TRACELOOP_TRACE_CONTENT=true (the default).
    """
    if os.environ.get("TRACELOOP_TRACE_CONTENT", "true").lower() == "false":
        return {"ok": False, "reason": "TRACELOOP_TRACE_CONTENT is disabled — prompts not stored"}

    exporter = InMemorySpanExporter()
    provider = TracerProvider()
    provider.add_span_processor(SimpleSpanProcessor(exporter))
    trace.set_tracer_provider(provider)

    from traceloop.sdk import Traceloop
    Traceloop.init(app_name="prompt-capture-probe", disable_batch=True)

    client = openai.OpenAI()
    client.chat.completions.create(
        model="gpt-4o-mini",
        max_tokens=5,
        messages=[{"role": "user", "content": "Reply: monitoring-probe-ok"}],
    )

    spans = exporter.get_finished_spans()
    for span in spans:
        attrs = dict(span.attributes or {})
        # Traceloop captures prompts in gen_ai.prompt.0.content
        if "gen_ai.prompt.0.content" in attrs:
            content = attrs["gen_ai.prompt.0.content"]
            if "monitoring-probe-ok" in str(content):
                return {"ok": True}

    return {"ok": False, "reason": "prompt content not found in span attributes"}

Workflow Tracing Accuracy

Traceloop's key feature is threading trace context through multi-step LLM workflows. If context propagation breaks, you get isolated spans instead of a connected trace tree.

# monitoring/workflow_trace_check.py
import uuid
from traceloop.sdk import Traceloop
from traceloop.sdk.decorators import workflow, task
from opentelemetry.sdk.trace.export.in_memory_span_exporter import InMemorySpanExporter
from opentelemetry.sdk.trace.export import SimpleSpanProcessor
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry import trace
import openai


def check_workflow_context_propagation() -> dict:
    """
    Verifies that child spans in a workflow share the same trace ID as the parent.
    """
    exporter = InMemorySpanExporter()
    provider = TracerProvider()
    provider.add_span_processor(SimpleSpanProcessor(exporter))
    trace.set_tracer_provider(provider)

    Traceloop.init(app_name="workflow-probe", disable_batch=True)
    client = openai.OpenAI()

    probe_id = str(uuid.uuid4())[:8]

    @workflow(name=f"probe-workflow-{probe_id}")
    def probe_workflow():
        @task(name="step-1")
        def step_one():
            return client.chat.completions.create(
                model="gpt-4o-mini",
                max_tokens=5,
                messages=[{"role": "user", "content": "step 1"}],
            )

        @task(name="step-2")
        def step_two():
            return client.chat.completions.create(
                model="gpt-4o-mini",
                max_tokens=5,
                messages=[{"role": "user", "content": "step 2"}],
            )

        step_one()
        step_two()

    probe_workflow()

    finished = exporter.get_finished_spans()
    if len(finished) < 3:
        return {
            "ok": False,
            "reason": f"expected ≥3 spans (workflow + 2 tasks), got {len(finished)}",
        }

    trace_ids = {s.context.trace_id for s in finished}
    connected = len(trace_ids) == 1

    return {
        "ok": connected,
        "span_count": len(finished),
        "trace_ids": len(trace_ids),
        "reason": None if connected else "spans have different trace IDs — context propagation broken",
    }

End-to-End Pipeline Heartbeat

Combine the above checks into a single health endpoint that Vigilmon can poll:

# app/routes/health.py
from fastapi import APIRouter
from fastapi.responses import JSONResponse
from monitoring.span_completeness_check import check_span_completeness
from monitoring.workflow_trace_check import check_workflow_context_propagation

router = APIRouter()

@router.get("/health/traceloop-pipeline")
async def traceloop_pipeline_health():
    span_check = check_span_completeness()
    if not span_check["ok"]:
        return JSONResponse(
            {"status": "degraded", "detail": span_check},
            status_code=503,
        )

    workflow_check = check_workflow_context_propagation()
    if not workflow_check["ok"]:
        return JSONResponse(
            {"status": "degraded", "detail": workflow_check},
            status_code=503,
        )

    return {
        "status": "ok",
        "span_attributes_complete": True,
        "workflow_tracing_connected": True,
    }

Vigilmon setup:

1. URL: https://your-app.com/health/traceloop-pipeline
2. Interval: 15 minutes
3. Keyword check: "status":"ok"
4. Timeout: 60 seconds (multiple LLM calls are made)

Cost note: This heartbeat makes several LLM calls on each run. Use gpt-4o-mini or your cheapest available model, and set the interval to 15 minutes to keep probe costs under $1/day.

Monitoring Span Export Lag

If the OTLP exporter is batching spans and the batch processor has a large delay, spans can appear healthy locally but arrive late in your backend — creating gaps in time-series visualizations.

# monitoring/export_lag_check.py
import time
import httpx

def check_export_lag(backend_api_url: str, api_key: str, max_lag_seconds: int = 60) -> dict:
    """
    Compares a local timestamp with the most recent span timestamp in the backend.
    Large differences indicate export batching lag.
    """
    local_now = time.time()

    try:
        resp = httpx.get(
            f"{backend_api_url}/api/traces/latest",
            headers={"Authorization": f"Bearer {api_key}"},
            timeout=10,
        )
        data = resp.json()
        latest_span_ts = data.get("startTimeUnixNano", 0) / 1e9

        lag = local_now - latest_span_ts
        return {
            "ok": lag < max_lag_seconds,
            "lag_seconds": round(lag, 1),
            "max_allowed": max_lag_seconds,
        }
    except Exception as exc:
        return {"ok": False, "error": str(exc)}

Alerting Configuration

| Monitor | Interval | Alert when | |---------|----------|-----------| | Traceloop Cloud API health | 2 min | Any failure | | OTLP collector health | 2 min | Not Server available | | OTLP HTTP receiver | 5 min | Timeout or connection refused | | Span completeness heartbeat | 15 min | Missing required attributes | | Workflow context propagation | 15 min | Disconnected trace IDs | | Export lag check | 10 min | Lag >60s |

Traceloop collector failures and Cloud API failures should alert immediately — your LLM tracing data is accumulating a gap that can't be backfilled. Span completeness and workflow issues are high priority but not on-call-paging urgency; route them to your team's observability Slack channel.

OpenTelemetry Collector Configuration Reference

If self-hosting with the OpenTelemetry Collector, a minimal config to verify:

# otelcol-config.yaml
extensions:
  health_check:
    endpoint: 0.0.0.0:13133

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

exporters:
  otlp:
    endpoint: your-backend:4317
    tls:
      insecure: true

service:
  extensions: [health_check]
  pipelines:
    traces:
      receivers: [otlp]
      exporters: [otlp]

The health_check extension is what exposes localhost:13133/ — make sure it's enabled. Without it, you have no external way to verify collector health.

Summary

| What to monitor | Signal | Vigilmon check | |----------------|--------|---------------| | Traceloop Cloud API | /health → ok | HTTP keyword | | OTLP collector | :13133/ → Server available | HTTP keyword | | OTLP receiver | :4318/v1/traces reachable | HTTP any-status | | Span completeness | Required gen_ai.* attributes present | Custom health endpoint | | Workflow tracing | All spans share trace ID | Custom health endpoint | | Export lag | Latest backend span < 60s old | Custom health endpoint |

Traceloop is your window into LLM call behavior. Monitor it actively — silent failures in the tracing pipeline create gaps that are impossible to backfill, and they surface at the worst possible time: when you need to debug a production incident.

Further Reading

• Monitoring OpenLIT with Vigilmon
• Monitoring Helicone with Vigilmon
• Monitoring Phoenix (Arize) with Vigilmon