How to Monitor TruLens with Vigilmon

TruLens is the open-source LLM evaluation framework from TruEra that brings systematic quality assessment to RAG pipelines and LLM applications. Its RAG triad — context relevance, groundedness, and answer relevance — gives you feedback functions that quantify quality automatically, and its integrations with LangChain, LlamaIndex, and custom pipelines mean you can wire evaluation into any workflow. But evaluation infrastructure has its own operational requirements: if your TruLens dashboard goes down, if evaluation runs stall silently, or if the backing database fills up and rejects writes, your quality regression tests stop producing data at the worst possible time. Vigilmon keeps watch over TruLens so your evaluation guardrails stay active.

What You'll Set Up

• HTTP uptime monitor for the TruLens dashboard
• Cron heartbeat to verify evaluation runs are executing on schedule
• TCP port monitor for the backing evaluation database
• Alert channel configuration for ML team on-call

Prerequisites

• TruLens installed and running (pip install trulens-eval)
• At least one instrumented LLM app sending records to TruLens
• TruLens dashboard accessible on a known port (default: 8501)
• A free Vigilmon account

Step 1: Monitor the TruLens Dashboard

TruLens ships a Streamlit-based dashboard that surfaces evaluation scores, feedback function results, and chain traces. This is the primary interface for your ML team to catch regressions — if it goes down, quality monitoring is effectively blind.

1. Log in to vigilmon.online and click Add Monitor.
2. Set Type to HTTP / HTTPS.
3. Enter your TruLens dashboard URL: https://trulens.yourdomain.com (or http://your-server:8501 if not behind a reverse proxy).
4. Set Check interval to 2 minutes.
5. Set Expected HTTP status to 200.
6. Click Save.

If you run TruLens locally during development you can skip this step — but for shared team dashboards or CI/CD evaluation environments, this monitor is essential.

Step 2: Add a Health Endpoint to Your TruLens Setup

TruLens doesn't ship a dedicated health endpoint by default. Add a lightweight HTTP server alongside your evaluation process that Vigilmon can probe:

# trulens_health.py
from http.server import HTTPServer, BaseHTTPRequestHandler
import threading
import json

class HealthHandler(BaseHTTPRequestHandler):
    def do_GET(self):
        if self.path == "/health":
            self.send_response(200)
            self.send_header("Content-Type", "application/json")
            self.end_headers()
            self.wfile.write(json.dumps({"status": "ok"}).encode())
        else:
            self.send_response(404)
            self.end_headers()

    def log_message(self, format, *args):
        pass  # suppress access logs

def start_health_server(port=8502):
    server = HTTPServer(("0.0.0.0", port), HealthHandler)
    thread = threading.Thread(target=server.serve_forever, daemon=True)
    thread.start()
    return server

Start it alongside your evaluation process:

from trulens_eval import Tru
from trulens_health import start_health_server

tru = Tru()
start_health_server(port=8502)
tru.start_dashboard()

Then add a Vigilmon HTTP monitor on port 8502/health to confirm the Python process is alive.

Step 3: Add a Heartbeat to Verify Evaluation Runs

The most important signal TruLens produces is whether evaluation feedback functions are running against your LLM pipeline's outputs. An evaluation job that silently crashes — or a pipeline that stops submitting records — is a blind spot that no uptime check can catch.

1. In Vigilmon, click Add Monitor → Cron Heartbeat.
2. Set the expected ping interval to match your evaluation batch frequency (e.g. 60 minutes for hourly evaluation sweeps).
3. Copy the heartbeat URL.
4. Ping it after each successful evaluation batch:

import requests
from trulens_eval import Tru, TruChain, feedback

tru = Tru()
f_relevance = feedback.Feedback(feedback.OpenAI().relevance).on_input_output()

@TruChain(app_id="rag-pipeline-v2", feedbacks=[f_relevance])
def run_rag(query: str):
    return rag_chain.invoke(query)

def run_evaluation_batch(queries: list[str]):
    for query in queries:
        run_rag(query)

    # Flush and confirm to Vigilmon
    tru.get_leaderboard(app_ids=["rag-pipeline-v2"])
    requests.get("https://vigilmon.online/heartbeat/abc123", timeout=5)

If your evaluation batch job crashes or is never triggered (e.g. because a CI/CD step was skipped), the heartbeat stops and Vigilmon alerts.

Step 4: Monitor the Backing Evaluation Database

TruLens stores all evaluation records in a SQL database — SQLite by default, PostgreSQL for team/production setups. If the database fills up, becomes corrupted, or the PostgreSQL connection is lost, TruLens silently drops evaluation records.

For SQLite deployments, monitor disk space on the host server and use the health server from Step 2 to expose a disk-space check:

import shutil

class HealthHandler(BaseHTTPRequestHandler):
    def do_GET(self):
        if self.path == "/health":
            usage = shutil.disk_usage("/")
            free_pct = usage.free / usage.total
            status = 200 if free_pct > 0.1 else 503
            self.send_response(status)
            self.send_header("Content-Type", "application/json")
            self.end_headers()
            self.wfile.write(json.dumps({
                "status": "ok" if status == 200 else "disk_full",
                "free_pct": round(free_pct * 100, 1)
            }).encode())

For PostgreSQL-backed TruLens:

1. In Vigilmon, click Add Monitor → TCP Port.
2. Enter your PostgreSQL host and port 5432.
3. Set Check interval to 1 minute.

A TCP failure here means TruLens cannot persist new evaluation records.

Step 5: SSL Certificate Alerts

If your TruLens dashboard is exposed over HTTPS (recommended for team access), monitor the certificate:

1. Open the HTTP monitor for the dashboard (Step 1).
2. Enable Monitor SSL certificate under the SSL section.
3. Set Alert when certificate expires in less than 21 days.
4. Click Save.

Step 6: Configure Alert Channels

1. Go to Alert Channels in Vigilmon and add Slack, email, or a webhook.
2. Route heartbeat alerts to your ML team's on-call channel — missed evaluation batches are a model quality risk.
3. Route dashboard and database alerts to a separate DevOps channel.
4. Set Consecutive failures before alert to 2 for the dashboard monitor — Streamlit has occasional restart cycles.
5. Set heartbeat alerts to fire on the first missed ping — a missed evaluation run is always worth investigating.

Summary

| Monitor | Target | What It Catches | |---|---|---| | HTTP uptime (dashboard) | trulens.yourdomain.com | Dashboard unavailable | | HTTP uptime (health) | :8502/health | Evaluation process crash | | Cron heartbeat | Heartbeat URL | Evaluation batch not running | | TCP port | PostgreSQL :5432 | Database unreachable | | SSL certificate | Dashboard domain | TLS cert expiry |

TruLens gives your team systematic quality guardrails over LLM pipelines — but those guardrails only protect you when they're actually running. With Vigilmon watching the evaluation process, batch execution cadence, and backing database, you get the operational layer that ensures your quality assessment never goes dark when a regression is quietly shipping to production.