How to Monitor Literal AI with Vigilmon

Literal AI is the LLM observability platform that lets you trace every thread, manage evaluation datasets, replay prompts in the playground, and run human annotation workflows — giving teams building LLM applications the systematic quality tracking they need to move fast without breaking things. But your observability tooling has its own uptime requirements: if Literal AI's ingestion API goes down, your production traces are lost, regressions go undetected, and your human evaluation pipeline stalls. Vigilmon monitors Literal AI's own availability so you have a meta-layer of observability over your observability stack.

What You'll Set Up

• HTTP uptime monitor for the Literal AI API endpoint
• Cron heartbeat to verify trace ingestion is flowing
• SSL certificate monitoring for self-hosted Literal AI instances
• Alert channel configuration for immediate on-call notification

Prerequisites

• Literal AI cloud account or self-hosted instance
• At least one LLM application sending traces to Literal AI
• A free Vigilmon account

Step 1: Monitor the Literal AI API Endpoint

Your application sends traces to Literal AI via its GraphQL or REST API. If this endpoint goes down, traces are silently dropped and your evaluation pipeline sees a data gap.

1. Log in to vigilmon.online and click Add Monitor.
2. Set Type to HTTP / HTTPS.
3. Enter the Literal AI API URL. For cloud: https://cloud.getliteral.ai. For self-hosted: https://your-literal-domain.com.
4. Set Check interval to 1 minute.
5. Set Expected HTTP status to 200.
6. Click Save.

This confirms the API gateway is reachable from the public internet, which is the same network path your application uses.

Step 2: Add a Heartbeat to Verify Trace Ingestion

An HTTP uptime check tells you the endpoint is reachable, but not that it's actually ingesting data. A stalled trace ingestion pipeline (e.g. a backing database that's up but rejecting writes) will pass an uptime check while silently losing all your production traces.

Add a Vigilmon cron heartbeat by instrumenting your LLM application to ping Vigilmon after successfully sending a trace batch:

1. In Vigilmon, click Add Monitor → Cron Heartbeat.
2. Set the expected ping interval slightly above your trace flush interval (e.g. 5 minutes if your SDK flushes every 3 minutes).
3. Copy the heartbeat URL.
4. Add the heartbeat ping after a successful trace flush in your application:

import literalai
import httpx

client = literalai.LiteralClient(api_key=os.getenv("LITERAL_API_KEY"))

async def run_llm_step(user_message: str):
    with client.thread(name="user-session") as thread:
        with client.step(type="llm", name="generate") as step:
            response = await call_llm(user_message)
            step.output = {"content": response}

    # Confirm trace delivery to Vigilmon
    async with httpx.AsyncClient() as http:
        await http.get("https://vigilmon.online/heartbeat/abc123")

    return response

If your application stops sending traces — due to an SDK misconfiguration, a crashed worker, or a Literal AI write failure — the heartbeat stops firing and Vigilmon alerts.

Step 3: Monitor the Literal AI Evaluation Dashboard

Literal AI's dataset and annotation dashboard is a separate web surface from the ingestion API. Your human evaluators access it directly; if it goes down, annotation sessions are blocked.

1. In Vigilmon, add a second HTTP monitor.
2. Set the URL to https://cloud.getliteral.ai/dashboard (or your self-hosted equivalent).
3. Set Check interval to 5 minutes — dashboard availability is less critical than ingestion.
4. Set Expected HTTP status to 200.

This gives you a separate alert stream for dashboard-layer failures that don't affect trace ingestion.

Step 4: SSL Certificate Alerts for Self-Hosted Literal AI

Self-hosted Literal AI deployments typically run behind a reverse proxy with a TLS certificate. A lapsed certificate breaks both your application's trace SDK (which sends over HTTPS) and your evaluators' browser access to the dashboard.

1. Open the HTTP monitor for your self-hosted instance.
2. Enable Monitor SSL certificate under the SSL section.
3. Set Alert when certificate expires in less than 21 days.
4. Click Save.

A 21-day alert window gives you enough lead time to renew manually if your automated renewal (certbot, Traefik ACME) fails silently.

Step 5: Monitor the Backing Database (Self-Hosted)

Self-hosted Literal AI relies on a PostgreSQL database for trace storage and a Redis cache for queuing. Add TCP monitors to catch infrastructure failures before they cause trace loss:

1. In Vigilmon, click Add Monitor → TCP Port.
2. Add monitors for each backing service:

| Service | Default Port | What It Catches | |---|---|---| | PostgreSQL | 5432 | Trace storage unavailable | | Redis | 6379 | Ingestion queue failure | | Literal AI API | 3000 | API process crash |

For each monitor, set Check interval to 1 minute and configure alerts to fire after 2 consecutive failures.

Step 6: Configure Alert Channels

1. Go to Alert Channels in Vigilmon and add Slack, email, or a webhook to your ML team's on-call rotation.
2. Route API and heartbeat alerts to a high-priority channel — these indicate trace data loss.
3. Route dashboard alerts to a lower-priority channel — these affect evaluator productivity but not production data integrity.
4. Use Maintenance windows when upgrading your self-hosted Literal AI deployment:

curl -X POST https://vigilmon.online/api/maintenance \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -d '{"monitor_id": "abc123", "duration_minutes": 20}'

Summary

| Monitor | Target | What It Catches | |---|---|---| | HTTP uptime (API) | cloud.getliteral.ai | API gateway down | | Cron heartbeat | Heartbeat URL | Trace ingestion stalled | | HTTP uptime (dashboard) | /dashboard | Evaluator access blocked | | SSL certificate | Self-hosted domain | TLS cert expiry | | TCP port | PostgreSQL, Redis | Backing store failure |

Literal AI gives you deep visibility into your LLM application's quality — but that visibility has its own availability requirements. With Vigilmon monitoring the ingestion API, trace delivery cadence, and self-hosted infrastructure, you ensure your observability stack stays healthy so you never miss a regression in production.