tutorial

Monitoring LangSmith with Vigilmon: LLM Observability Platform Health Checks

How to monitor LangSmith with Vigilmon — HTTP health checks on the LangSmith API, tracing endpoint availability, SSL certificate monitoring, and alerting for your LLM application observability stack.

LangSmith is LangChain's LLM application development and production observability platform providing distributed tracing of LLM chains, evaluation datasets, prompt versioning, and latency/cost dashboards for applications built on OpenAI, Anthropic, and open-source models. When LangSmith's tracing API goes down, your LLM applications silently stop sending traces — you lose the chain-level visibility needed to debug prompt failures, measure token costs, and track latency regressions. When the LangSmith dashboard becomes unreachable, your team cannot review trace data, evaluate prompt versions, or monitor production LLM behavior. Vigilmon gives you external uptime monitoring for LangSmith — tracing API availability, dashboard health, SSL certificate validity, and alerting before your LLM observability layer goes dark.

What You'll Build

  • An HTTP monitor on LangSmith's API endpoint to detect tracing ingestion failures
  • A monitor on the LangSmith dashboard to catch web application outages
  • An SSL certificate monitor for your LangSmith instance domain
  • Alerting rules that notify your team when LLM trace collection stops working

Prerequisites

  • A running LangSmith instance — self-hosted (https://langsmith.example.com) or LangSmith Cloud (https://api.smith.langchain.com)
  • The LangSmith API endpoint network-reachable from Vigilmon probes
  • A free account at vigilmon.online

Step 1: Understand LangSmith's Architecture

LangSmith separates the tracing backend that receives LLM chain traces from the web dashboard that developers use to review runs and manage evaluations. The critical external-facing components are:

| Component | Default path | Role | |---|---|---| | Runs API | /api/v1/runs | Receives trace spans from LangChain and LangSmith SDKs | | Sessions API | /api/v1/sessions | Manages trace sessions and project groupings | | Feedback API | /api/v1/feedback | Receives human evaluation feedback on runs | | Dashboard / frontend | / | Browser-accessible trace review and evaluation UI |

For self-hosted LangSmith, all components run under a single domain. For LangSmith Cloud, the API runs at api.smith.langchain.com and the dashboard at smith.langchain.com.

Step 2: Monitor the LangSmith Runs API

The Runs API is the most critical component — it receives all trace spans from LangChain applications, agent loops, and custom LLM pipelines. A liveness check on the runs endpoint confirms the ingestion service is accepting connections:

curl -I https://api.smith.langchain.com/api/v1/runs
# A 200, 401, or 405 confirms the endpoint is alive
  1. Log in to Vigilmon → Add Monitor → HTTP.
  2. URL: https://langsmith.example.com/api/v1/runs (or https://api.smith.langchain.com/api/v1/runs for LangSmith Cloud).
  3. Check interval: 60 seconds.
  4. Response timeout: 15 seconds.
  5. Expected status: 200 or 401 (unauthenticated GET returns 401 — set to match your instance behavior).
  6. Label: LangSmith runs API.
  7. Click Save.

This monitor catches:

  • LangSmith API container crashes
  • Database write failures preventing trace ingestion
  • Queue saturation causing trace drops in high-volume applications
  • Network policy changes that block SDK access to the runs endpoint

Alert sensitivity: Set to trigger after 1 consecutive failure. A single runs API outage means every LangChain application using LANGCHAIN_TRACING_V2=true stops sending traces immediately.

Step 3: Monitor the LangSmith Dashboard

The LangSmith web frontend serves the trace inspection and prompt evaluation UI. An HTTP check on the dashboard confirms the web layer is serving requests:

curl -I https://smith.langchain.com
# Returns 200 with text/html content
  1. Add Monitor → HTTP.
  2. URL: https://langsmith.example.com (or https://smith.langchain.com for Cloud).
  3. Check interval: 60 seconds.
  4. Response timeout: 15 seconds.
  5. Expected status: 200.
  6. Keyword: LangSmith (appears in the page title).
  7. Label: LangSmith dashboard.
  8. Click Save.

Self-hosted note: LangSmith's dashboard and API share the same application platform. Dashboard unavailability often signals a broader service degradation — check both the frontend container and the backend API when this monitor fires.

Step 4: Monitor the Sessions API

LangSmith groups traces into sessions for project-level organization. If the sessions API is degraded, new traces may be accepted but associated with incorrect projects, making trace filtering and evaluation unreliable:

curl -I https://langsmith.example.com/api/v1/sessions
# Returns 200 or 401 depending on authentication state
  1. Add Monitor → HTTP.
  2. URL: https://langsmith.example.com/api/v1/sessions.
  3. Check interval: 2 minutes.
  4. Response timeout: 10 seconds.
  5. Expected status: 200 or 401.
  6. Label: LangSmith sessions API.
  7. Click Save.

Step 5: Monitor SSL Certificates

LangSmith's TLS certificate protects both the dashboard and the API endpoint that LangChain SDKs call from production applications. An expired certificate causes SDKs to raise SSL errors, silently disabling tracing across all applications that use LangSmith:

openssl s_client -connect langsmith.example.com:443 2>/dev/null | openssl x509 -noout -dates
  1. Add Monitor → SSL Certificate.
  2. Domain: langsmith.example.com.
  3. Alert when expiry is within: 30 days.
  4. Alert again: 14 days, 7 days, 3 days, 1 day.
  5. Click Save.

Why SSL matters for LangSmith specifically: LangChain SDKs call the LangSmith API asynchronously in background threads. An expired certificate causes silent SSL failures — your LLM application continues to work but all traces are dropped without any error reaching your application logs or users.

Step 6: Configure Alerting

In Vigilmon under Settings → Notifications, configure your alert channels:

| Monitor | Trigger | Action | |---|---|---| | Runs API | Non-200/401 response or timeout | Check LangSmith backend pod; review trace queue depth; inspect container logs | | Dashboard | Non-200 or keyword missing | LangSmith frontend degraded; check web container health | | Sessions API | Unexpected status | Trace project association may fail; new sessions not created correctly | | SSL certificate | < 30 days to expiry | Renew certificate; verify LangChain SDKs can still connect after renewal |

Alert after: 1 consecutive failure for the runs API. 2 consecutive failures for dashboard and sessions monitors.

Common LangSmith Failure Modes and What Vigilmon Catches

| Scenario | Vigilmon monitor | |---|---| | API container OOM killed | Runs API monitor fires; all LangChain traces stop silently | | Database write saturation under high trace volume | API returns 500; monitor fires within 60 s | | ClickHouse storage full (self-hosted) | Trace writes fail; runs API monitor fires | | SSL certificate expired | SSL monitor alerts at 30-day threshold; LangChain SDK raises SSL errors | | Sessions API degraded | Sessions monitor fires; trace project grouping breaks | | Kubernetes pod restart during high-traffic period | Runs API monitor fires; in-flight traces may be dropped | | DNS cutover failure after LangSmith migration | All monitors fire; production LLM apps lose tracing | | High LLM call volume causes trace queue overflow | API response time increases; Vigilmon timeout alert fires | | Frontend build failure after upgrade | Dashboard monitor fires; API may still accept traces | | Feedback API degraded | Evaluation pipelines cannot log human feedback |

Monitoring Trace Completeness vs. Service Availability

Vigilmon monitors whether LangSmith's endpoints are reachable — it does not verify that traces are being ingested and stored end-to-end. For deeper data quality monitoring:

  • Trace gap detection: If LangSmith shows no new traces for an expected active application, check whether the runs API monitor is healthy and inspect the LangChain SDK environment variable LANGCHAIN_TRACING_V2.
  • Sampling-induced gaps: LangSmith supports trace sampling — a drop in trace volume may reflect intentional sampling configuration rather than a service outage. Verify monitoring status before escalating.
  • Async flush behavior: LangChain SDKs flush traces asynchronously. After an API recovery, check whether buffered traces from the outage window were successfully flushed or dropped.

Vigilmon catches infrastructure-level failures — endpoint unreachability, certificate expiry, service crashes. End-to-end trace completeness and LLM data quality monitoring require LangSmith's internal metrics alongside external uptime monitoring.

LangSmith gives you full visibility into your LLM chains, agent loops, and prompt evaluations, but your observability platform itself needs monitoring. A LangSmith API outage silently stops tracing for every LangChain application in production — you lose the chain-level context you need to debug failures and track costs. Vigilmon checks LangSmith's runs API, dashboard, and SSL certificate on a 60-second cycle, so you know within a minute when your LLM observability infrastructure stops collecting traces.

Start monitoring LangSmith in under 5 minutes — register free at vigilmon.online.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles