tutorial

Monitoring Text Generation Inference (TGI) with Vigilmon

Hugging Face TGI is a high-performance LLM inference server — here's how to monitor its HTTP API, GPU memory, token throughput, and latency with Vigilmon before inference failures hit your users.

Text Generation Inference (TGI) is Hugging Face's open-source, production-grade inference server for large language models. It powers Hugging Face's own Inference Endpoints service and is the go-to choice for self-hosted LLM deployments: continuous batching, tensor parallelism across multiple GPUs, quantization support (bitsandbytes, GPTQ, AWQ), speculative decoding, and an OpenAI-compatible streaming API. When TGI goes down or slows down, every downstream application that calls it — chatbots, RAG pipelines, code assistants — fails silently or returns errors to end users.

Vigilmon gives you continuous uptime monitoring, latency tracking, and alerting for every layer of a TGI deployment: the HTTP inference API, the Prometheus metrics endpoint, GPU resource headroom, and queue depth — so you catch problems before they become incidents.

What You'll Set Up

  • HTTP availability monitor for the TGI inference API (port 3000)
  • Liveness probe on /health for startup and runtime health
  • Prometheus metrics scrape health check (/metrics endpoint)
  • Token throughput regression alert (tokens/sec)
  • GPU memory utilization alert (>90% VRAM = OOM risk)
  • Request queue depth alert (continuous batching backlog)
  • Time-to-first-token (TTFT) latency alert
  • gRPC endpoint availability check

Prerequisites

  • TGI running and accessible (default port 3000; gRPC on 8033)
  • A free Vigilmon account

Why Monitoring TGI Is Different from a Typical Web Service

TGI is stateful in ways a standard API is not. The server loads multi-gigabyte model weights into GPU VRAM on startup — a process that can take minutes. During inference, it batches incoming requests continuously rather than handling them one-by-one, so a single slow request can raise latency for the entire batch. Token throughput and GPU memory utilization are first-class operational signals, not secondary metrics. Standard HTTP uptime checks are necessary but insufficient; you also need to watch the inference pipeline itself.

Step 1: Monitor the TGI HTTP API (Port 3000)

The TGI HTTP API is the main inference endpoint. Every application sending completions requests talks to it. Add a basic uptime monitor:

  1. Log in to vigilmon.online and click Add Monitor.
  2. Set Type to HTTP / HTTPS.
  3. Enter your TGI server URL: http://your-server:3000/health
  4. Set Check interval to 1 minute.
  5. Set Expected HTTP status to 200.
  6. Click Save.

TGI's /health endpoint returns 200 OK when the model is loaded and the server is ready to accept requests. A non-200 response means TGI is either still loading, crashed, or OOM. This is your primary liveness signal.

If TGI is behind a reverse proxy or load balancer, monitor the upstream proxy URL and add a separate monitor for the TGI instance directly, so you can distinguish proxy failures from TGI failures.

Step 2: Monitor the Prometheus Metrics Endpoint

TGI exposes a rich /metrics endpoint in Prometheus exposition format. This endpoint is your window into token throughput, queue depth, latency distributions, and GPU utilization. If /metrics goes dark, you lose all visibility into the inference pipeline.

  1. Add a second Vigilmon monitor.
  2. Set Type to HTTP / HTTPS.
  3. Enter: http://your-server:3000/metrics
  4. Set Check interval to 1 minute.
  5. Set Expected HTTP status to 200.
  6. Click Save.

Name this monitor something like "TGI Metrics Endpoint" to distinguish it from the API health monitor. If you're running a Prometheus scraper, this monitor serves as a belt-and-suspenders check that the scrape target is reachable.

Step 3: Set Up Queue Depth Alerting

TGI uses continuous batching, which means it processes tokens across many concurrent requests in parallel. When the request queue backs up beyond your SLA, new requests wait — and TTFT spikes. Queue depth is one of the most actionable TGI metrics.

TGI exposes queue depth via Prometheus:

tgi_queue_size

Scrape this metric with Prometheus and alert when the value exceeds your threshold. A common starting point is alerting when tgi_queue_size > 10 for more than 60 seconds.

If you're not running a full Prometheus stack, configure a Vigilmon cron heartbeat for a script that polls the metric and sends a heartbeat only when the queue is healthy:

#!/bin/bash
QUEUE_SIZE=$(curl -s http://your-server:3000/metrics | \
  grep '^tgi_queue_size ' | awk '{print $2}')

THRESHOLD=10
if (( $(echo "$QUEUE_SIZE < $THRESHOLD" | bc -l) )); then
  curl -fsS --retry 3 "https://vigilmon.online/api/v1/heartbeats/YOUR_HEARTBEAT_ID" > /dev/null
fi

Add this to a crontab running every 2 minutes. If the queue exceeds the threshold, the heartbeat stops sending and Vigilmon alerts you.

Step 4: Monitor Token Throughput

Token throughput (tokens per second) is the primary performance indicator for a TGI deployment. Regressions appear after model changes, config tuning, or hardware issues. Track it to detect silent degradation.

TGI exposes:

tgi_request_generated_tokens_total   # cumulative tokens generated

Derive tokens/sec by tracking the rate of this counter. With Prometheus:

rate(tgi_request_generated_tokens_total[5m])

Alert when the 5-minute rate drops below your baseline. For a first deployment, let the server run for a day, note the average throughput, and set your alert threshold at 70% of that baseline.

For a Vigilmon-native approach, wrap the check in a cron heartbeat that measures throughput by timing a known test prompt:

#!/bin/bash
START=$(date +%s%N)
RESPONSE=$(curl -s -X POST http://your-server:3000/generate \
  -H 'Content-Type: application/json' \
  -d '{"inputs": "The quick brown fox", "parameters": {"max_new_tokens": 50}}')
END=$(date +%s%N)

TOKENS=$(echo "$RESPONSE" | python3 -c "import sys,json; d=json.load(sys.stdin); print(len(d.get('details',{}).get('tokens',[])))" 2>/dev/null || echo 0)
ELAPSED_MS=$(( (END - START) / 1000000 ))

# Only send heartbeat if throughput looks healthy
if [ "$TOKENS" -gt 10 ] && [ "$ELAPSED_MS" -lt 5000 ]; then
  curl -fsS --retry 3 "https://vigilmon.online/api/v1/heartbeats/YOUR_HEARTBEAT_ID" > /dev/null
fi

Run this every 5 minutes via cron. If throughput degrades below the threshold, the heartbeat stops and you get an alert.

Step 5: GPU Memory Utilization Alert

TGI loads model weights into VRAM on startup and allocates KV cache for active requests. If VRAM runs out, inference fails with an OOM error and requests are dropped. Alert before you hit 90% VRAM utilization to give yourself a buffer.

Export GPU memory utilization as a Prometheus metric using nvidia_gpu_memory_used_bytes from the NVIDIA GPU Exporter, then alert when utilization exceeds 90%.

For a Vigilmon cron heartbeat approach:

#!/bin/bash
# Requires nvidia-smi
USED=$(nvidia-smi --query-gpu=memory.used --format=csv,noheader,nounits | head -1)
TOTAL=$(nvidia-smi --query-gpu=memory.total --format=csv,noheader,nounits | head -1)
PCT=$(echo "scale=0; $USED * 100 / $TOTAL" | bc)

# Send heartbeat only when VRAM is under 90%
if [ "$PCT" -lt 90 ]; then
  curl -fsS --retry 3 "https://vigilmon.online/api/v1/heartbeats/YOUR_HEARTBEAT_ID" > /dev/null
fi

Schedule this every 2 minutes. Name the heartbeat "TGI GPU Memory" and set the heartbeat interval to 4 minutes in Vigilmon so a single missed check doesn't alert — you want two consecutive missed checks to trigger the alert.

Step 6: Time-to-First-Token (TTFT) Monitoring

TTFT is the latency from when a streaming request arrives at TGI to when the first token is sent back. It is the primary UX latency metric for streaming LLM applications — users perceive TTFT as the "thinking pause." TGI exposes:

tgi_request_duration_seconds{quantile="0.99"}  # P99 TTFT

Alert when P99 TTFT exceeds your SLA. For a cron heartbeat approach, measure TTFT directly by timing the first token of a streaming response:

#!/bin/bash
START=$(date +%s%N)
curl -s -X POST http://your-server:3000/generate_stream \
  -H 'Content-Type: application/json' \
  -d '{"inputs": "Hello", "parameters": {"max_new_tokens": 1}}' \
  --max-time 10 > /tmp/tgi_first_token.txt
END=$(date +%s%N)

ELAPSED_MS=$(( (END - START) / 1000000 ))
SLA_MS=3000  # 3 second TTFT SLA

if [ "$ELAPSED_MS" -lt "$SLA_MS" ]; then
  curl -fsS --retry 3 "https://vigilmon.online/api/v1/heartbeats/YOUR_HEARTBEAT_ID" > /dev/null
fi

Schedule every 3 minutes. A P99 TTFT above your SLA indicates GPU saturation, batching inefficiency, or a model that needs tensor parallelism across more GPUs.

Step 7: Model Load Health (Startup Probe)

TGI takes time to load model weights on startup — anywhere from 30 seconds for small models to several minutes for large ones. If the model fails to load (corrupt weights, insufficient VRAM, invalid model ID), TGI stays in a loading state and /health returns 503.

Add a separate Vigilmon monitor for the health endpoint with a longer initial grace period to distinguish startup-time failures from runtime failures:

  1. Add a monitor for http://your-server:3000/health.
  2. Set Check interval to 2 minutes.
  3. Set Alert after to 3 consecutive failures (this gives TGI up to 6 minutes to start before alerting).
  4. Set up an alert notification to your on-call channel.

Label this monitor "TGI Model Load" to distinguish it from your runtime liveness monitor.

When you deploy a new model version, check this monitor's status before routing production traffic to the instance.

Step 8: gRPC Endpoint Health

TGI exposes a gRPC API on port 8033 for high-throughput production clients. If you have applications using the gRPC interface, monitor its availability separately from the HTTP API.

Configure a Vigilmon TCP monitor for the gRPC port:

  1. Add a monitor.
  2. Set Type to TCP.
  3. Set Host to your TGI server hostname or IP.
  4. Set Port to 8033.
  5. Set Check interval to 1 minute.
  6. Click Save.

A TCP check confirms the gRPC listener is accepting connections. For a deeper gRPC health check (using the standard gRPC health protocol), wrap a grpc_health_probe call in a cron heartbeat:

#!/bin/bash
# grpc_health_probe must be installed
grpc_health_probe -addr=your-server:8033 -connect-timeout=2s -rpc-timeout=2s
if [ $? -eq 0 ]; then
  curl -fsS --retry 3 "https://vigilmon.online/api/v1/heartbeats/YOUR_HEARTBEAT_ID" > /dev/null
fi

Alerting Configuration

Set up alerting for each monitor in Vigilmon under Settings → Notifications:

| Monitor | Alert condition | Recommended channel | |---------|----------------|---------------------| | TGI HTTP API (/health) | 1 failure | PagerDuty / Slack | | Prometheus metrics (/metrics) | 2 consecutive failures | Slack | | GPU Memory heartbeat | Missed heartbeat (2 intervals) | Slack | | TTFT heartbeat | Missed heartbeat (1 interval) | PagerDuty | | Queue depth heartbeat | Missed heartbeat (1 interval) | Slack | | gRPC TCP check | 2 consecutive failures | Slack |

For TTFT and queue depth alerts, keep the on-call rotation aware — these indicate GPU resource exhaustion and require either capacity scaling or request throttling at the application layer.

Interpreting TGI Alerts

HTTP API down (/health returning non-200): TGI crashed, is OOM, or the model failed to load. Check docker logs tgi or journalctl -u tgi for OOM kill messages or CUDA errors.

Queue depth alert: The GPU is undersized for current load. Options: increase tensor parallelism, reduce max batch size, or horizontally scale TGI instances behind a load balancer.

TTFT spike: Usually caused by queue backlog (slow batching) or long prompt prefill. Check tgi_request_prefill_duration_seconds to distinguish prefill latency from decode latency.

GPU memory >90%: Reduce max_batch_total_tokens in TGI config, use more aggressive quantization (AWQ vs bitsandbytes), or move to a larger GPU.

Prometheus metrics endpoint down: TGI may still be serving inference requests but your observability stack is blind. Restart TGI if the metrics endpoint fails to recover on its own.

Conclusion

TGI is a serious production inference server, and it deserves serious monitoring. With Vigilmon watching your HTTP API, Prometheus endpoint, GPU memory headroom, TTFT latency, and request queue depth, you get full-stack observability without a heavy metrics infrastructure. Add these monitors before your first production deployment — the 15 minutes of setup will save you from an outage you would otherwise diagnose blind.

Get started at vigilmon.online.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles