How to Monitor vLLM with Vigilmon (LLM Inference Serving)

Running LLM inference at scale with vLLM is powerful — PagedAttention, continuous batching, and multi-GPU tensor parallelism let you serve models like Llama-3, Mistral, and Qwen at production throughput. But production LLM serving introduces a class of silent failures that neither your orchestrator nor your model metrics will catch on their own.

In this tutorial you'll set up external uptime monitoring for your vLLM inference server using Vigilmon — free tier, no credit card required.

Why vLLM needs external monitoring

vLLM exposes an OpenAI-compatible REST API. Your orchestration layer might watch GPU utilization, queue depth, and tokens-per-second. But those metrics are internal. External monitoring tells you what your applications actually see when they call your inference endpoint:

The HTTP server starts but hangs before loading the model — GPU OOM during weight loading leaves the API process alive but not responding to /health
CUDA driver mismatch after a kernel upgrade — vLLM starts, logs no errors, but inference requests return 500
Port binding conflict when a previous worker didn't exit cleanly — the new process starts on a different port or fails silently
Network path issues between your app servers and the GPU node — the model is healthy but firewall rules, load balancer changes, or SDN configuration breaks the path
Tensor-parallel coordinator dies without killing worker processes — throughput collapses but the endpoint still accepts requests and returns errors

The pattern is the same: your internal dashboards stay green while your applications see failures. External monitoring from a neutral vantage point catches what internal metrics miss.

What you'll need

A running vLLM server (single-GPU or multi-GPU)
The vLLM OpenAI-compatible API accessible on a reachable endpoint
A free Vigilmon account (sign up takes 30 seconds)

Step 1: Verify your vLLM health endpoint

vLLM ships with a /health endpoint out of the box since v0.2.0. Start the server with your chosen model:

python -m vllm.entrypoints.openai.api_server \
  --model meta-llama/Meta-Llama-3-8B-Instruct \
  --host 0.0.0.0 \
  --port 8000 \
  --max-model-len 8192 \
  --tensor-parallel-size 1

Once the server is running, verify the health endpoint responds:

curl -s http://localhost:8000/health
# Returns 200 OK when the server is ready

For version information and model details:

curl -s http://localhost:8000/v1/models | python3 -m json.tool

If you're running behind a reverse proxy (nginx, Caddy, Traefik), make sure /health is forwarded and not blocked by authentication middleware. Vigilmon needs to reach the raw health endpoint without tokens.

nginx proxy example

server {
    listen 443 ssl;
    server_name inference.yourdomain.com;

    # Health check — no auth required
    location /health {
        proxy_pass http://127.0.0.1:8000/health;
        proxy_set_header Host $host;
    }

    # API — may require auth
    location /v1/ {
        proxy_pass http://127.0.0.1:8000/v1/;
        proxy_set_header Host $host;
        proxy_read_timeout 300s;  # LLM generation can be slow
    }
}

Step 2: Create an HTTP monitor in Vigilmon

Click Monitors → New Monitor
Set the Type to HTTP
Enter your endpoint URL: https://inference.yourdomain.com/health
Set Check interval to 1 minute — LLM inference downtime has high cost; catch it fast
Set Expected status to 200
Under Alert after, set 1 failure — LLM serving failures are rarely transient; alert immediately
Click Save

Vigilmon will begin checking from multiple geographic regions every minute.

What the health check catches

| Failure scenario | What Vigilmon sees | |-----------------|-------------------| | vLLM process crashed | HTTP connection refused → alert | | Model loading OOM killed | HTTP connection refused → alert | | CUDA driver issue returning 500s | HTTP 500 → alert | | nginx/reverse proxy misconfigured | HTTP 502/504 → alert | | DNS failure for inference domain | DNS resolution failure → alert | | TLS certificate expired | SSL handshake failure → alert |

Step 3: Add a TCP monitor for the raw inference port

If vLLM is on a dedicated GPU node accessible by internal hostname, add a TCP monitor to catch network-level failures separately from application-level failures:

Click Monitors → New Monitor
Set the Type to TCP
Enter: gpu-node-01.internal:8000
Set Check interval to 1 minute
Click Save

This catches cases where the HTTP layer is unhealthy but you want to distinguish "port not listening" (process crashed) from "port listening but returning errors" (application-level fault).

Step 4: Monitor inference latency with keyword checks

vLLM's /health endpoint returns a simple 200 but doesn't tell you about warm vs. cold model state. Add a secondary check against the models endpoint to verify the model is actually loaded:

Click Monitors → New Monitor
Set the Type to HTTP
Enter: https://inference.yourdomain.com/v1/models
Set Keyword must contain: meta-llama (or your model family)
Set Check interval: 5 minutes
Click Save

This check fails if vLLM starts but the model doesn't finish loading — a common failure mode when the GPU node has less VRAM than expected.

Step 5: Configure alerts

Navigate to Alerts → New Alert Channel and configure where Vigilmon sends notifications:

Email alerts:

Add your on-call email or engineering distribution list
Set escalation to page again if the issue persists after 15 minutes

Slack alerts:

Create an incoming webhook in your Slack workspace
In Vigilmon, go to Alerts → Slack
Paste the webhook URL
Map it to your #ml-ops or #inference-alerts channel

PagerDuty / OpsGenie:

Use Vigilmon's webhook alert channel
Point it at your PagerDuty Events API v2 endpoint or OpsGenie alert API

Example webhook payload Vigilmon sends when the inference server goes down:

{
  "monitor_name": "vLLM /health",
  "status": "down",
  "url": "https://inference.yourdomain.com/health",
  "started_at": "2026-01-15T10:23:00Z",
  "duration_seconds": 45
}

Step 6: Correlate Vigilmon alerts with vLLM logs

When an alert fires, follow this triage runbook:

# 1. Check if the vLLM process is still running
systemctl status vllm
# or
ps aux | grep vllm

# 2. Check for GPU OOM errors
dmesg | grep -i "killed process" | tail -20
journalctl -u vllm --since "10 minutes ago" | grep -E "OOM|CUDA|error" -i

# 3. Check CUDA device availability
nvidia-smi

# 4. Check if the port is bound
ss -tlnp | grep 8000

# 5. Try a direct health check bypassing the proxy
curl -v http://localhost:8000/health

Decision tree:

Port not bound → vLLM process crashed; check journalctl -u vllm for CUDA/OOM errors, restart service
Port bound, 200 from localhost but not from Vigilmon → proxy or network issue; check nginx logs
Port bound, 500 from localhost → CUDA driver issue or model loading failure; check nvidia-smi and vLLM logs
Port bound, models endpoint missing the model → model still loading or failed to load weights

Step 7: Create a status page for your inference infrastructure

If you run multiple models or serve multiple teams:

Go to Status Pages → New Status Page
Name it: "ML Inference Infrastructure"
Add monitors grouped by model:
- Llama-3 8B: /health check + /v1/models keyword check
- Mistral 7B: same pair
- Embedding service: separate endpoint
Publish the page

Share the status page URL with:

Application teams consuming your inference API
On-call engineers who need visibility without SSH access to GPU nodes
Leadership who want to understand inference availability SLAs

vLLM deployment patterns and monitoring implications

Single-node serving

[Application] → [nginx proxy] → [vLLM on localhost:8000]

Monitor: https://inference.yourdomain.com/health (HTTP) + hostname:8000 (TCP)

Multi-node tensor parallelism

[Application] → [Load Balancer] → [vLLM node-0 (coordinator)]
                                → [vLLM node-1 (worker)]

Monitor: the load balancer endpoint (HTTP) + each node's port (TCP). The coordinator dying while workers stay up is a common failure mode.

Kubernetes deployment

# vllm-service.yaml
apiVersion: v1
kind: Service
metadata:
  name: vllm-inference
spec:
  selector:
    app: vllm
  ports:
    - port: 80
      targetPort: 8000
  type: LoadBalancer
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: vllm
spec:
  replicas: 1
  selector:
    matchLabels:
      app: vllm
  template:
    spec:
      containers:
        - name: vllm
          image: vllm/vllm-openai:latest
          args:
            - "--model"
            - "meta-llama/Meta-Llama-3-8B-Instruct"
            - "--host"
            - "0.0.0.0"
          resources:
            limits:
              nvidia.com/gpu: 1
          livenessProbe:
            httpGet:
              path: /health
              port: 8000
            initialDelaySeconds: 120  # Model loading takes time
            periodSeconds: 30
          readinessProbe:
            httpGet:
              path: /health
              port: 8000
            initialDelaySeconds: 120
            periodSeconds: 10

For Kubernetes vLLM deployments, monitor the LoadBalancer external IP in addition to the internal readiness probes — the probe only tells you what the scheduler sees, not what users see.

Putting it all together

| Monitor | Type | Purpose | |---------|------|---------| | https://inference.yourdomain.com/health | HTTP | Catch process crashes, CUDA failures, proxy issues | | https://inference.yourdomain.com/v1/models | HTTP keyword | Verify model is loaded and serving | | gpu-node-01:8000 | TCP | Distinguish network failures from application failures |

Combined, these three monitors give you full-stack visibility: is the network path open? Is the process listening? Is the model loaded and responding?

What's next

Heartbeat monitors — if you run periodic model evaluation scripts or fine-tuning jobs, use Vigilmon heartbeats to alert when scheduled jobs stop reporting
SSL certificate monitoring — Vigilmon tracks your inference endpoint's TLS certificate and alerts before expiry
Multi-environment coverage — add separate monitors for dev, staging, and production inference endpoints

Get started free at vigilmon.online — no credit card, monitors start running in under a minute.