How to Monitor OpenLLM with Vigilmon

OpenLLM is an open-source platform for running, fine-tuning, and deploying large language models in production. Built by BentoML, it lets you serve models like Llama, Mistral, Falcon, and other open-weight LLMs with a standardized OpenAI-compatible API — giving you model runner lifecycle management, streaming inference, and multi-model routing without cloud vendor lock-in.

But LLM serving failures are notoriously opaque. A model that stalled mid-generation and left a request hanging, a GPU worker that ran out of VRAM and is silently OOM-killing requests, or a multi-model router that dropped one backend out of rotation may look fine to a process monitor while your downstream applications time out. Vigilmon adds the external visibility layer your LLM infrastructure needs: uptime checks on inference endpoints, latency threshold alerts, and heartbeat monitoring for async inference pipelines.

Why OpenLLM Needs External Monitoring

OpenLLM exposes internal metrics and logs — but those require active observation. External monitoring with Vigilmon adds:

• Availability checks that detect when inference endpoints stop responding entirely, not just when they log errors
• Latency alerting so you catch inference slowdowns before users notice degraded response quality
• Token throughput monitoring to spot when model loading or CUDA contention is throttling generation speed
• Multi-model routing health checks that verify all backends in a model ensemble are serving
• Heartbeat monitoring for async batch inference jobs that must prove forward progress

Step 1: Verify OpenLLM's Health and Metrics Endpoints

OpenLLM exposes a built-in /readyz readiness endpoint and a /metrics Prometheus endpoint on every server instance. Check they are accessible:

# Start OpenLLM with a model (e.g., Mistral 7B)
openllm start mistral --model-id mistralai/Mistral-7B-Instruct-v0.1

# Verify readiness endpoint
curl http://localhost:3000/readyz
# Expected: {"status": "ready"}

# Verify OpenAI-compatible health
curl http://localhost:3000/v1/models
# Expected: JSON list of loaded models

# Verify Prometheus metrics
curl http://localhost:3000/metrics | head -30

OpenLLM also exposes a /v1/models endpoint (OpenAI-compatible) listing all currently loaded models. This is the best target for an availability check because it proves both that the server is up and that at least one model is loaded and ready to serve.

If you run multiple models on separate ports, check each:

# Model-specific readiness
curl http://localhost:3000/readyz   # Mistral 7B
curl http://localhost:3001/readyz   # Llama 3 8B
curl http://localhost:3002/readyz   # Falcon 7B

Step 2: Build an OpenLLM Health Endpoint

Write a lightweight health proxy that performs an end-to-end inference check — a short prompt that validates the model is actually generating, not just reporting ready:

# openllm_health.py
from flask import Flask, jsonify
import requests, os, time

app = Flask(__name__)

OPENLLM_BASE_URL = os.environ.get('OPENLLM_BASE_URL', 'http://localhost:3000')
LATENCY_WARN_MS  = int(os.environ.get('LATENCY_WARN_MS', '5000'))
LATENCY_CRIT_MS  = int(os.environ.get('LATENCY_CRIT_MS', '15000'))
PROBE_MODEL      = os.environ.get('PROBE_MODEL', 'mistralai/Mistral-7B-Instruct-v0.1')
PROBE_PROMPT     = os.environ.get('PROBE_PROMPT', 'Say "ok"')

@app.route('/health/openllm')
def health():
    issues = []

    # Readiness check
    try:
        r = requests.get(f'{OPENLLM_BASE_URL}/readyz', timeout=5)
        if r.status_code != 200 or r.json().get('status') != 'ready':
            return jsonify(status='down', reason='not_ready'), 503
    except Exception as e:
        return jsonify(status='down', reason=f'readyz_failed: {e}'), 503

    # Models list check
    try:
        r = requests.get(f'{OPENLLM_BASE_URL}/v1/models', timeout=5)
        models = r.json().get('data', [])
        model_ids = [m['id'] for m in models]
        if not model_ids:
            issues.append('no_models_loaded')
    except Exception as e:
        issues.append(f'models_endpoint_error: {e}')

    # Inference latency probe
    latency_ms = None
    try:
        payload = {
            'model': PROBE_MODEL,
            'messages': [{'role': 'user', 'content': PROBE_PROMPT}],
            'max_tokens': 5,
            'stream': False,
        }
        t0 = time.time()
        r = requests.post(
            f'{OPENLLM_BASE_URL}/v1/chat/completions',
            json=payload, timeout=LATENCY_CRIT_MS / 1000 + 5
        )
        latency_ms = int((time.time() - t0) * 1000)
        if r.status_code != 200:
            issues.append(f'inference_error_{r.status_code}')
        elif latency_ms > LATENCY_CRIT_MS:
            issues.append(f'inference_critical_latency_{latency_ms}ms')
        elif latency_ms > LATENCY_WARN_MS:
            issues.append(f'inference_slow_{latency_ms}ms')
    except requests.Timeout:
        issues.append(f'inference_timeout_{LATENCY_CRIT_MS}ms')
    except Exception as e:
        issues.append(f'inference_probe_failed: {e}')

    if issues:
        return jsonify(status='degraded', issues=issues, latency_ms=latency_ms), 503

    return jsonify(
        status='ok',
        models=model_ids,
        latency_ms=latency_ms,
    ), 200

@app.route('/health/openllm/throughput')
def throughput():
    """Parse token throughput from Prometheus metrics."""
    try:
        r = requests.get(f'{OPENLLM_BASE_URL}/metrics', timeout=5)
        lines = r.text.splitlines()
        metrics = {}
        for line in lines:
            if line.startswith('#'):
                continue
            if 'tokens_total' in line or 'generation_tokens' in line or 'request_success' in line:
                parts = line.split()
                if len(parts) == 2:
                    metrics[parts[0]] = parts[1]
        return jsonify(status='ok', metrics=metrics), 200
    except Exception as e:
        return jsonify(status='error', reason=str(e)), 503

if __name__ == '__main__':
    app.run(host='0.0.0.0', port=3010)

Run the sidecar alongside your OpenLLM server:

OPENLLM_BASE_URL=http://localhost:3000 \
PROBE_MODEL=mistralai/Mistral-7B-Instruct-v0.1 \
LATENCY_WARN_MS=5000 \
LATENCY_CRIT_MS=20000 \
python openllm_health.py

Step 3: Multi-Model Routing Health Checks

When you run multiple models behind an OpenLLM server or a custom router, each backend must be monitored independently. A routing failure that drops one model out of the pool degrades quality even if aggregate availability looks normal.

# multi_model_health.py
import requests, os
from flask import Flask, jsonify

app = Flask(__name__)

# Comma-separated: "http://host1:3000,http://host2:3001"
BACKENDS = os.environ.get('OPENLLM_BACKENDS', 'http://localhost:3000').split(',')

@app.route('/health/openllm/routing')
def routing():
    results = []
    all_ok = True

    for backend in BACKENDS:
        backend = backend.strip()
        try:
            r = requests.get(f'{backend}/readyz', timeout=5)
            models_r = requests.get(f'{backend}/v1/models', timeout=5)
            models = [m['id'] for m in models_r.json().get('data', [])]
            status = 'ok' if r.status_code == 200 and models else 'degraded'
        except Exception as e:
            status = 'down'
            models = []
            all_ok = False

        if status != 'ok':
            all_ok = False

        results.append({'backend': backend, 'status': status, 'models': models})

    http_status = 200 if all_ok else 503
    return jsonify(
        status='ok' if all_ok else 'degraded',
        backends=results,
    ), http_status

if __name__ == '__main__':
    app.run(host='0.0.0.0', port=3011)

This endpoint returns 503 as soon as any backend is unreachable, giving Vigilmon a clean binary signal for multi-model routing health.

Step 4: Configure Vigilmon HTTP Monitors

1. Log in to vigilmon.online and go to Monitors → New Monitor
2. Choose HTTP / HTTPS
3. Set the URL to your OpenLLM health endpoint: https://your-app.example.com/health/openllm
4. Set the check interval to 1 minute
5. Under Expected response, configure:
   • Status code: 200
   • Response body contains: "status":"ok"
   • Response time threshold: 30000ms (inference probes take longer than typical HTTP checks)
6. Under Alert channels, assign your Slack or PagerDuty integration
7. Save the monitor

Add monitors for each aspect of your LLM infrastructure:

| Monitor URL | Purpose | Interval | |---|---|---| | /health/openllm | Readiness + inference latency probe | 2 min | | /health/openllm/throughput | Token throughput metrics | 5 min | | /health/openllm/routing | Multi-model backend availability | 1 min | | OpenLLM /readyz direct | Raw readiness (fast, no inference probe) | 30 sec |

For the raw readiness monitor, point Vigilmon directly at http://your-host:3000/readyz — this gives you a 30-second availability check without incurring inference latency on every probe.

Step 5: Heartbeat Monitoring for Async Inference Pipelines

Many OpenLLM deployments include batch inference workers — jobs that pull prompts from a queue, run inference, and write completions back. If the batch worker stalls (VRAM OOM, network partition, model loading hang), no error surfaces in the server's readiness check.

Vigilmon heartbeat monitors detect this: your batch worker pings Vigilmon after each successful batch. Silence triggers an alert.

Set Up the Heartbeat Monitor

1. In Vigilmon, go to Monitors → New Monitor → Heartbeat
2. Set the name: openllm-batch-worker
3. Set the expected interval: 10 minutes
4. Set the grace period: 15 minutes
5. Save — copy the heartbeat URL: https://vigilmon.online/heartbeat/abc123xyz

Wire Into Your Batch Worker

# batch_inference_worker.py
import requests, os, time

VIGILMON_URL  = os.environ['VIGILMON_HEARTBEAT_URL']
OPENLLM_URL   = os.environ.get('OPENLLM_BASE_URL', 'http://localhost:3000')
BATCH_SIZE    = int(os.environ.get('BATCH_SIZE', '10'))

def run_inference(prompt, model):
    r = requests.post(
        f'{OPENLLM_URL}/v1/chat/completions',
        json={
            'model': model,
            'messages': [{'role': 'user', 'content': prompt}],
            'max_tokens': 512,
        },
        timeout=120
    )
    r.raise_for_status()
    return r.json()['choices'][0]['message']['content']

def ping_vigilmon():
    try:
        requests.get(VIGILMON_URL, timeout=5)
    except Exception:
        pass

def main():
    processed = 0
    model = os.environ.get('INFERENCE_MODEL', 'mistralai/Mistral-7B-Instruct-v0.1')

    for prompt in get_prompts_from_queue():  # your queue implementation
        result = run_inference(prompt, model)
        save_result(result)                  # your storage implementation
        processed += 1

        if processed % BATCH_SIZE == 0:
            ping_vigilmon()
            print(f"Processed {processed} prompts, heartbeat sent")

if __name__ == '__main__':
    main()

Set BATCH_SIZE to a count that completes within half the heartbeat interval — if your heartbeat is 10 minutes, send a ping every 5 minutes of processing.

Step 6: Alert Routing

| Monitor | Alert Channel | Priority | |---|---|---| | Raw readiness /readyz | PagerDuty | P1 | | Inference latency probe /health/openllm | Slack + PagerDuty | P1 | | Multi-model routing /health/openllm/routing | Slack + PagerDuty | P1 | | Token throughput /health/openllm/throughput | Slack | P2 | | Heartbeat: batch worker | Slack + email | P2 |

Latency threshold guidelines:

• 5000ms inference latency: warn — model may be cold-starting or GPU is under memory pressure
• 15000ms inference latency: critical — generation is stalled, likely OOM or deadlocked worker
• 30000ms response time on health endpoint: treat as down — the probe itself timed out

For GPU-heavy deployments, also monitor the host machine's GPU utilization via a separate endpoint. Sustained 100% GPU utilization with rising latency usually indicates memory pressure before an OOM crash.

Summary

OpenLLM inference failures are silent until your downstream applications start timing out. External monitoring with Vigilmon catches them first:

| Monitor Type | What It Covers | |---|---| | HTTP monitor on /readyz | Server and model loading readiness | | HTTP monitor on /health/openllm | End-to-end inference latency and model availability | | HTTP monitor on /health/openllm/routing | Multi-model backend pool health | | HTTP monitor on /health/openllm/throughput | Token throughput and Prometheus metrics | | Heartbeat monitor | Async batch inference worker liveness |

Get started free at vigilmon.online — your first OpenLLM monitor is running in under two minutes.