Monitoring TorchServe with Vigilmon

TorchServe lets ML teams deploy trained PyTorch models as REST and gRPC endpoints with built-in batching, versioning, and A/B testing. In production, inference latency spikes, worker saturation, and model registration failures are silent until users start complaining. Vigilmon gives you continuous monitoring of TorchServe's inference endpoint, management API, metrics, and GPU utilization — so you catch serving degradation before it becomes a customer-facing incident.

What You'll Set Up

• HTTP uptime monitor for the inference endpoint (port 8080)
• Management API health check (port 8081)
• Metrics endpoint availability (port 8082)
• Model registration health verification
• Inference latency heartbeat
• Worker utilization alerting
• SSL certificate expiry alerts

Prerequisites

• TorchServe installed and running with at least one registered model
• Inference endpoint accessible at port 8080 (or a reverse-proxied domain)
• A free Vigilmon account

Step 1: Monitor the Inference Endpoint (Port 8080)

Port 8080 is TorchServe's primary inference endpoint — the URL client applications call for predictions. If it's down, every prediction request fails.

1. Log in to vigilmon.online and click Add Monitor.
2. Set Type to HTTP / HTTPS.
3. Enter the inference server URL: http://torchserve.yourdomain.com:8080/ping.
4. Set Check interval to 1 minute.
5. Set Expected HTTP status to 200.
6. Click Save.

TorchServe provides a /ping endpoint at the inference port that returns {"status": "Healthy"} when the server is ready:

curl http://localhost:8080/ping
# {"status": "Healthy"}

This is the canonical liveness probe — prefer it over the root URL.

Step 2: Monitor the Management API (Port 8081)

Port 8081 is TorchServe's management REST API for registering new models, scaling workers, and checking model status. If it's unreachable, you can't add or update models in production.

1. Click Add Monitor → HTTP / HTTPS.
2. Enter: http://torchserve.yourdomain.com:8081/models.
3. Set Expected HTTP status to 200.
4. Set Check interval to 2 minutes.
5. Click Save.

The GET /models endpoint lists all registered models and their status. A healthy response looks like:

{
  "models": [
    {"modelName": "resnet50", "modelUrl": "resnet50.mar"},
    {"modelName": "bert-classifier", "modelUrl": "bert-classifier.mar"}
  ]
}

If the management API returns a non-200 or is unreachable, model registration and scaling operations will fail silently.

Step 3: Monitor the Metrics Endpoint (Port 8082)

Port 8082 exposes Prometheus-compatible metrics for inference request rates, latency histograms, and worker utilization. Monitoring this endpoint confirms TorchServe's metrics pipeline is healthy.

1. Click Add Monitor → HTTP / HTTPS.
2. Enter: http://torchserve.yourdomain.com:8082/metrics.
3. Set Expected HTTP status to 200.
4. Enable Keyword check and look for ts_inference_requests_total in the response body.
5. Set Check interval to 2 minutes.
6. Click Save.

A healthy metrics response contains Prometheus-format lines like:

# HELP ts_inference_requests_total Total number of inference requests
# TYPE ts_inference_requests_total counter
ts_inference_requests_total{model_name="resnet50",level="model"} 1024

If the metrics endpoint is down, your Prometheus/Grafana stack loses inference data silently.

Step 4: Verify Model Registration Health

Not all model registration failures surface as HTTP errors — a model can be registered but in a failed state. Use a Vigilmon HTTP monitor with a keyword check to verify production models are actually loaded.

1. Click Add Monitor → HTTP / HTTPS.
2. Enter: http://torchserve.yourdomain.com:8081/models/resnet50 (replace with your model name).
3. Set Expected HTTP status to 200.
4. Enable Keyword check and look for "status": "READY" in the response body.
5. Set Check interval to 5 minutes.
6. Click Save.

The GET /models/{model_name} endpoint returns detailed model status:

[{
  "modelName": "resnet50",
  "modelVersion": "1.0",
  "modelUrl": "resnet50.mar",
  "runtime": "python",
  "minWorkers": 1,
  "maxWorkers": 4,
  "batchSize": 1,
  "status": "READY"
}]

A "status": "LOADING" or "FAILED" for a production model is an immediate incident.

Step 5: Inference Latency Heartbeat

End-to-end inference latency is the metric users feel most directly. Use a Vigilmon cron heartbeat combined with an external latency probe to track p95 latency trends.

1. Click Add Monitor → Cron Heartbeat.
2. Set the expected ping interval to 2 minutes.
3. Copy the heartbeat URL.
4. Deploy a lightweight latency probe that runs every minute:

import time
import requests

TORCHSERVE_URL = "http://localhost:8080/predictions/resnet50"
VIGILMON_HEARTBEAT = "https://vigilmon.online/heartbeat/abc123"
LATENCY_THRESHOLD_MS = 500  # alert if p95 > 500ms

# Sample input for resnet50 (replace with your model's input format)
TEST_INPUT = {"data": "base64encodedimage..."}

def probe():
    start = time.time()
    resp = requests.post(TORCHSERVE_URL, json=TEST_INPUT, timeout=10)
    latency_ms = (time.time() - start) * 1000
    
    if resp.status_code == 200 and latency_ms < LATENCY_THRESHOLD_MS:
        requests.get(VIGILMON_HEARTBEAT, timeout=5)
    else:
        print(f"Latency probe failed: {resp.status_code}, {latency_ms:.0f}ms")

if __name__ == "__main__":
    probe()

Run this with a cron job every 2 minutes. If latency spikes above threshold or the inference call fails, the heartbeat stops and Vigilmon alerts.

Step 6: Worker Thread Utilization Alerting

TorchServe scales inference workers per model. When all workers are busy, requests queue — and when the queue fills, requests are rejected with HTTP 503. Monitor the metrics endpoint for worker saturation:

Add a keyword-based monitor that checks for high worker utilization in the Prometheus metrics:

1. Create a lightweight HTTP endpoint in a monitoring sidecar that reads TorchServe metrics and exposes a health status:

from flask import Flask, jsonify
import requests

app = Flask(__name__)

@app.route('/worker-health')
def worker_health():
    metrics = requests.get('http://localhost:8082/metrics').text
    
    # Parse worker busy ratio from Prometheus metrics
    for line in metrics.splitlines():
        if 'ts_queue_latency_microseconds' in line and not line.startswith('#'):
            # High queue latency indicates worker saturation
            value = float(line.split()[-1])
            if value > 1_000_000:  # > 1 second queue wait
                return jsonify({'status': 'saturated', 'queue_latency_us': value}), 503
    
    return jsonify({'status': 'ok'})

2. Add a Vigilmon HTTP monitor targeting http://torchserve.yourdomain.com:8083/worker-health.
3. Set Expected HTTP status to 200.

Step 7: GPU Utilization Monitoring

For GPU-accelerated TorchServe deployments, GPU utilization is a key capacity signal. TorchServe exposes GPU metrics through its metrics endpoint:

# Check GPU metrics from TorchServe metrics endpoint
curl http://localhost:8082/metrics | grep -i gpu
# GPUMemoryUtilization{model_name="resnet50",level="host"} 0.45
# GPUUtilization{model_name="resnet50",level="host"} 0.82

Set up a heartbeat probe that alerts when GPU memory usage exceeds a safe threshold:

import requests

def check_gpu():
    metrics = requests.get('http://localhost:8082/metrics').text
    for line in metrics.splitlines():
        if line.startswith('GPUMemoryUtilization') and not line.startswith('#'):
            utilization = float(line.split()[-1])
            if utilization > 0.90:  # > 90% GPU memory
                raise Exception(f"GPU memory critical: {utilization:.0%}")
    requests.get('https://vigilmon.online/heartbeat/gpu123', timeout=5)

Step 8: SSL Certificate Expiry Alerts

Production TorchServe deployments typically sit behind an nginx or Envoy reverse proxy that handles TLS. Monitor the certificate expiry on the public domain:

1. Open the HTTP monitor for https://torchserve.yourdomain.com/ (or the inference proxy domain).
2. Enable Monitor SSL certificate.
3. Set Alert when certificate expires in less than 21 days.
4. Click Save.

If TorchServe is directly exposed (e.g., using TorchServe's built-in SSL support), the same applies to the direct port-8080 URL.

Step 9: Configure Alert Channels

1. Go to Alert Channels in Vigilmon and add Slack, email, or a PagerDuty webhook.
2. Set Consecutive failures before alert to 2 on the inference endpoint monitor — brief network blips can cause single-probe failures during model loading.
3. Use Maintenance windows during model deployments:

# Suppress alerts during model registration
curl -X POST https://vigilmon.online/api/maintenance \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -d '{"monitor_id": "abc123", "duration_minutes": 5}'

# Register new model version
curl -X POST "http://localhost:8081/models?url=resnet50-v2.mar&model_name=resnet50&initial_workers=2"

Summary

| Monitor | Target | What It Catches | |---|---|---| | Inference endpoint | http://torchserve.domain:8080/ping | Server crash, inference port down | | Management API | http://torchserve.domain:8081/models | Management API unreachable | | Metrics endpoint | http://torchserve.domain:8082/metrics | Metrics pipeline failure | | Model status | GET /models/{name} + keyword check | Model in FAILED/LOADING state | | Latency heartbeat | Heartbeat URL | p95 latency above threshold | | Worker saturation | Sidecar health endpoint | All workers busy, queue overflow | | GPU utilization | Heartbeat + metrics parse | GPU memory exhaustion | | SSL certificate | HTTPS proxy domain | TLS renewal failure |

TorchServe abstracts a lot of complexity — batching, worker management, model versioning — but that abstraction means failures can be silent. With Vigilmon watching each port and a latency heartbeat tracking end-to-end inference performance, you get the visibility needed to keep production model serving reliable.