tutorial

Monitoring Seldon Core with Vigilmon: Inference Health, Replica Readiness, Latency Thresholds & SSL Certificate Alerts

How to monitor Seldon Core ML model serving with Vigilmon — inference health checks, replica readiness, latency SLO alerting, drift detector monitoring, and SSL certificate alerts.

How to Monitor Seldon Core with Vigilmon (Inference Health, Latency & Replica Alerts)

Seldon Core is the most widely deployed open-source platform for serving machine learning models at scale on Kubernetes. It handles inference routing, canary deployments, A/B tests, and drift detection — but with that power comes complexity. When a model deployment is unhealthy, the failure modes can be subtle: one replica accepting traffic while others silently return errors, drift detection models running out of sync, or a canary receiving 100% of traffic after a misconfigured rollout.

In this tutorial you'll set up production monitoring for your Seldon Core deployments using Vigilmon, covering inference health, latency thresholds, and replica-level alerting.

Why Seldon Core deployments need external monitoring

Seldon Core runs inside Kubernetes, which means Kubernetes-internal health signals are available — but they don't tell the full story:

  • Inference server error rate — a model container returns 200 to Kubernetes health probes but returns prediction errors to inference requests; the Kubernetes scheduler thinks everything is healthy
  • Replica skew — one of three replicas is loading a stale model version after a failed rollout; 33% of requests get wrong predictions, but pod status shows Running for all replicas
  • Canary routing stuck — a Seldon SeldonDeployment canaryTrafficPercent was set to 10% but a config drift caused it to serve 100%; no Kubernetes alarm fires
  • Pre/post processors failing — your transformer component (feature engineering or post-processing) silently returns errors while the predictor itself is healthy
  • Explainer endpoint degraded — the explainability endpoint for regulatory compliance returns 503, which internal probes don't check
  • External load balancer issue — Seldon's Istio ingress or the Ambassador gateway routing to your model is broken; pods are healthy but client requests fail

External monitoring from Vigilmon catches all of these because it probes the inference endpoint from outside the cluster, exactly as client applications do.

What you'll need

  • A Kubernetes cluster with Seldon Core installed
  • At least one SeldonDeployment running and serving predictions
  • A free Vigilmon account (sign up takes 30 seconds)

Step 1: Understand the Seldon Core endpoint structure

Seldon Core generates inference endpoints in a predictable pattern. For a deployment named my-classifier in namespace production:

# REST prediction endpoint
POST http://<ingress>/seldon/production/my-classifier/api/v1.0/predictions

# Health endpoint
GET http://<ingress>/seldon/production/my-classifier/api/v1.0/health/ping

# Ready endpoint  
GET http://<ingress>/seldon/production/my-classifier/api/v1.0/health/ready

# Status endpoint (deployment-level health)
GET http://<ingress>/seldon/production/my-classifier/api/v1.0/status

Get your ingress address:

kubectl get svc -n istio-system istio-ingressgateway
# NAME                   TYPE           CLUSTER-IP      EXTERNAL-IP     PORT(S)
# istio-ingressgateway   LoadBalancer   10.96.0.100     203.0.113.50    80:31380/TCP,443:31390/TCP

# Or for Ambassador:
kubectl get svc -n default ambassador

Test the health ping:

curl http://203.0.113.50/seldon/production/my-classifier/api/v1.0/health/ping
# {"status":"ok"}

Test a prediction:

curl -X POST http://203.0.113.50/seldon/production/my-classifier/api/v1.0/predictions \
  -H "Content-Type: application/json" \
  -d '{"data": {"ndarray": [[1.0, 2.0, 3.0, 4.0]]}}'
# {"data":{"ndarray":[[0.97,0.03]]},"meta":{}}

Step 2: Add a custom health endpoint with model metadata

Seldon's built-in /health/ping is minimal. For richer monitoring, add a health endpoint to your model server that returns model version and replica information:

# MyModel.py (Seldon microservice format)
import os
import joblib
import numpy as np

class MyModel:
    def __init__(self):
        self.model = joblib.load("/app/model.pkl")
        self.model_version = os.getenv("MODEL_VERSION", "unknown")
        self.ready = True

    def predict(self, X, features_names=None):
        return self.model.predict_proba(X)

    def health_status(self):
        """Custom health check — exposed at /health/status by Seldon."""
        return {
            "status": "ok",
            "model_version": self.model_version,
            "model_loaded": self.ready
        }

Seldon automatically exposes the health_status method at a custom path. Alternatively, use a Kubernetes sidecar to expose a richer health endpoint:

# seldon-deployment.yaml
apiVersion: machinelearning.seldon.io/v1
kind: SeldonDeployment
metadata:
  name: my-classifier
  namespace: production
spec:
  predictors:
  - name: default
    graph:
      name: classifier
      type: MODEL
      implementation: SKLEARN_SERVER
      modelUri: gs://your-bucket/models/classifier/v3
      envSecretRefName: seldon-gcs-secret
    componentSpecs:
    - spec:
        containers:
        - name: classifier
          env:
          - name: MODEL_VERSION
            value: "v3"
          readinessProbe:
            httpGet:
              path: /api/v1.0/health/ready
              port: 9000
            initialDelaySeconds: 30
            periodSeconds: 10
          livenessProbe:
            httpGet:
              path: /api/v1.0/health/ping
              port: 9000
            initialDelaySeconds: 60
            periodSeconds: 30
    replicas: 3
    traffic: 100

Apply it:

kubectl apply -f seldon-deployment.yaml

# Watch it come up
kubectl rollout status deployment/my-classifier-default-0-classifier -n production

Step 3: Set up HTTP monitoring in Vigilmon

With your deployment running, add Vigilmon monitors:

Monitor 1: Inference health ping

  1. Log in to vigilmon.online → Monitors → New Monitor
  2. Type: HTTP / HTTPS
  3. URL: http://203.0.113.50/seldon/production/my-classifier/api/v1.0/health/ping
  4. Check interval: 1 minute
  5. Expected response: status code 200, body contains "status":"ok"
  6. Save

Monitor 2: Readiness check (all replicas serving)

  1. Monitors → New Monitor → HTTP / HTTPS
  2. URL: http://203.0.113.50/seldon/production/my-classifier/api/v1.0/health/ready
  3. Check interval: 2 minutes
  4. Expected response: status code 200
  5. Save

The readiness endpoint returns 200 only when all replicas are ready, so a partial replica failure causes this check to fail — giving you replica-level alerting without querying the Kubernetes API.

Step 4: Add inference smoke-test monitoring

Health pings tell you the server is alive. To verify that the actual prediction path works, set up a lightweight inference probe.

Vigilmon supports HTTP monitors with POST method and custom bodies. Create a monitor that sends a real inference request:

  1. Monitors → New Monitor → HTTP / HTTPS
  2. URL: http://203.0.113.50/seldon/production/my-classifier/api/v1.0/predictions
  3. Method: POST
  4. Request headers: Content-Type: application/json
  5. Request body: {"data": {"ndarray": [[1.0, 2.0, 3.0, 4.0]]}}
  6. Check interval: 5 minutes
  7. Expected response: status code 200, body contains "ndarray"
  8. Save

This confirms the entire inference path — model loading, preprocessing, prediction, postprocessing — is working correctly, not just the HTTP layer.

Step 5: Monitor inference latency

Model inference latency is a critical SLO metric. A spike usually indicates:

  • Model reload after an eviction
  • Increased input complexity or batch size
  • GPU contention from other workloads
  • Network slowdown to a remote model registry

Set response time thresholds on your inference smoke-test monitor:

  1. Open the inference smoke-test monitor → Alerts → Response Time
  2. Set: Alert if response time exceeds 500ms (adjust to your p95 baseline)
  3. Save

For latency-sensitive deployments, consider setting multiple thresholds:

  • Warning at 300ms — Slack notification
  • Critical at 1000ms — PagerDuty page

Step 6: Monitor drift detection endpoints

If you've deployed a Seldon drift detector (e.g., using Alibi Detect), monitor it separately:

# Typical drift detector endpoint pattern
curl -X POST http://203.0.113.50/seldon/production/my-classifier-drift/api/v1.0/predictions \
  -H "Content-Type: application/json" \
  -d '{"data": {"ndarray": [[1.0, 2.0, 3.0, 4.0]]}}'

Add a monitor for the drift detector:

  1. Monitors → New Monitor → HTTP / HTTPS
  2. URL: your drift detector endpoint
  3. Check interval: 5 minutes
  4. Expected response: 200
  5. Save

Drift detectors are easy to overlook — they don't directly serve predictions, but a dead drift detector means you lose your early warning system for model degradation in production.

Step 7: Configure alerts

Slack — immediate notification:

  1. Settings → Notifications → Add Channel → Slack
  2. Paste your webhook URL
  3. Assign to health and readiness monitors

Email escalation:

  1. Settings → Notifications → Add Channel → Email
  2. Add your ML engineering team's email
  3. Set delay: 5 minutes (avoid noise from transient probe failures during rollouts)

Tip for canary deployments: During a canary rollout, Seldon briefly has mixed traffic between model versions. Add a temporary 10-minute alert delay on the inference smoke-test monitor during planned rollouts to suppress false positives from in-flight version transitions.

Step 8: Status page for model consumers

Data science teams, product managers, and downstream services all need visibility into model health:

  1. Status Pages → New Status Page
  2. Add all monitors: health ping, readiness, inference smoke test, drift detector
  3. Name it "ML Inference Platform"
  4. Share the URL in your ML platform documentation

Use Maintenance Windows during model retraining and deployment cycles (typically 2–5 minutes of downtime during a blue/green switch) to suppress alerts and show a maintenance notice on the status page.

Monitoring coverage matrix

| Failure type | Vigilmon monitor | How detected | |---|---|---| | Seldon process crash | HTTP /health/ping | Connection refused or non-200 | | Replica not ready | HTTP /health/ready | Returns 503 when replicas < desired | | Prediction path broken | POST inference probe | Non-200 or missing response body | | Inference latency spike | Response time threshold | Exceeds configured ms threshold | | Ingress/LB routing failure | Any external HTTP monitor | Connection error or wrong status | | Drift detector down | HTTP drift endpoint | Non-200 response |

Conclusion

Seldon Core manages complex model serving workflows — replicas, canaries, shadow deployments, drift detectors — and a failure in any layer can silently degrade prediction quality or throughput. With Vigilmon you get:

  • Multi-layer health monitoring that distinguishes HTTP server liveness from replica readiness from inference correctness
  • Response time thresholds tuned to your model's latency SLO
  • Drift detector monitoring so your data quality watchdog is itself watched
  • Status pages for ML platform visibility across your organization

Create your free Vigilmon account and have your first Seldon deployment monitored in under 10 minutes.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles