tutorial

Monitoring Replicate with Vigilmon: API Availability, Webhook Delivery & Prediction Pipeline Health

How to monitor Replicate ML model hosting with Vigilmon — API availability, prediction webhook delivery, deployment endpoint uptime, and SSL certificate monitoring.

Replicate is the hosted platform for running and fine-tuning open-source ML models via a simple API — providing versioned model deployments, prediction caching, webhook delivery, and a marketplace of community models covering Llama, SDXL, Whisper, and hundreds more. When the Replicate API is degraded, prediction requests queue indefinitely or return errors. When webhook delivery fails, your application never learns that a long-running prediction completed. When a deployment's custom domain goes down, your users can't reach your model's dedicated endpoint. Vigilmon gives you external visibility into Replicate's availability and your integration's health before failed predictions become customer complaints.

What You'll Build

  • An HTTP monitor on the Replicate API to detect platform-level degradation
  • A webhook receiver monitor to track whether Replicate's outbound webhooks reach your infrastructure
  • An HTTP monitor on a Replicate Deployment's dedicated endpoint
  • An SSL certificate monitor for your custom deployment domain
  • An alerting setup that distinguishes API failures from webhook delivery issues

Prerequisites

  • A Replicate account with at least one model or deployment in active use
  • A Replicate API token (from replicate.com/account)
  • Optional: a Replicate Deployment with a dedicated endpoint URL
  • A free account at vigilmon.online

Step 1: Understand Replicate's Observable Surfaces

Replicate is a fully managed platform. The infrastructure is Replicate's responsibility; what you observe externally:

| Surface | Observable via | |---|---| | Replicate API (api.replicate.com) | HTTP availability monitoring | | Prediction status endpoint | Authenticated HTTP polling | | Deployment dedicated endpoint | HTTP uptime monitoring | | Outbound webhooks (prediction completion) | Webhook receiver + Vigilmon HTTP monitor | | Custom deployment domain SSL | SSL certificate monitoring |

External monitoring focuses on the API availability, webhook delivery, and deployment endpoints.

Step 2: Monitor the Replicate API

The Replicate API is the entry point for all prediction requests. Monitoring its availability gives early warning of platform-level degradation:

curl -H "Authorization: Bearer $REPLICATE_API_TOKEN" \
  https://api.replicate.com/v1/models/stability-ai/sdxl
# Returns model metadata JSON if the API is healthy
  1. Log in to Vigilmon → Add Monitor → HTTP.
  2. URL: https://api.replicate.com/v1/models/stability-ai/sdxl.
  3. Method: GET.
  4. Headers: Authorization: Bearer <your-replicate-api-token>.
  5. Check interval: 2 minutes.
  6. Response timeout: 15 seconds.
  7. Expected status: 200.
  8. Keyword: "url" (present in all model metadata responses).
  9. Label: Replicate API availability.
  10. Click Save.

Choose a lightweight, frequently-used model for this check (like a small open-source model metadata endpoint). Model metadata requests don't consume GPU credits.

This monitor catches:

  • Replicate API server errors (5xx responses)
  • Authentication failures (if your token expires or is revoked)
  • Platform-level outages affecting all predictions
  • Network-level issues between Vigilmon's probe nodes and Replicate's API

Alert sensitivity: Set to trigger after 2 consecutive failures to avoid noise from transient API hiccups.

Step 3: Monitor a Replicate Deployment Endpoint

Replicate Deployments provide dedicated, always-warm endpoints for specific model versions — eliminating cold starts for production-critical models. If you have a deployment, its endpoint URL is directly monitorable:

# Replicate Deployment endpoints follow this pattern:
# https://api.replicate.com/v1/deployments/{owner}/{name}/predictions

curl -X POST \
  -H "Authorization: Bearer $REPLICATE_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"input": {"prompt": "health check"}}' \
  https://api.replicate.com/v1/deployments/your-org/your-deployment/predictions

For lightweight availability monitoring without burning prediction credits, monitor the deployment's GET endpoint instead:

curl -H "Authorization: Bearer $REPLICATE_API_TOKEN" \
  https://api.replicate.com/v1/deployments/your-org/your-deployment
# Returns deployment configuration and status
  1. Add Monitor → HTTP.
  2. URL: https://api.replicate.com/v1/deployments/your-org/your-deployment.
  3. Method: GET.
  4. Headers: Authorization: Bearer <your-replicate-api-token>.
  5. Check interval: 2 minutes.
  6. Response timeout: 15 seconds.
  7. Expected status: 200.
  8. Keyword: "currentRelease" (present in deployment detail responses).
  9. Label: Replicate deployment: your-deployment.
  10. Click Save.

Step 4: Monitor Webhook Delivery with a Heartbeat

Replicate supports webhooks for prediction completion — when a long-running prediction finishes, Replicate POSTs the result to your webhook URL. If your server is unreachable, the prediction result is lost. Monitor your webhook endpoint's availability:

# Your webhook receiver should be reachable at your server
curl -X POST https://app.example.com/webhooks/replicate \
  -H "Content-Type: application/json" \
  -d '{"id": "health-check"}'
  1. Add Monitor → HTTP.
  2. URL: https://app.example.com/webhooks/replicate.
  3. Method: POST.
  4. Body: {"id": "health-check"}.
  5. Check interval: 60 seconds.
  6. Response timeout: 10 seconds.
  7. Expected status: 200.
  8. Label: Replicate webhook receiver.
  9. Click Save.

If your webhook endpoint requires a valid Replicate signature (replicate-webhook-secret), add a health check route that accepts unauthenticated pings at a separate path (e.g., /webhooks/replicate/health) to avoid false positives from signature validation failures.

Step 5: Monitor Prediction Pipeline Health with Heartbeats

For applications that run Replicate predictions on a schedule (batch captioning, nightly model fine-tuning, daily image generation), use a heartbeat pattern to detect silent pipeline failures:

import replicate
import requests

HEARTBEAT_URL = "https://vigilmon.online/api/heartbeat/YOUR_HEARTBEAT_TOKEN"

def run_daily_pipeline():
    output = replicate.run(
        "stability-ai/sdxl:version-hash",
        input={"prompt": "daily batch item"}
    )
    # ... process output
    
    # Signal success to Vigilmon
    requests.get(HEARTBEAT_URL, timeout=5)

# Run this function in your scheduled job (cron, Airflow, Prefect, etc.)

In Vigilmon:

  1. Add Monitor → Heartbeat.
  2. Expected interval: 24 hours (or your pipeline's schedule).
  3. Grace period: 30 minutes.
  4. Label: Replicate daily pipeline.
  5. Copy the heartbeat URL into your pipeline code.
  6. Click Save.

Vigilmon alerts you if the heartbeat URL hasn't been called within the expected interval, meaning your pipeline silently stopped running.

Step 6: Monitor SSL Certificates

If you're routing Replicate API calls through your own proxy or reverse proxy with a custom domain, monitor that domain's SSL certificate:

openssl s_client -connect api.example.com:443 2>/dev/null | openssl x509 -noout -dates
  1. Add Monitor → SSL Certificate.
  2. Domain: your proxy domain (e.g., api.example.com).
  3. Alert when expiry is within: 30 days.
  4. Alert again: 14 days, 7 days, 3 days, 1 day.
  5. Click Save.

Replicate's own api.replicate.com certificate is managed by Replicate. You only need to monitor SSL for domains you control.

Step 7: Configure Alerting

In Vigilmon under Settings → Notifications, configure your alert channels:

| Monitor | Trigger | Action | |---|---|---| | Replicate API | Non-200 or keyword missing | Check Replicate status page; pause prediction-heavy features; notify users of delays | | Deployment endpoint | Non-200 or keyword missing | Deployment may be paused or misconfigured; check Replicate dashboard; verify deployment is active | | Webhook receiver | Non-200 or server unreachable | Fix webhook server; check for deploy failures; Replicate will retry but may exhaust retries | | Pipeline heartbeat | Silence beyond interval | Check scheduler; verify prediction didn't fail; check Replicate rate limits | | SSL certificate | < 30 days to expiry | Renew proxy or custom domain certificate |

Alert after: 2 consecutive failures for API and deployment monitors. 1 failure for the webhook receiver monitor.

Common Replicate Failure Modes and What Vigilmon Catches

| Scenario | Vigilmon monitor | |---|---| | Replicate platform outage | API monitor fires; check status.replicate.com | | API token revoked or expired | API monitor fires (401 response) | | Deployment paused (inactive for too long) | Deployment monitor catches changed status | | Webhook server unreachable | Webhook receiver monitor fires; predictions complete but results are lost | | Webhook signature validation rejecting Replicate | Webhook receiver monitor may pass but actual webhooks fail — use a separate health path | | Scheduled pipeline silently stopped | Heartbeat monitor fires after interval + grace | | Proxy SSL certificate expires | SSL monitor alerts at 30-day threshold | | Rate limit hit on free tier | API returns 429; API monitor fires if 429 not in expected status | | Cold start timeout on community model | No always-warm guarantee; use Deployments for latency-sensitive paths | | Network issue between your app and Replicate | Webhook receiver fires; API monitor from Vigilmon may pass (Vigilmon → Replicate is fine) |

Monitoring Model Quality

Vigilmon monitors Replicate's API availability and your webhook infrastructure — it doesn't monitor the quality of predictions or model accuracy drift. For ML quality monitoring:

  • Output validation: For each prediction, validate output format and basic quality signals (non-empty output, expected dimensions for images, non-empty transcription for audio).
  • Latency tracking: Log prediction latency per model version. Spikes may indicate model loading issues or cold starts on community models.
  • Error rate tracking: Track the ratio of failed predictions to successful ones per model version. A sudden increase in failure rate may indicate a model version compatibility issue.
  • Fine-tuned model regression: When deploying a new fine-tune, run a fixed validation set through both the old and new version and compare outputs before switching traffic.

Vigilmon catches infrastructure-level failures (API down, webhook delivery broken). Model quality monitoring requires application-level instrumentation.

Replicate makes it easy to integrate open-source ML models without managing GPU infrastructure — but your application's reliability depends on both Replicate's platform being healthy and your webhook receivers being reachable. Vigilmon gives you external visibility into both: API availability, deployment endpoint status, webhook receiver health, and pipeline execution via heartbeats, so you know the moment Replicate becomes unavailable or your prediction pipeline silently stops running.

Start monitoring Replicate in under 5 minutes — register free at vigilmon.online.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles