How to Monitor CVAT with Vigilmon (Annotation Jobs, Workers, and API Health)

CVAT (Computer Vision Annotation Tool) is the backbone of many ML pipelines — it manages annotation task queues, coordinates worker nodes, stores petabytes of image and video data, and serves a REST/gRPC API consumed by labelers and downstream automation. When any piece of that stack degrades silently, annotation throughput collapses and model training schedules slip.

In this tutorial you'll set up external uptime and latency monitoring for CVAT using Vigilmon — free tier, no credit card required.

Why CVAT needs external monitoring

CVAT's internal health indicators — Celery worker status pages, Redis queue length metrics, Django admin panels — all require you to be logged in and looking at the right dashboard. External monitoring fills a different role: it tells you what the annotation platform looks like from the outside, continuously, without human intervention.

Common CVAT failure modes that internal dashboards miss:

Annotation API down — a Gunicorn worker crash or OOM kill takes down /api/ while the Django admin panel (served by a separate process) still responds
Worker queue stalls — Celery workers are connected to Redis but no longer processing jobs; the queue depth climbs silently
Storage backend unreachable — an S3 bucket permission change or NFS mount failure means CVAT can't read annotation source images, but the API returns 200 for metadata requests
Reverse proxy misconfiguration — an nginx restart after a certificate renewal drops the X-Forwarded-For header CVAT depends on for auth; users get 403 but the container is healthy
Cold-start latency spikes — after a node replacement, CVAT's model loading causes API latency to spike from 80ms to 8s for several minutes

External monitoring from multiple geographic regions catches all of these and alerts you before users file bug reports.

What you'll need

A running CVAT instance (Docker Compose or Kubernetes)
The CVAT REST API accessible over HTTPS (or HTTP for internal monitoring)
A free Vigilmon account

Step 1: Verify CVAT's health endpoint

CVAT exposes a built-in health endpoint at /api/health/. Verify it's reachable:

curl -sf https://cvat.example.com/api/health/ | jq .

You should see something like:

{
  "database": "OK",
  "cache": "OK",
  "storage": "OK"
}

If the response is empty or returns non-200, check that:

The CVAT server container is running: docker compose ps cvat_server
nginx is proxying /api/ correctly: docker compose logs cvat_proxy

If your CVAT deployment doesn't have a /api/health/ route (older versions), you can use /api/server/about as a functional equivalent — it requires no authentication and returns server version info.

Step 2: Add a lightweight annotation queue health probe

CVAT's job queue health depends on Celery workers consuming tasks from Redis. The built-in health check verifies database and cache connectivity but doesn't confirm workers are active. Add a custom probe endpoint to your CVAT deployment:

# cvat/server/health_extended.py
from django.http import JsonResponse
from django.views import View
from celery.app.control import Inspect

class AnnotationQueueHealthView(View):
    def get(self, request):
        i = Inspect()
        active_workers = i.active()
        worker_count = len(active_workers) if active_workers else 0

        status = "ok" if worker_count > 0 else "degraded"
        return JsonResponse({
            "status": status,
            "active_workers": worker_count,
        }, status=200 if status == "ok" else 503)

from .health_extended import AnnotationQueueHealthView

urlpatterns = [
    # ... existing patterns
    path('health/workers/', AnnotationQueueHealthView.as_view()),
]

Now you have two endpoints to monitor:

/api/health/ — database, cache, storage subsystems
/api/health/workers/ — Celery annotation worker availability

Step 3: Create monitors in Vigilmon

Log into Vigilmon and create monitors for each endpoint.

Monitor 1: CVAT API health

Click Add Monitor and configure:

| Field | Value | |-------|-------| | Monitor type | HTTP(S) | | Name | CVAT API Health | | URL | https://cvat.example.com/api/health/ | | Check interval | 1 minute | | Expected status | 200 | | Alert threshold | 2 consecutive failures |

Monitor 2: Annotation worker availability

| Field | Value | |-------|-------| | Monitor type | HTTP(S) | | Name | CVAT Annotation Workers | | URL | https://cvat.example.com/api/health/workers/ | | Check interval | 2 minutes | | Expected status | 200 | | Alert threshold | 2 consecutive failures |

Monitor 3: CVAT storage backend

Storage failures often appear as slow responses rather than outright 500 errors. Use a keyword monitor to catch degraded states:

| Field | Value | |-------|-------| | Monitor type | HTTP(S) – Keyword | | Name | CVAT Storage | | URL | https://cvat.example.com/api/health/ | | Expected keyword | "storage": "OK" | | Check interval | 2 minutes |

Step 4: Monitor API response latency

CVAT annotation API calls are latency-sensitive — labelers notice when task loading slows from 100ms to 2s. Enable response time tracking in Vigilmon:

Open each HTTP monitor you created
Enable Response Time Tracking
Set a Response Time Alert threshold (recommended: 2000ms for the main API)

This creates a latency history chart and triggers alerts when the API slows down, even if it's still returning 200 OK.

Step 5: Monitor the CVAT TCP port directly

If your CVAT is behind a load balancer that performs health checks, it's possible for the load balancer to report healthy while the upstream CVAT port is unreachable from outside. Add a TCP monitor as a belt-and-suspenders check:

| Field | Value | |-------|-------| | Monitor type | TCP | | Name | CVAT TCP Port | | Host | cvat.example.com | | Port | 443 (or 80 for HTTP) | | Check interval | 1 minute |

Step 6: Configure alert routing

Go to Settings → Alerts in Vigilmon and configure where alerts should go when CVAT goes down.

Slack alert

POST https://hooks.slack.com/services/YOUR/SLACK/WEBHOOK
{
  "text": "🔴 CVAT monitor {{monitor.name}} is DOWN\nURL: {{monitor.url}}\nTime: {{alert.time}}\nReason: {{alert.reason}}"
}

PagerDuty integration

For annotation pipelines where downtime has a direct business cost, connect PagerDuty via Vigilmon's webhook:

POST https://events.pagerduty.com/v2/enqueue
Content-Type: application/json
{
  "routing_key": "YOUR_PD_ROUTING_KEY",
  "event_action": "trigger",
  "payload": {
    "summary": "CVAT {{monitor.name}} DOWN",
    "severity": "critical",
    "source": "vigilmon"
  }
}

Step 7: Add a status page for annotation teams

Annotation teams and ML engineers need to know when CVAT is down without having to check dashboards. Create a public status page in Vigilmon:

Go to Status Pages → New
Add all 4 monitors you created
Set the page title to "CVAT Platform Status"
Share the URL with your annotation team

When the annotation API or workers degrade, the status page updates automatically — no manual incident posting required.

What to monitor: CVAT-specific checklist

| Component | Monitor type | URL/endpoint | |-----------|-------------|--------------| | API health (DB + cache + storage) | HTTP keyword | /api/health/ | | Annotation worker availability | HTTP status | /api/health/workers/ | | API response latency | HTTP with timing | /api/server/about | | TCP port reachability | TCP | port 443 | | Annotation task list (auth required) | HTTP | /api/tasks (with token) |

For the authenticated endpoint, Vigilmon supports custom request headers. Set Authorization: Token YOUR_API_TOKEN to monitor endpoints that require a logged-in session.

Common CVAT failure patterns and how monitoring catches them

Celery worker OOM kill: Workers processing large video annotation jobs can exhaust memory. The /api/health/workers/ endpoint goes to 503 within seconds. Vigilmon alerts fire after 2 consecutive failures (2–4 minutes of detection time).

Redis connection lost: The annotation queue uses Redis as its broker. If Redis restarts, workers disconnect. The /api/health/ endpoint reports "cache": "FAIL" — the keyword monitor catches this immediately.

Storage permission error: An S3 IAM policy rotation can revoke CVAT's bucket access. The health check reports "storage": "FAIL" before any user notices they can't load annotation images.

Gunicorn worker timeout: Long annotation export jobs can block all Gunicorn workers, causing API response times to spike. The response time alert fires before the API starts dropping connections entirely.

Summary

| What | Why | |------|-----| | /api/health/ keyword monitor | Catches DB, cache, and storage subsystem failures | | /api/health/workers/ status monitor | Confirms Celery workers are active and processing | | Response time alert (2000ms) | Detects latency spikes before API becomes unusable | | TCP monitor on port 443 | Catches network-layer failures the HTTP monitor can't reach | | Status page for annotation team | Reduces noise from Slack questions when CVAT is degraded |

CVAT outages directly stall annotation throughput and delay model training schedules. External monitoring with Vigilmon gives you a reliable, independent signal that alerts you faster than any internal dashboard — so your team finds out about problems before your annotators do.