tutorial

Monitoring Argilla with Vigilmon

Argilla is the open source data annotation platform for LLMs and NLP — here's how to monitor its FastAPI server, Elasticsearch backend, annotation APIs, and disk health with Vigilmon.

Argilla is the open source platform teams use to curate training datasets, collect human feedback on LLM outputs (RLHF/RLAIF), and run annotation campaigns for fine-tuning and evaluation. It integrates with Hugging Face, LangChain, and popular ML frameworks — and because it holds your ground-truth annotation data, its availability is directly tied to the quality of every model you ship. Vigilmon lets you monitor Argilla's FastAPI server, Elasticsearch or OpenSearch backend, annotation and feedback APIs, and disk space from a single dashboard.

What You'll Set Up

  • Web UI and API server availability monitor (port 6900)
  • Elasticsearch / OpenSearch connectivity check
  • Annotation record ingestion API health check
  • Feedback dataset listing API health check
  • User authentication service health check
  • Disk space alert for Elasticsearch nodes
  • SSL/TLS certificate expiry monitoring

Prerequisites

  • Argilla server running and accessible (default port 6900)
  • Elasticsearch or OpenSearch running as the data store
  • A free Vigilmon account

Step 1: Monitor the Web UI and API Server

Argilla's FastAPI server at port 6900 is the main application interface for annotation campaigns and dataset management. If it goes down, SDK clients cannot submit records and annotators cannot access the UI.

  1. Log in to vigilmon.online and click Add Monitor.
  2. Set Type to HTTP / HTTPS.
  3. Enter the Argilla status endpoint: http://your-argilla-host:6900/api/v1/status
  4. Set Check interval to 1 minute.
  5. Set Expected HTTP status to 200.
  6. Click Save.

The /api/v1/status endpoint returns a JSON payload with the server version and status — it is lightweight and does not require authentication.

Step 2: Monitor Elasticsearch / OpenSearch Connectivity

Elasticsearch (or OpenSearch) is Argilla's primary data store. Every dataset, annotation record, and feedback response is stored in Elasticsearch indices. Without it, Argilla cannot read or write any data.

Add a TCP monitor to confirm Elasticsearch is accepting connections:

  1. Click Add Monitor → TCP Port.
  2. Enter your Elasticsearch host and port: your-es-host:9200.
  3. Set Check interval to 1 minute.
  4. Click Save.

For a deeper check, add an HTTP monitor against the Elasticsearch cluster health endpoint:

  1. Click Add Monitor → HTTP / HTTPS.
  2. Enter: http://your-es-host:9200/_cluster/health.
  3. Set Expected HTTP status to 200.
  4. Add a response body contains assertion: "status":"green" (or "yellow" if you run a single-node cluster).
  5. Click Save.

A red cluster status means shards are unassigned and Argilla data may be partially unavailable.

Step 3: Monitor the Annotation Record Ingestion API

The record ingestion endpoint (POST /api/v1/datasets/:id/records) is used by SDK clients to submit LLM outputs for annotation. If this endpoint is degraded, production annotation pipelines silently stop populating datasets.

Because POST endpoints cannot be probed without a request body, use a GET probe on a closely related read endpoint instead:

  1. Click Add Monitor → HTTP / HTTPS.
  2. Enter: http://your-argilla-host:6900/api/v1/datasets
  3. Set Expected HTTP status to 200.
  4. Under Request Headers, add Authorization: Bearer YOUR_API_KEY.
  5. Set Check interval to 2 minutes.
  6. Click Save.

A healthy datasets listing confirms the annotation API layer is reachable and authenticated access is working.

Step 4: Monitor the Feedback Dataset API

The feedback dataset listing endpoint (GET /api/v1/me/datasets) verifies that authenticated users can see their datasets. This endpoint exercises the user context layer on top of the core API.

  1. Click Add Monitor → HTTP / HTTPS.
  2. Enter: http://your-argilla-host:6900/api/v1/me/datasets
  3. Under Request Headers, add Authorization: Bearer YOUR_API_KEY.
  4. Set Expected HTTP status to 200.
  5. Set Check interval to 5 minutes.
  6. Click Save.

Step 5: Monitor the User Authentication Service

The login endpoint validates that the JWT token issuance path is healthy. An authentication failure blocks every SDK client and every human annotator.

  1. Click Add Monitor → HTTP / HTTPS.
  2. Enter: http://your-argilla-host:6900/api/v1/token
  3. Set Method to GET (this endpoint returns token info for an authenticated request).
  4. Under Request Headers, add Authorization: Bearer YOUR_API_KEY.
  5. Set Expected HTTP status to 200.
  6. Set Check interval to 5 minutes.
  7. Click Save.

Step 6: Monitor Disk Space on Elasticsearch Nodes

Annotation datasets can be large. Elasticsearch has disk watermark thresholds that trigger index read-only mode when disk fills up — at that point Argilla cannot write new records.

Check disk usage via the Elasticsearch nodes stats API:

  1. Click Add Monitor → HTTP / HTTPS.
  2. Enter: http://your-es-host:9200/_nodes/stats/fs
  3. Set Expected HTTP status to 200.
  4. Set Check interval to 5 minutes.
  5. Click Save.

For proactive alerting, configure a Vigilmon webhook to POST to a script that parses nodes.*.fs.total.available_in_bytes and alerts when available space drops below 15%. Alternatively, add an OS-level disk space alert through your server monitoring stack and forward it to Vigilmon's alert channel.

Step 7: SSL/TLS Certificate Expiry Monitoring

If Argilla is served over HTTPS (recommended for any production deployment), add a certificate expiry check:

  1. Open the HTTP monitor you created in Step 1.
  2. Update the URL to use https://your-argilla-host/api/v1/status.
  3. Enable Monitor SSL certificate.
  4. Set Alert when certificate expires in less than 21 days.
  5. Click Save.

A 21-day window gives you enough time to renew before annotators and SDK clients start seeing TLS errors.

Step 8: Configure Alert Channels

  1. Go to Alert Channels in Vigilmon and add Slack, email, or a webhook.
  2. Set Consecutive failures before alert to 2 on the FastAPI server monitor — brief container restarts should not page your team.
  3. Set alert threshold to 1 for Elasticsearch — any connectivity loss immediately degrades Argilla.

For Celery workers (if you've enabled background task processing in Argilla), add a Cron Heartbeat monitor:

  1. Click Add Monitor → Cron Heartbeat.
  2. Set the expected ping interval to match your task cadence.
  3. In your Celery beat configuration, send a GET request to the heartbeat URL after each successful task cycle.

Summary

| Monitor | Target | What It Catches | |---|---|---| | FastAPI server | :6900/api/v1/status | Argilla app crash or restart | | Elasticsearch TCP | :9200 | Data store unreachable | | ES cluster health | /_cluster/health | Red/degraded cluster state | | Datasets API | /api/v1/datasets | Annotation API layer failure | | Feedback datasets | /api/v1/me/datasets | User context / auth layer failure | | Auth endpoint | /api/v1/token | Authentication service degradation | | Disk space | /_nodes/stats/fs | ES disk watermark approaching | | SSL certificate | HTTPS endpoint | Certificate near expiry |

Argilla is the annotation layer between raw LLM outputs and high-quality fine-tuning data. When it goes down undetected, annotation campaigns stall, RLHF pipelines miss their data windows, and model training gets delayed. With Vigilmon watching every critical path — from the FastAPI server to Elasticsearch disk watermarks — you catch problems before they affect your annotation team or your training schedule.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles