tutorial

Monitoring AIM Stack with Vigilmon

AIM Stack is the open source self-hosted experiment tracking platform for ML teams — here's how to monitor its server process, web UI, run tracking API, artifact storage, and query performance with Vigilmon.

AIM Stack is the open source, self-hosted experiment tracking and AI observability platform that lets teams log, compare, and visualize ML training runs, hyperparameters, metrics, and system resources. It is a popular alternative to MLflow and Weights & Biases for teams who want full data ownership and a rich interactive UI without vendor lock-in. Because AIM holds the entire history of your training runs — hyperparameter configs, metric curves, and system resource logs — its availability directly affects your team's ability to compare experiments and make training decisions. Vigilmon lets you watch every layer of the AIM stack from a single dashboard.

What You'll Set Up

  • AIM server process availability monitor (port 43800)
  • Web UI availability check
  • Run tracking API health check
  • Metric logging endpoint health check
  • Artifact storage and disk space monitoring
  • PostgreSQL connectivity check (for multi-user remote server)
  • SSL/TLS certificate expiry monitoring

Prerequisites

  • AIM server running (aim server or aim up) on port 43800
  • A free Vigilmon account

Step 1: Monitor the AIM Server Process

The aim server process is the single process that handles all experiment tracking, metric logging, and web UI access. If it stops, all training runs lose their ability to log metrics and the experiment comparison dashboard becomes unavailable.

  1. Log in to vigilmon.online and click Add Monitor.
  2. Set Type to HTTP / HTTPS.
  3. Enter: http://your-aim-host:43800
  4. Set Check interval to 1 minute.
  5. Set Expected HTTP status to 200.
  6. Click Save.

The root URL serves the React dashboard — a 200 response confirms the AIM server is alive.

Step 2: Monitor the Web UI

The AIM web UI is a React-based experiment comparison dashboard served by the AIM server. Add a dedicated monitor for it to distinguish UI serving failures from API failures:

  1. Click Add Monitor → HTTP / HTTPS.
  2. Enter: http://your-aim-host:43800
  3. Add a response body contains assertion: AIM (or any string from the dashboard's HTML that confirms the UI is being served, not a 502 from a proxy).
  4. Set Check interval to 2 minutes.
  5. Click Save.

Step 3: Monitor the Run Tracking API

The run tracking endpoint (POST /api/v1/runs) is what AIM SDK clients call to create new runs and log parameters at the start of training. If this endpoint fails, training scripts cannot register their runs and all subsequent metric logs for that session are lost.

  1. Click Add Monitor → HTTP / HTTPS.
  2. Enter: http://your-aim-host:43800/api/v1/runs
  3. Set Method to GET.
  4. Set Expected HTTP status to 200.
  5. Set Check interval to 2 minutes.
  6. Click Save.

A GET on the runs list endpoint confirms the API layer is up and routing correctly without creating a test run.

Step 4: Monitor the Metric Logging Endpoint

The metric batch write endpoint (POST /api/v1/runs/:id/metric/get-batch) is called at high throughput during training — typically every few seconds per GPU. Latency or failures here cause metric gaps that make experiment comparisons unreliable.

Check the metric API is responding by probing the runs list and verifying the API path is alive:

  1. Click Add Monitor → HTTP / HTTPS.
  2. Enter: http://your-aim-host:43800/api/v1/info
  3. Set Expected HTTP status to 200.
  4. Set Check interval to 1 minute.
  5. Click Save.

The /api/v1/info endpoint (available in recent AIM versions) returns server metadata and is a reliable liveness probe for the entire API layer.

Step 5: Monitor Artifact Storage and Disk Space

AIM stores metric sequences, artifacts, and logs on disk in the .aim repository directory (default: ~/.aim or a custom path set with --repo). If the disk fills up or write permissions are lost, AIM silently fails to persist new metrics — training continues but experiment data is lost.

Use a Vigilmon cron heartbeat to run a disk space check:

  1. Click Add Monitor → Cron Heartbeat.
  2. Set the expected ping interval to 30 minutes.
  3. Copy the heartbeat URL: https://vigilmon.online/heartbeat/abc123.
  4. On the AIM host, add a cron job that checks free space and pings Vigilmon only when space is available:
#!/bin/bash
# /etc/cron.d/aim-disk-check — runs every 30 minutes
THRESHOLD=85  # alert when disk usage exceeds 85%
USAGE=$(df /path/to/aim/repo | awk 'NR==2 {print $5}' | tr -d '%')
if [ "$USAGE" -lt "$THRESHOLD" ]; then
  curl -s https://vigilmon.online/heartbeat/abc123
fi

When disk usage exceeds the threshold, the heartbeat ping is not sent, and Vigilmon alerts after the expected interval passes.

Step 6: Monitor PostgreSQL (Multi-User Remote Server)

If you are running AIM in remote server mode with a PostgreSQL backend (for multi-user access with user accounts), add a TCP monitor:

  1. Click Add Monitor → TCP Port.
  2. Enter your PostgreSQL host and port: your-pg-host:5432.
  3. Set Check interval to 1 minute.
  4. Click Save.

A PostgreSQL outage in this configuration means new user sessions cannot be authenticated and run metadata cannot be written.

Step 7: Monitor Run Comparison Query Performance

AIM's query engine filters and aggregates runs for the dashboard comparison views. Under load, slow queries can make the dashboard unusable even when the server is technically up.

Add an HTTP monitor with a response time threshold:

  1. Open the web UI monitor from Step 2.
  2. Under Performance, set Alert if response time exceeds 5000 ms.
  3. Click Save.

A sustained response time above 5 seconds on the main dashboard usually indicates a slow query under a large experiment set or insufficient server memory.

Step 8: SSL/TLS Certificate Expiry Monitoring

If AIM is served over HTTPS via a reverse proxy (nginx, Caddy, Traefik), add a certificate expiry check:

  1. Click Add Monitor → HTTP / HTTPS.
  2. Enter the HTTPS URL of your AIM deployment.
  3. Enable Monitor SSL certificate.
  4. Set Alert when certificate expires in less than 21 days.
  5. Click Save.

Step 9: Configure Alert Channels

  1. Go to Alert Channels in Vigilmon and add Slack, email, or a webhook.
  2. Set Consecutive failures before alert to 2 on the web UI monitor — brief restarts during AIM upgrades should not page your team.
  3. Set alert threshold to 1 on the run tracking API — a single failure during an active training run means metric data is being lost.

For teams running overnight training jobs, configure alert escalation so disk space and run API failures page the on-call engineer even outside business hours.

Summary

| Monitor | Target | What It Catches | |---|---|---| | Server process | :43800 | AIM server crash or restart | | Web UI | :43800 (body check) | UI serving failure, proxy error | | Run tracking API | /api/v1/runs | Run creation endpoint down | | API liveness | /api/v1/info | Full API layer failure | | Disk space | Cron heartbeat | Artifact storage full | | PostgreSQL TCP | :5432 | Database unreachable (remote mode) | | Response time | Dashboard URL | Slow query degrading UI | | SSL certificate | HTTPS endpoint | Certificate near expiry |

AIM Stack gives ML teams full ownership of their experiment history — every metric curve, hyperparameter sweep, and checkpoint reference lives on your infrastructure. An unmonitored AIM server means training runs can silently lose their metrics, overnight GPU jobs produce nothing auditable, and experiment comparisons become unreliable. With Vigilmon watching the server, the API, and the disk, you keep your training runs trustworthy.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles