Monitoring Guild.ai with Vigilmon

Guild.ai lets data scientists track ML experiments, compare runs, and automate hyperparameter search without touching their training code. It captures metrics, source snapshots, hyperparameters, and system info automatically — but when the underlying infrastructure fails, runs are silently lost and experiment history becomes inconsistent. Vigilmon gives you uptime monitoring, heartbeat checks, and SSL certificate alerts so you know about Guild infrastructure failures before they corrupt your experiment data.

What You'll Set Up

• HTTP uptime monitor for the Guild remote server endpoint
• Artifact storage health check (read/write access to .guild directory)
• Run database integrity heartbeat (SQLite-based run index)
• Metric log file accessibility check
• Experiment comparison API health monitor
• Hyperparameter search scheduler heartbeat
• SSL certificate expiry alerts for remote Guild endpoints

Prerequisites

• Guild.ai installed (pip install guildai) with at least one active project
• Guild remote server configured (if using remote runs) accessible at a domain or IP
• A free Vigilmon account

Step 1: Monitor the Guild Remote Server

If your team uses Guild's remote run feature to submit experiments to a shared server, that server is your most critical endpoint. When it's unreachable, no new runs can be submitted remotely.

1. Log in to vigilmon.online and click Add Monitor.
2. Set Type to HTTP / HTTPS.
3. Enter your Guild remote server URL: https://guild.yourdomain.com/ (or the HTTPS endpoint configured in your .guild/remotes.yml).
4. Set Check interval to 1 minute.
5. Set Expected HTTP status to 200.
6. Click Save.

If your Guild remote server exposes a health endpoint, use that instead of the root URL:

https://guild.yourdomain.com/healthz

For SSH-based remotes (where Guild connects via SSH rather than HTTPS), monitor the SSH port instead:

1. Click Add Monitor → TCP Port.
2. Enter the remote host address.
3. Set Port to 22.
4. Set Check interval to 2 minutes.
5. Click Save.

Step 2: Monitor Artifact Storage Health

Guild stores run artifacts, source snapshots, flags, and metrics in the .guild/runs directory (locally) or a configured remote storage path. Storage read/write failures silently discard run data mid-experiment.

Add a heartbeat from a lightweight cron job that verifies write access to the Guild storage directory:

1. Click Add Monitor → Cron Heartbeat.
2. Set the expected ping interval to 5 minutes.
3. Copy the heartbeat URL (e.g., https://vigilmon.online/heartbeat/abc123).
4. Create a cron job that checks storage and pings Vigilmon:

#!/bin/bash
# /usr/local/bin/check-guild-storage.sh

GUILD_DIR="${HOME}/.guild/runs"
TEST_FILE="${GUILD_DIR}/.healthcheck"

# Verify write access
if echo "ok" > "${TEST_FILE}" && [ "$(cat ${TEST_FILE})" = "ok" ]; then
    rm -f "${TEST_FILE}"
    curl -s "https://vigilmon.online/heartbeat/abc123" > /dev/null
fi

Register it in cron:

# Check every 5 minutes
*/5 * * * * /usr/local/bin/check-guild-storage.sh

If the Guild storage directory becomes read-only (disk full, permission change, NFS mount failure), the heartbeat stops and Vigilmon alerts after 5 minutes.

Step 3: Heartbeat for Run Database Integrity

Guild uses a SQLite-based run index stored in .guild/runs/. Each run directory contains a guild.db file with run metadata. Index corruption or permission loss means Guild can't list or compare runs.

1. Click Add Monitor → Cron Heartbeat.
2. Set the expected ping interval to 10 minutes.
3. Copy the heartbeat URL.
4. Create a script that validates the run database is queryable:

#!/usr/bin/env python3
# check-guild-db.py

import subprocess
import requests

try:
    # Use Guild's own CLI to query the run list — if the DB is corrupted,
    # this command will fail with a non-zero exit code
    result = subprocess.run(
        ['guild', 'runs', 'list', '--limit', '1'],
        capture_output=True,
        timeout=10
    )
    if result.returncode == 0:
        requests.get('https://vigilmon.online/heartbeat/abc123', timeout=5)
except Exception:
    pass  # Heartbeat not sent — Vigilmon will alert

Run this on a 10-minute cron schedule. A corrupted or inaccessible run index will cause guild runs list to fail, stopping the heartbeat.

Step 4: Monitor Metric Log File Accessibility

Guild captures TensorBoard-compatible event files and CSV metric logs in each run's output directory. If the log directory becomes inaccessible, metric capture silently fails mid-training.

Add an HTTP monitor wrapping a lightweight file-system health check. If you run a small Flask or FastAPI app alongside your Guild server, add a /metrics-health endpoint:

from flask import Flask, jsonify
import os

app = Flask(__name__)

@app.route('/metrics-health')
def metrics_health():
    log_dir = os.path.expanduser('~/.guild/runs')
    if not os.path.isdir(log_dir) or not os.access(log_dir, os.R_OK | os.W_OK):
        return jsonify({'status': 'error', 'detail': 'log directory not accessible'}), 503
    return jsonify({'status': 'ok'})

Then monitor it:

1. Click Add Monitor → HTTP / HTTPS.
2. Enter http://localhost:8080/metrics-health (or your accessible health server URL).
3. Set Expected HTTP status to 200.
4. Set Check interval to 5 minutes.

Step 5: Monitor the Experiment Comparison API

The guild compare command queries the run database to compute comparison tables and hyperparameter relationships. Slow or failing queries indicate index degradation.

Wrap the comparison query in a timed health check script:

#!/usr/bin/env python3
# check-guild-compare.py

import subprocess
import time
import requests

start = time.time()
try:
    result = subprocess.run(
        ['guild', 'compare', '--csv', '-'],
        capture_output=True,
        timeout=15
    )
    elapsed = time.time() - start
    # Alert via heartbeat only if query completes quickly (under 10s)
    if result.returncode == 0 and elapsed < 10:
        requests.get('https://vigilmon.online/heartbeat/abc123', timeout=5)
except Exception:
    pass

Schedule this every 15 minutes. If comparison queries start timing out, the heartbeat stops before your team notices degraded experiment comparison performance.

Step 6: Heartbeat for Hyperparameter Search Scheduler

When using Guild's Bayesian or grid search (guild run --optimize), Guild spawns a background scheduler process to manage trial submission. If that process exits early, the search silently stops generating new trials.

1. Click Add Monitor → Cron Heartbeat.
2. Set the expected ping interval to 2 minutes.
3. Copy the heartbeat URL.
4. Add a ping call inside your Guild optimizer script or wrap the search process:

#!/bin/bash
# run-guild-search.sh — wraps a Guild hyperparameter search with heartbeating

HEARTBEAT_URL="https://vigilmon.online/heartbeat/abc123"

# Start the optimizer run in background
guild run train.py --optimize --max-trials 50 &
GUILD_PID=$!

# Ping the heartbeat every 60 seconds while the search is running
while kill -0 $GUILD_PID 2>/dev/null; do
    curl -s "$HEARTBEAT_URL" > /dev/null
    sleep 60
done

If the Guild search scheduler crashes or exits unexpectedly, the heartbeat loop ends and Vigilmon fires an alert.

Step 7: SSL Certificate Expiry Alerts

If you run a Guild remote server with HTTPS, certificate renewal failures are silent until your team sees connection errors when submitting remote runs.

1. Open the HTTP monitor you created in Step 1 for https://guild.yourdomain.com/.
2. Enable Monitor SSL certificate.
3. Set Alert when certificate expires in less than 21 days.
4. Click Save.

For teams using Let's Encrypt with auto-renewal (certbot or acme.sh), also add a heartbeat from the renewal cron job so you know renewals are running successfully:

# In /etc/cron.d/certbot or your renewal hook
0 12 * * * root certbot renew --quiet && curl -s https://vigilmon.online/heartbeat/renew123

Step 8: Configure Alert Channels

1. Go to Alert Channels in Vigilmon and add Slack, email, or a webhook for your ML team.
2. Set Consecutive failures before alert to 2 on the remote server HTTP monitor — brief network blips shouldn't page the team.
3. Use Maintenance windows in Vigilmon during Guild version upgrades:

curl -X POST https://vigilmon.online/api/maintenance \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -d '{"monitor_id": "abc123", "duration_minutes": 10}'

pip install --upgrade guildai

Summary

| Monitor | Target | What It Catches | |---|---|---| | Remote server | https://guild.domain.com/ | Server down, network unreachable | | SSH port | TCP :22 | SSH remote unreachable | | Artifact storage | Cron heartbeat | Disk full, NFS failure, permission loss | | Run database | Cron heartbeat | SQLite corruption, index inaccessible | | Metric log directory | HTTP health endpoint | Log write failure, filesystem issue | | Comparison API | Cron heartbeat | Slow DB queries, index degradation | | Search scheduler | Process heartbeat | Optimizer crash, early exit | | SSL certificate | Remote server domain | TLS renewal failure |

Guild.ai's power is its ability to track experiments without code changes — but the infrastructure it relies on (storage, database, remote server) can fail silently. With Vigilmon monitoring each layer, you'll catch storage failures, database corruption, and scheduler crashes before they corrupt your experiment history or silently halt a hyperparameter search.