Monitoring Valohai with Vigilmon

Valohai gives ML teams reproducible training infrastructure with automatic data versioning, experiment tracking, deployment pipelines, and audit trails. But as a self-hosted platform, when the Django web app, Celery workers, or execution agents fail, training jobs queue silently and experiments stop running. Vigilmon gives you per-component monitoring across Valohai's architecture — so failures surface in your alert channel, not in your ML team's Slack.

What You'll Set Up

• HTTP uptime monitor for the Valohai web application (port 8000)
• PostgreSQL and Redis TCP connectivity checks
• Celery worker heartbeat monitoring
• Execution agent per-machine health checks
• Data store reachability monitor
• Deployment endpoint availability
• Pipeline scheduler health
• SSL certificate expiry alerts

Prerequisites

• Valohai self-hosted deployment running (Django app + Celery workers + execution agents)
• Web application accessible at port 8000 or via a reverse-proxied domain
• A free Vigilmon account

Step 1: Monitor the Valohai Web Application

The Valohai Django app at port 8000 is both the primary UI and the REST API for submitting jobs, viewing experiments, and managing pipelines. If it's down, your ML team can't interact with the platform.

1. Log in to vigilmon.online and click Add Monitor.
2. Set Type to HTTP / HTTPS.
3. Enter the web app URL: https://valohai.yourdomain.com/health/ (or http://localhost:8000/health/).
4. Set Check interval to 1 minute.
5. Set Expected HTTP status to 200.
6. Click Save.

Valohai's Django app should expose a /health/ endpoint. If it doesn't exist, add a minimal one:

# urls.py
from django.http import JsonResponse

def health_check(request):
    return JsonResponse({'status': 'ok'})

urlpatterns = [
    path('health/', health_check, name='health'),
    # ... other routes
]

A richer health check can verify database and cache connectivity before returning 200.

Step 2: Monitor PostgreSQL Connectivity (TCP)

PostgreSQL stores Valohai's projects, executions, data versions, deployment endpoints, and audit logs. A database outage stops all experiment recording and pipeline state tracking.

1. Click Add Monitor → TCP Port.
2. Enter your PostgreSQL host.
3. Set Port to 5432.
4. Set Check interval to 1 minute.
5. Click Save.

For an application-level check, add a database health endpoint to the Django app:

from django.db import connection
from django.http import JsonResponse

def db_health(request):
    try:
        with connection.cursor() as cursor:
            cursor.execute("SELECT 1")
        return JsonResponse({'status': 'ok', 'database': 'connected'})
    except Exception as e:
        return JsonResponse({'status': 'error', 'detail': str(e)}, status=503)

Register this at /health/db/ and add a separate Vigilmon HTTP monitor for it.

Step 3: Monitor Redis Connectivity (TCP)

Redis is Valohai's Celery broker and WebSocket pub/sub backend for live execution log streaming. If Redis is down, background task processing stops and users can't see live training logs.

1. Click Add Monitor → TCP Port.
2. Enter your Redis host.
3. Set Port to 6379.
4. Set Check interval to 1 minute.
5. Click Save.

Add a Redis health endpoint:

import redis as redis_client
from django.http import JsonResponse

REDIS_HOST = 'redis-host'

def redis_health(request):
    try:
        r = redis_client.Redis(host=REDIS_HOST, port=6379, socket_timeout=2)
        r.ping()
        return JsonResponse({'status': 'ok', 'redis': 'connected'})
    except redis_client.ConnectionError as e:
        return JsonResponse({'status': 'error', 'detail': str(e)}, status=503)

Step 4: Heartbeat Monitoring for Celery Workers

Valohai's Celery workers handle execution scheduling, data versioning, and pipeline orchestration. They don't expose an HTTP endpoint — use Vigilmon's cron heartbeat.

1. Click Add Monitor → Cron Heartbeat.
2. Set the expected ping interval to 5 minutes.
3. Copy the heartbeat URL.
4. Add a periodic Celery task:

# tasks.py
from celery import shared_task
import requests

@shared_task
def vigilmon_heartbeat():
    """Periodic health ping to confirm Celery workers are alive."""
    requests.get('https://vigilmon.online/heartbeat/abc123', timeout=5)

Register it in your Celery beat schedule:

# celery.py or settings.py
app.conf.beat_schedule = {
    'vigilmon-heartbeat': {
        'task': 'myapp.tasks.vigilmon_heartbeat',
        'schedule': 60.0,  # every 60 seconds
    },
}

If workers crash or the Celery broker becomes unreachable, the heartbeat stops and Vigilmon alerts after 5 minutes.

Step 5: Per-Agent Execution Health Checks

Valohai uses worker agents on compute machines or cloud VMs to run training jobs. Each agent connects to the Valohai API and polls for work. An unreachable agent means training jobs assigned to it will never start.

Add a health endpoint to each Valohai execution agent machine:

# agent_health.py — run as a lightweight Flask sidecar on each agent machine
from flask import Flask, jsonify
import subprocess

app = Flask(__name__)

@app.route('/health')
def health():
    # Check if the Valohai worker process is running
    result = subprocess.run(['pgrep', '-f', 'valohai-worker'], capture_output=True)
    if result.returncode == 0:
        return jsonify({'status': 'ok', 'agent': 'running'})
    return jsonify({'status': 'error', 'agent': 'not running'}), 503

if __name__ == '__main__':
    app.run(host='0.0.0.0', port=8001)

For each execution agent machine:

1. Click Add Monitor → HTTP / HTTPS.
2. Enter: http://agent-machine-ip:8001/health.
3. Set Expected HTTP status to 200.
4. Set Check interval to 2 minutes.

Step 6: Monitor Data Store Connectivity

Valohai tracks all inputs and outputs with content hashes for data versioning, writing to S3, GCS, or Azure Blob. If the data store is unreachable, executions fail to write outputs and data versioning breaks.

For S3-compatible stores:

1. Click Add Monitor → HTTP / HTTPS.
2. Enter a lightweight S3 health check URL (e.g., a known-good object or the bucket endpoint).
3. Set Expected HTTP status to 200 or 403 (403 on a public-access-blocked bucket means the service is reachable).
4. Set Check interval to 5 minutes.

For a write-read-delete connectivity check from within your infrastructure:

import boto3

def check_s3(request):
    s3 = boto3.client('s3')
    test_key = '.valohai-healthcheck'
    bucket = 'your-valohai-bucket'
    try:
        s3.put_object(Bucket=bucket, Key=test_key, Body=b'ok')
        s3.get_object(Bucket=bucket, Key=test_key)
        s3.delete_object(Bucket=bucket, Key=test_key)
        return JsonResponse({'status': 'ok', 'storage': 's3 reachable'})
    except Exception as e:
        return JsonResponse({'status': 'error', 'detail': str(e)}, status=503)

Step 7: Monitor Deployment Endpoints

Valohai can serve prediction APIs from deployed models. If you use Valohai deployments, each deployed endpoint should be monitored independently:

1. Click Add Monitor → HTTP / HTTPS.
2. Enter the deployment endpoint URL: https://your-deployment.valohai.cloud/predict.
3. Set Expected HTTP status to 200.
4. Optionally enable Keyword check for an expected response field.
5. Set Check interval to 2 minutes.

For POST-based prediction endpoints, use a cron heartbeat probe that sends a test inference request:

import requests

def probe_deployment():
    test_payload = {"inputs": [1.0, 2.0, 3.0]}  # replace with your model's input schema
    resp = requests.post(
        'https://your-deployment.valohai.cloud/predict',
        json=test_payload,
        timeout=10
    )
    if resp.status_code == 200:
        requests.get('https://vigilmon.online/heartbeat/deploy123', timeout=5)
    else:
        print(f"Deployment probe failed: {resp.status_code}")

Step 8: Pipeline Scheduler Health

Valohai pipelines chain training stages together. The pipeline scheduler is a Celery beat process — monitor it the same way as Celery workers, with a separate heartbeat that fires from the pipeline scheduling task:

@shared_task
def pipeline_scheduler_heartbeat():
    """Confirm the pipeline scheduler beat process is alive."""
    requests.get('https://vigilmon.online/heartbeat/pipeline123', timeout=5)

# In beat schedule
app.conf.beat_schedule.update({
    'pipeline-scheduler-heartbeat': {
        'task': 'myapp.tasks.pipeline_scheduler_heartbeat',
        'schedule': 120.0,  # every 2 minutes
    },
})

Step 9: SSL Certificate Expiry Alerts

1. Open the HTTP monitor for https://valohai.yourdomain.com/.
2. Enable Monitor SSL certificate.
3. Set Alert when certificate expires in less than 21 days.
4. Click Save.

Repeat for each deployment endpoint domain if they use separate certificates.

Step 10: Configure Alert Channels

1. Go to Alert Channels in Vigilmon and add Slack, email, or a webhook.
2. Set Consecutive failures before alert to 2 on the web app monitor — Django restarts can cause brief probe failures.
3. Use Maintenance windows during platform upgrades:

curl -X POST https://vigilmon.online/api/maintenance \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -d '{"monitor_id": "abc123", "duration_minutes": 30}'

# Perform platform upgrade
docker-compose up -d --force-recreate

Summary

| Monitor | Target | What It Catches | |---|---|---| | Web app | https://valohai.domain.com/health/ | Django server crash | | PostgreSQL | TCP :5432 | Database connectivity loss | | Redis | TCP :6379 | Broker/websocket failure | | Celery workers | Heartbeat URL | Worker crash, queue stall | | Execution agents | http://agent:8001/health | Per-machine agent failure | | Data store | S3/GCS health endpoint | Storage write/read failure | | Deployment endpoint | Prediction API URL | Model serving down | | Pipeline scheduler | Heartbeat URL | Pipeline scheduling stopped | | SSL certificate | HTTPS domain | TLS renewal failure |

Valohai's self-hosted architecture gives ML teams full control over their training infrastructure — and full responsibility for its uptime. With Vigilmon monitoring every layer, from the Django web app to per-agent health checks to deployment endpoints, you get the observability needed to keep reproducible training pipelines running reliably.