Monitoring Polyaxon with Vigilmon

Polyaxon gives ML teams a unified control plane for running reproducible experiments, tracking metrics, and deploying models across GPU infrastructure. It's powerful — but it's also a multi-component system running on Kubernetes, and when any layer fails, experiment scheduling silently stalls. Vigilmon gives you per-component uptime monitoring, heartbeat checks for background workers, and SSL certificate alerts so you know about failures before your ML team does.

What You'll Set Up

• HTTP uptime monitor for the Polyaxon API server
• PostgreSQL and Redis TCP connectivity checks
• Celery worker heartbeat monitoring
• Kubernetes API health check
• Artifact store (S3/GCS/NFS) reachability monitor
• Web UI availability check
• SSL certificate expiry alerts

Prerequisites

• Polyaxon deployed on Kubernetes (Helm chart or operator install)
• Polyaxon API server accessible at a domain or ingress IP
• A free Vigilmon account

Step 1: Monitor the Polyaxon API Server

The Polyaxon API server is a Django REST API that manages all experiments, runs, projects, and artifact metadata. If it's down, your ML team can't submit jobs or query results.

1. Log in to vigilmon.online and click Add Monitor.
2. Set Type to HTTP / HTTPS.
3. Enter the API server URL: https://polyaxon.yourdomain.com/api/v1/ (or your ingress URL).
4. Set Check interval to 1 minute.
5. Set Expected HTTP status to 200.
6. Click Save.

Polyaxon exposes a health endpoint at /healthz on the API server pod. If you have internal network access (e.g., a monitoring agent inside the cluster), prefer that over the root API URL:

https://polyaxon.yourdomain.com/healthz

The /healthz endpoint checks Django application health without requiring authentication.

Step 2: Monitor PostgreSQL Connectivity (TCP)

PostgreSQL is Polyaxon's primary data store — it holds all experiment metadata, run logs, hyperparameters, metrics, and artifact references. A lost database connection means experiments can't be saved or queried.

1. Click Add Monitor → TCP Port.
2. Enter your PostgreSQL host (e.g., postgres.yourdomain.com or the Kubernetes service name accessible via ingress).
3. Set Port to 5432.
4. Set Check interval to 1 minute.
5. Click Save.

If PostgreSQL runs inside the cluster without an external ingress, expose a TCP probe via a LoadBalancer service or use a sidecar exporter with an HTTP wrapper. Alternatively, add a /db-health endpoint to the Polyaxon API that verifies the database connection:

# In a custom Django management command or view
from django.db import connection

def db_health(request):
    try:
        connection.ensure_connection()
        return JsonResponse({'status': 'ok'})
    except Exception as e:
        return JsonResponse({'status': 'error', 'detail': str(e)}, status=503)

Step 3: Monitor Redis Connectivity (TCP)

Redis serves as Polyaxon's Celery broker for background tasks and pub/sub channel for real-time log streaming. If Redis is unreachable, experiment scheduling halts and live log streaming breaks.

1. Click Add Monitor → TCP Port.
2. Enter your Redis host.
3. Set Port to 6379.
4. Set Check interval to 1 minute.
5. Click Save.

For a richer check, add a Redis ping endpoint to your API:

import redis

def redis_health(request):
    try:
        r = redis.Redis(host='redis-host', port=6379)
        r.ping()
        return JsonResponse({'status': 'ok'})
    except redis.ConnectionError as e:
        return JsonResponse({'status': 'error', 'detail': str(e)}, status=503)

Step 4: Heartbeat Monitoring for Celery Workers

Polyaxon's Celery workers handle experiment scheduling, artifact cleanup, and notifications. They don't expose an HTTP port — use Vigilmon's cron heartbeat to confirm they're alive.

1. Click Add Monitor → Cron Heartbeat.
2. Set the expected ping interval to 5 minutes.
3. Copy the heartbeat URL (e.g., https://vigilmon.online/heartbeat/abc123).
4. Add a periodic Celery task that pings the heartbeat URL:

# tasks.py
from celery import shared_task
import requests

@shared_task
def vigilmon_heartbeat():
    requests.get('https://vigilmon.online/heartbeat/abc123', timeout=5)

Register the task in your Celery beat schedule:

# settings.py
CELERY_BEAT_SCHEDULE = {
    'vigilmon-heartbeat': {
        'task': 'myapp.tasks.vigilmon_heartbeat',
        'schedule': 60.0,  # every 60 seconds
    },
}

If Celery workers crash or the queue backs up, the heartbeat stops pinging and Vigilmon alerts after 5 minutes.

Step 5: Monitor Kubernetes API Connectivity

Polyaxon schedules training jobs as Kubernetes pods — if the K8s API server is unreachable, no new experiments can be dispatched. Monitor the Kubernetes API health endpoint:

1. Click Add Monitor → HTTP / HTTPS.
2. Enter the Kubernetes API health URL: https://your-k8s-api:6443/healthz.
3. Set Expected HTTP status to 200.
4. Set Check interval to 1 minute.
5. Click Save.

If the K8s API is only reachable from inside the cluster, expose a lightweight proxy or use an in-cluster monitoring agent. For managed Kubernetes (EKS, GKE, AKS), the control plane health endpoint is typically available at your cluster's API server address.

Step 6: Monitor the Artifact Store

Polyaxon stores model checkpoints, datasets, and experiment outputs in object storage (S3, GCS, Azure Blob) or NFS. Storage connectivity issues mean artifacts can't be written or read during training.

For S3-compatible stores, add an HTTP monitor targeting the storage bucket endpoint:

1. Click Add Monitor → HTTP / HTTPS.
2. Enter a lightweight health check URL for your object store (e.g., an S3 bucket listing endpoint or a known-good object URL).
3. Set Expected HTTP status to 200 or 403 (a 403 on a bucket listing means the service is up but access is restricted — that's fine for connectivity checks).
4. Set Check interval to 5 minutes.

For NFS or local storage, add a heartbeat from a Polyaxon worker that writes and reads a test file:

import os

def check_artifact_store(request):
    test_path = '/mnt/artifacts/.healthcheck'
    try:
        with open(test_path, 'w') as f:
            f.write('ok')
        with open(test_path, 'r') as f:
            assert f.read() == 'ok'
        os.remove(test_path)
        return JsonResponse({'status': 'ok'})
    except Exception as e:
        return JsonResponse({'status': 'error', 'detail': str(e)}, status=503)

Step 7: Monitor the Polyaxon Web UI

The React dashboard is how your ML team compares experiments and manages runs. It's served through the same ingress as the API but is a separate concern.

1. Click Add Monitor → HTTP / HTTPS.
2. Enter the web UI URL: https://polyaxon.yourdomain.com/.
3. Set Expected HTTP status to 200.
4. Enable Keyword check and look for Polyaxon in the response body.
5. Set Check interval to 5 minutes.

Step 8: SSL Certificate Expiry Alerts

Polyaxon's ingress handles TLS termination. Certificate renewal failures are silent until users see browser warnings.

1. Open the HTTP monitor for https://polyaxon.yourdomain.com/.
2. Enable Monitor SSL certificate.
3. Set Alert when certificate expires in less than 21 days.
4. Click Save.

Step 9: Configure Alert Channels

1. Go to Alert Channels in Vigilmon and add Slack, email, or a webhook.
2. Set Consecutive failures before alert to 2 on the API server monitor — Kubernetes pod restarts can cause brief probe failures.
3. Use Maintenance windows in Vigilmon during Helm upgrades or cluster maintenance:

curl -X POST https://vigilmon.online/api/maintenance \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -d '{"monitor_id": "abc123", "duration_minutes": 15}'

helm upgrade polyaxon polyaxon/polyaxon --values values.yaml

Summary

| Monitor | Target | What It Catches | |---|---|---| | API server | https://polyaxon.domain.com/healthz | Django API crash, ingress failure | | PostgreSQL | TCP :5432 | Database connectivity loss | | Redis | TCP :6379 | Broker unreachable, log streaming broken | | Celery worker | Heartbeat URL | Worker crash, queue backlog | | Kubernetes API | https://k8s-api:6443/healthz | K8s control plane unreachable | | Artifact store | S3/GCS endpoint or NFS health | Storage I/O failure | | Web UI | https://polyaxon.domain.com/ | Dashboard unavailable | | SSL certificate | Each ingress domain | TLS renewal failure |

Polyaxon's power comes from orchestrating many moving parts — and that means more things that can fail silently. With Vigilmon watching each layer, from the API server to the Kubernetes control plane to Celery workers, you catch failures before they stall your ML team's experiment pipeline.