Monitoring Your Stable Horde Worker with Vigilmon: Worker Process Health, GPU Utilization & API Connectivity Checks

Stable Horde is a distributed AI computing network where participants contribute their GPU resources to a shared pool for Stable Diffusion image generation and LLM text inference. You run a horde-worker-reGen process on your machine, and the Horde API dispatches generation jobs to it. Contributing workers earn kudos — Horde's resource-sharing currency — which can be spent to generate images yourself or shared with others. But a Horde worker is a long-running daemon that depends on GPU hardware, outbound API connectivity, and model availability. Process crashes, VRAM exhaustion, and network interruptions can silently de-register your worker from the Horde pool. Vigilmon monitors worker process health, API bridge connectivity, and GPU utilization to detect failures before they cause extended downtime.

What You'll Build

• A worker process availability heartbeat
• GPU/VRAM and job completion monitoring
• Horde API bridge connectivity check (outbound)
• Job acceptance rate tracking
• Worker status check via Horde public API
• Alerting for process crashes, GPU failures, and de-registration events

Prerequisites

• horde-worker-reGen installed and running
• At least one Stable Diffusion model downloaded for the worker
• A registered account at aihorde.net with an API key
• A free account at vigilmon.online

Step 1: Understand Stable Horde Worker Architecture

The horde-worker-reGen process communicates outbound with the Horde API bridge (no inbound ports required). The worker:

1. Polls the Horde API for available jobs
2. Downloads generation parameters
3. Runs inference locally using GPU/CPU
4. Returns results to the Horde API

There is no built-in HTTP health endpoint on the worker itself — monitoring requires a combination of process checks, GPU probes, and Horde API queries.

Step 2: Monitor Worker Process Health with a Heartbeat

Use a cron job that checks whether the horde-worker-reGen process is running and sends a Vigilmon heartbeat:

#!/bin/bash
# /etc/cron.d/horde-worker-heartbeat — runs every 5 minutes

WORKER_PROCESS="horde-worker-reGen"

# Check if the worker process is running
if pgrep -f "$WORKER_PROCESS" > /dev/null 2>&1; then
  curl -fsS -X POST https://vigilmon.online/api/heartbeat/YOUR-HEARTBEAT-ID
fi

Create a Heartbeat monitor in Vigilmon:

1. Log in to Vigilmon → Add Monitor → Heartbeat.
2. Expected interval: 5 minutes.
3. Grace period: 10 minutes.
4. Alert: when heartbeat is missed.

This catches process crashes and unclean shutdowns. If the worker process exits unexpectedly, the heartbeat stops within one check window.

Step 3: Monitor the Horde API Bridge Connectivity

The worker must maintain outbound HTTP connectivity to the Horde API endpoint to receive jobs. Test connectivity separately from process health — the worker process can be running while the Horde API is temporarily unreachable:

#!/bin/bash
# /etc/cron.d/horde-api-connectivity — runs every 5 minutes

HORDE_API="https://aihorde.net/api/v2/status/heartbeat"

# Check outbound connectivity to the Horde API
HTTP_STATUS=$(curl -s -o /dev/null -w "%{http_code}" \
  --max-time 15 "$HORDE_API")

if [ "$HTTP_STATUS" = "200" ]; then
  curl -fsS -X POST https://vigilmon.online/api/heartbeat/YOUR-API-CONNECTIVITY-HEARTBEAT-ID
fi

Add a second Heartbeat monitor for API connectivity, or use Vigilmon's HTTP monitor directly:

1. Add Monitor → HTTP.
2. URL: https://aihorde.net/api/v2/status/heartbeat
3. Check interval: 60 seconds.
4. Expected status: 200.
5. Keyword: OK or message.

This monitor checks whether the Horde API is reachable from your monitoring network. For your worker's outbound connectivity specifically, use the cron heartbeat script above which runs on the worker host itself.

Step 4: Monitor Worker Registration Status via the Horde API

The Horde API exposes worker status publicly. You can monitor whether your worker is online and actively accepting jobs:

# Check your worker's status via the Horde API
curl -s "https://aihorde.net/api/v2/workers?name=YOUR-WORKER-NAME" \
  -H "apikey: YOUR-API-KEY"

The response includes worker state, last heartbeat time, kudos accumulated, and jobs completed. Add a Vigilmon HTTP monitor:

1. Add Monitor → HTTP.
2. URL: https://aihorde.net/api/v2/workers?name=YOUR-WORKER-NAME
3. Expected status: 200.
4. Keyword: YOUR-WORKER-NAME (confirms your worker appears in the registry).
5. Check interval: 5 minutes.

If your worker de-registers from the Horde (too many failed jobs, extended offline period, API key issue), this check fires.

Step 5: Monitor GPU Utilization and VRAM

GPU availability is the most critical dependency for a Horde worker — without GPU, inference either fails or falls back to slow CPU. Add GPU monitoring to the process heartbeat:

#!/bin/bash
# Enhanced heartbeat with GPU/VRAM check

# Check GPU availability and free VRAM
GPU_NAME=$(nvidia-smi --query-gpu=name --format=csv,noheader 2>/dev/null | head -1)
FREE_VRAM=$(nvidia-smi --query-gpu=memory.free --format=csv,noheader,nounits 2>/dev/null | head -1)
GPU_UTIL=$(nvidia-smi --query-gpu=utilization.gpu --format=csv,noheader,nounits 2>/dev/null | head -1)

WORKER_PROCESS="horde-worker-reGen"

if [ -z "$GPU_NAME" ]; then
  echo "GPU not available"
  exit 1
fi

if [ "$FREE_VRAM" -lt "1000" ]; then
  echo "VRAM critically low: ${FREE_VRAM}MB free"
  # Don't ping — worker may be processing a job, this is normal
  # Only skip ping if worker process is also not running
fi

if pgrep -f "$WORKER_PROCESS" > /dev/null 2>&1; then
  curl -fsS -X POST https://vigilmon.online/api/heartbeat/YOUR-HEARTBEAT-ID
fi

Note on GPU utilization: A Horde worker may show 0% GPU utilization between jobs — this is normal idle state. Low utilization becomes a concern only if combined with a drop in job acceptance rate (measured separately below).

Step 6: Monitor Job Acceptance and Completion Rate

Use the Horde API to track your worker's job performance over time. The following script checks kudos accumulation as a proxy for healthy job completion:

#!/bin/bash
# /etc/cron.d/horde-kudos-check — runs every 30 minutes
# Checks that kudos are accumulating (sign of successful job completions)

WORKER_NAME="YOUR-WORKER-NAME"
API_KEY="YOUR-API-KEY"
KUDOS_FILE="/tmp/horde_kudos_last"

CURRENT_KUDOS=$(curl -s "https://aihorde.net/api/v2/workers?name=$WORKER_NAME" \
  -H "apikey: $API_KEY" | \
  python3 -c "import sys,json; data=json.load(sys.stdin); \
    w=[x for x in data if x.get('name')=='$WORKER_NAME']; \
    print(int(w[0].get('kudos_generated',0)) if w else 0)" 2>/dev/null)

if [ -z "$CURRENT_KUDOS" ]; then
  exit 1
fi

# Compare with previous value
LAST_KUDOS=$(cat "$KUDOS_FILE" 2>/dev/null || echo "0")

if [ "$CURRENT_KUDOS" -gt "$LAST_KUDOS" ]; then
  echo "$CURRENT_KUDOS" > "$KUDOS_FILE"
  curl -fsS -X POST https://vigilmon.online/api/heartbeat/YOUR-KUDOS-HEARTBEAT-ID
fi

Create a Heartbeat monitor with a 60-minute interval and 30-minute grace period. If kudos stop accumulating for over an hour, the worker may be offline or failing jobs.

Step 7: Check Model Availability on Disk

The Horde worker must have the requested models downloaded locally to accept related jobs. A missing model causes job failures and eventual de-registration from those model queues:

#!/bin/bash
# /etc/cron.d/horde-models-check — runs every 15 minutes

MODELS_DIR="${HOME}/horde-worker-reGen/models"  # adjust to your install path
REQUIRED_MODELS=(
  "v1-5-pruned-emaonly.safetensors"
  # add other models your worker serves
)

ALL_PRESENT=true
for model in "${REQUIRED_MODELS[@]}"; do
  if [ ! -f "$MODELS_DIR/$model" ]; then
    echo "Missing model: $model"
    ALL_PRESENT=false
  fi
done

if $ALL_PRESENT; then
  curl -fsS -X POST https://vigilmon.online/api/heartbeat/YOUR-MODELS-HEARTBEAT-ID
fi

Step 8: Configure Alerting

Wire up alert channels in Vigilmon under Settings → Notifications:

| Monitor | Trigger | First action | |---|---|---| | Worker process heartbeat | Missed > 10 min | Check process: pgrep -f horde-worker-reGen; restart worker | | Horde API connectivity | Non-200 | Check internet connectivity and DNS; Horde status at aihorde.net/api/v2/status | | Worker registry check | Worker name missing | Check API key validity; review worker logs for de-registration errors | | Kudos heartbeat | Missed > 90 min | Worker may be failing all jobs; check job logs; verify GPU/models | | Model availability | Heartbeat missed | Re-download missing models; check disk space |

Alert after 1 missed heartbeat for the process check — a missed heartbeat is unambiguous. Alert after 2 consecutive failures for the HTTP API checks.

Common Stable Horde Worker Failure Modes and What Vigilmon Catches

| Scenario | Vigilmon monitor | |---|---| | Worker process crashes | Process heartbeat stops within 5 min | | Horde API temporarily unreachable | API connectivity monitor fires | | Worker de-registered after repeated failures | Worker registry check fires | | GPU VRAM exhausted | GPU heartbeat stops; job completion falls | | Required model missing on disk | Model availability heartbeat stops | | Kudos/jobs stall (no new work received) | Kudos accumulation heartbeat stops over 90 min | | Internet connectivity lost | API connectivity and kudos heartbeats both stop |

Contributing a Stable Horde worker is a meaningful way to democratize access to AI image generation, but a worker daemon that crashes silently or gets de-registered provides no value to the Horde community or to your own kudos balance. Vigilmon's heartbeat monitoring catches process failures, GPU issues, and de-registration events — keeping your worker visible and productive.

Start monitoring your Stable Horde worker in under 5 minutes — register free at vigilmon.online.