tutorial

Monitoring Axolotl LLM Fine-Tuning with Vigilmon

Axolotl simplifies LLM fine-tuning on consumer and enterprise GPUs — but long-running training jobs have no built-in uptime alerting. Here's how to monitor TensorBoard, GPU health, checkpoint writes, and distributed workers with Vigilmon.

Axolotl wraps Hugging Face Transformers with a YAML-based configuration that makes LoRA, QLoRA, FSDP, and full fine-tuning accessible without writing complex training loops. But fine-tuning LLMs takes hours to days on expensive GPU hardware — and when something fails silently (OOM, lost checkpoint, dead W&B connection), you burn compute and lose training progress. Vigilmon gives you process heartbeats, GPU health checks, storage alerts, and SSL monitoring so you catch failures before they cost you another training run.

What You'll Set Up

  • TensorBoard server availability check (port 6006)
  • GPU health and CUDA availability heartbeat
  • Training checkpoint write health monitor
  • Dataset preprocessing pipeline heartbeat
  • Experiment tracker (W&B / MLflow / Comet) connectivity monitor
  • FSDP/DeepSpeed distributed worker heartbeat
  • Training process uptime monitor
  • Disk space saturation alert

Prerequisites

  • Axolotl installed (pip install axolotl) with at least one training configuration
  • GPU server accessible at a domain or IP (for HTTP-based monitors)
  • A free Vigilmon account

Step 1: Monitor TensorBoard Server Availability

Axolotl launches a TensorBoard instance on port 6006 during training runs to visualize loss curves, learning rate schedules, and gradient norms in real time. If TensorBoard goes down mid-run, your team loses live visibility into training health.

  1. Log in to vigilmon.online and click Add Monitor.
  2. Set Type to HTTP / HTTPS.
  3. Enter the TensorBoard URL: http://your-gpu-server.com:6006/.
  4. Set Check interval to 2 minutes.
  5. Set Expected HTTP status to 200.
  6. Click Save.

TensorBoard's root page returns a 200 when the server is running and has at least one event file loaded. If TensorBoard crashes or the port becomes inaccessible, the monitor fires immediately.

For teams running TensorBoard behind a reverse proxy with HTTPS, use the proxied URL and enable SSL certificate monitoring in the same step.

Step 2: GPU Health Heartbeat (CUDA Availability)

Axolotl fine-tuning requires substantial GPU VRAM — a QLoRA run on a 7B model needs at least 8 GB, while full fine-tuning of larger models needs tens of GB across multiple GPUs. CUDA errors or GPU node crashes kill training runs without warning.

  1. Click Add Monitor → Cron Heartbeat.
  2. Set the expected ping interval to 3 minutes.
  3. Copy the heartbeat URL (e.g., https://vigilmon.online/heartbeat/abc123).
  4. Add a GPU health check that pings only when all GPUs are healthy:
#!/usr/bin/env python3
# check-gpu-health.py

import subprocess
import requests

def check_gpus():
    result = subprocess.run(
        ['nvidia-smi', '--query-gpu=index,memory.used,memory.total,temperature.gpu,utilization.gpu',
         '--format=csv,noheader,nounits'],
        capture_output=True, text=True, timeout=10
    )
    if result.returncode != 0:
        return False  # nvidia-smi failed — GPU driver or device issue

    for line in result.stdout.strip().split('\n'):
        idx, mem_used, mem_total, temp, util = line.split(', ')
        # Alert if GPU temperature exceeds 90°C or memory is nearly full
        if int(temp) > 90:
            return False
        if int(mem_used) / int(mem_total) > 0.98:
            return False  # OOM imminent

    return True

if check_gpus():
    requests.get('https://vigilmon.online/heartbeat/abc123', timeout=5)

Schedule this script every 2 minutes:

*/2 * * * * /usr/bin/python3 /opt/axolotl/check-gpu-health.py

Step 3: Monitor Training Checkpoint Write Health

Axolotl saves model checkpoints to the output_dir specified in your YAML config at each save step. A disk full or permission error silently fails checkpoint writes — you only discover lost progress when the run completes and the output directory is empty.

  1. Click Add Monitor → Cron Heartbeat.
  2. Set the expected ping interval to 15 minutes.
  3. Copy the heartbeat URL.
  4. Add a checkpoint health check that verifies the output directory is writable and has recent checkpoints:
#!/usr/bin/env python3
# check-checkpoint-health.py

import os
import time
import glob
import requests

OUTPUT_DIR = '/path/to/axolotl/outputs'
HEARTBEAT_URL = 'https://vigilmon.online/heartbeat/abc123'

def check_checkpoint_health():
    # Verify write access
    test_file = os.path.join(OUTPUT_DIR, '.healthcheck')
    try:
        with open(test_file, 'w') as f:
            f.write('ok')
        os.remove(test_file)
    except OSError:
        return False  # Write failed — disk full or permission issue

    # Verify checkpoints were saved recently (within last 30 minutes)
    checkpoints = glob.glob(os.path.join(OUTPUT_DIR, 'checkpoint-*'))
    if checkpoints:
        latest = max(checkpoints, key=os.path.getmtime)
        age_minutes = (time.time() - os.path.getmtime(latest)) / 60
        if age_minutes > 30:
            return False  # No new checkpoint in 30 minutes during active training

    return True

if check_checkpoint_health():
    requests.get(HEARTBEAT_URL, timeout=5)

Step 4: Dataset Preprocessing Pipeline Heartbeat

Axolotl tokenizes and preprocesses datasets before training starts. This preprocessing can take minutes to hours for large datasets. If tokenization workers hang or cached dataset files become corrupted, training silently stalls at startup.

Add a heartbeat from the preprocessing completion hook in your training script:

# In your custom training callback or post-preprocessing hook
import requests

class VigilmonHeartbeatCallback:
    """Ping Vigilmon after each successful preprocessing step."""
    
    def on_preprocessing_complete(self, dataset_size: int):
        try:
            requests.get(
                'https://vigilmon.online/heartbeat/abc123',
                timeout=5
            )
        except Exception:
            pass  # Don't fail training if Vigilmon is unreachable

For standard Axolotl runs without custom callbacks, add a pre-training check script:

#!/bin/bash
# preprocess-and-ping.sh

python -m axolotl.cli.preprocess config.yml && \
    curl -s https://vigilmon.online/heartbeat/abc123

Set the heartbeat interval to match your expected preprocessing time plus a buffer (e.g., if preprocessing takes 20 minutes, set the interval to 30 minutes).

Step 5: Experiment Tracker Connectivity (W&B / MLflow / Comet)

Axolotl integrates with Weights & Biases, MLflow, and Comet for metric logging. If the tracker API becomes unreachable mid-training, metric logging silently fails and your experiment history has gaps.

For Weights & Biases, monitor the API endpoint:

  1. Click Add Monitor → HTTP / HTTPS.
  2. Enter https://api.wandb.ai/health.
  3. Set Expected HTTP status to 200.
  4. Set Check interval to 5 minutes.
  5. Click Save.

For self-hosted MLflow:

  1. Click Add Monitor → HTTP / HTTPS.
  2. Enter http://your-mlflow-server:5000/health.
  3. Set Expected HTTP status to 200.
  4. Set Check interval to 2 minutes.

For self-hosted Comet or any other tracker, monitor the tracker's health or root endpoint with a 200 check.

Step 6: FSDP/DeepSpeed Distributed Worker Heartbeat

When running multi-GPU training with FSDP or DeepSpeed, Axolotl spawns multiple worker processes that must stay in sync. If any worker crashes, the process group breaks and training halts — often with an opaque NCCL timeout error.

Monitor distributed worker health with a shared heartbeat file approach:

#!/usr/bin/env python3
# distributed-heartbeat-callback.py

import os
import time
import requests
import torch.distributed as dist

class DistributedHeartbeatCallback:
    """
    Each worker writes a timestamp. The rank-0 process checks all workers
    and pings Vigilmon only when all are alive.
    """
    
    def __init__(self, heartbeat_url: str, heartbeat_dir: str, world_size: int):
        self.heartbeat_url = heartbeat_url
        self.heartbeat_dir = heartbeat_dir
        self.world_size = world_size
        os.makedirs(heartbeat_dir, exist_ok=True)

    def on_step_end(self, rank: int, step: int):
        # Each worker writes its heartbeat
        with open(os.path.join(self.heartbeat_dir, f'worker_{rank}'), 'w') as f:
            f.write(str(time.time()))
        
        # Only rank-0 pings Vigilmon, after verifying all workers are alive
        if rank == 0 and step % 10 == 0:
            now = time.time()
            all_alive = all(
                now - float(open(os.path.join(self.heartbeat_dir, f'worker_{r}')).read()) < 120
                for r in range(self.world_size)
                if os.path.exists(os.path.join(self.heartbeat_dir, f'worker_{r}'))
            )
            if all_alive:
                try:
                    requests.get(self.heartbeat_url, timeout=5)
                except Exception:
                    pass

Set the Vigilmon heartbeat interval to 5 minutes. If any GPU worker dies and rank-0 stops seeing recent timestamps from all workers, the heartbeat stops.

Step 7: Training Process Uptime Monitor

The core Axolotl training process itself needs monitoring. A Python OOM kill, segfault, or unhandled exception can terminate the process without any HTTP or port being closed — making it invisible to standard TCP monitors.

Use a process-wrap heartbeat:

#!/bin/bash
# train-with-heartbeat.sh

HEARTBEAT_URL="https://vigilmon.online/heartbeat/abc123"
CONFIG="config.yml"

# Ping every 60 seconds while training is alive
(while true; do
    curl -s "$HEARTBEAT_URL" > /dev/null
    sleep 60
done) &
PING_PID=$!

# Run Axolotl training
python -m axolotl.cli.train "$CONFIG"
TRAIN_EXIT=$?

# Kill the heartbeat loop when training finishes
kill $PING_PID 2>/dev/null

exit $TRAIN_EXIT
  1. Click Add Monitor → Cron Heartbeat.
  2. Set the expected ping interval to 3 minutes.
  3. Use the heartbeat URL in the script above.

If the training process crashes, the ping loop exits and Vigilmon alerts after 3 minutes of silence.

Step 8: Disk Space Saturation Alert

LLM checkpoints are large — a 7B model checkpoint is 13–28 GB; a 70B model checkpoint can exceed 140 GB. Axolotl saves multiple checkpoints during training, and disk saturation mid-run causes catastrophic checkpoint failure and training abort.

Add a disk space monitor to your heartbeat script:

#!/usr/bin/env python3
# check-disk-space.py

import shutil
import requests

OUTPUT_DIR = '/path/to/axolotl/outputs'
HEARTBEAT_URL = 'https://vigilmon.online/heartbeat/abc123'
MIN_FREE_GB = 50  # Alert if less than 50 GB free

total, used, free = shutil.disk_usage(OUTPUT_DIR)
free_gb = free / (1024 ** 3)

if free_gb >= MIN_FREE_GB:
    requests.get(HEARTBEAT_URL, timeout=5)
# If free_gb < MIN_FREE_GB, heartbeat is NOT sent — Vigilmon alerts

Schedule this every 10 minutes. Set the Vigilmon heartbeat interval to 15 minutes to give a buffer for normal checkpoint writes.

Step 9: Configure Alert Channels

  1. Go to Alert Channels in Vigilmon and add Slack or PagerDuty for your ML infrastructure team.
  2. For TensorBoard and training process monitors, set Consecutive failures before alert to 2 — brief GPU driver reloads can cause a single probe failure.
  3. For disk space and checkpoint monitors, set Consecutive failures to 1 — disk saturation needs immediate response.
  4. Use Maintenance windows during planned checkpoint cleanups:
curl -X POST https://vigilmon.online/api/maintenance \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -d '{"monitor_id": "abc123", "duration_minutes": 20}'

# Clean up old checkpoints
find /path/to/outputs -name 'checkpoint-*' -mtime +7 -exec rm -rf {} +

Summary

| Monitor | Target | What It Catches | |---|---|---| | TensorBoard | HTTP :6006 | Server crash, port unreachable | | GPU health | Cron heartbeat | OOM imminent, temperature spike, driver failure | | Checkpoint writes | Cron heartbeat | Disk full, permission error, stalled saves | | Dataset preprocessing | Process heartbeat | Tokenization hang, corrupted cache | | W&B / MLflow | HTTP health endpoint | Tracker API down, metric logging gap | | Distributed workers | Shared heartbeat | NCCL failure, worker process death | | Training process | Process wrap heartbeat | Unhandled exception, OOM kill, segfault | | Disk space | Cron heartbeat | Storage saturation before checkpoint failure |

LLM fine-tuning runs are expensive — in both compute time and GPU cost. With Vigilmon monitoring your Axolotl training infrastructure from GPU health to checkpoint writes, you catch failures in minutes rather than discovering them hours later when the run is already dead.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles