tutorial

Monitoring Volcano on Kubernetes with Vigilmon

Volcano brings batch scheduling, gang scheduling, and queue-based fair-share resource allocation to Kubernetes — but when the Volcano controller crashes, a queue enters a deadlock state, or job pods get stuck in Pending indefinitely, your ML training and HPC workloads silently stall with no completion signal. Here's how to monitor Volcano scheduler health, queue depth, job completion rates, and resource allocation fairness with Vigilmon.

Volcano is a Kubernetes-native batch scheduling system designed for machine learning training, HPC workloads, and data processing jobs that need capabilities beyond what the default Kubernetes scheduler provides. Its gang scheduling ensures that all pods of an ML training job start simultaneously or not at all — preventing the scenario where partial pod starts consume GPU quotas without making progress. Its queue-based fair-share scheduling distributes cluster resources equitably across teams. When the Volcano scheduler crashes, a queue enters a blocked state due to a resource over-commitment, or gang scheduling fails to start a job because one node can't satisfy affinity constraints, jobs silently accumulate in Pending with no timeout or alerting mechanism. Vigilmon gives you external visibility into Volcano's scheduler health, queue state, job completion rates, and pod liveness so batch workload failures surface within minutes.

What You'll Set Up

  • Volcano scheduler and controller availability
  • Queue depth and job backlog monitoring
  • Job completion rate and stuck-job detection
  • Gang scheduling success/failure tracking
  • Resource quota and fair-share drift

Prerequisites

  • Volcano 1.7+ installed on Kubernetes
  • kubectl access to the cluster with permissions to read vcjobs, queues, and podgroups
  • The Volcano webhook and controller running in volcano-system namespace
  • A free Vigilmon account

Step 1: Monitor Volcano Scheduler and Controller Availability

Volcano runs three core components: the scheduler (assigns pods to nodes), the controller (manages VcJob lifecycle), and the admission webhook (validates Job and Queue resources). If any of these go down, jobs stop scheduling and new submissions are rejected. Monitor component health via the Kubernetes API:

  1. In Vigilmon, click Add Monitor → Cron Heartbeat.
  2. Set Expected ping interval to 5 minutes.
  3. Copy the heartbeat URL (e.g., https://vigilmon.online/heartbeat/abc123).
#!/bin/bash
# /usr/local/bin/volcano-component-check.sh

HEARTBEAT_URL="https://vigilmon.online/heartbeat/abc123"
NAMESPACE="volcano-system"
EXPECTED_COMPONENTS=("volcano-scheduler" "volcano-controller-manager" "volcano-admission")

ALL_HEALTHY=true

for COMPONENT in "${EXPECTED_COMPONENTS[@]}"; do
  READY=$(kubectl get deployment "$COMPONENT" \
    -n "$NAMESPACE" \
    -o jsonpath='{.status.readyReplicas}' 2>/dev/null)
  DESIRED=$(kubectl get deployment "$COMPONENT" \
    -n "$NAMESPACE" \
    -o jsonpath='{.spec.replicas}' 2>/dev/null)

  if [ -z "$READY" ] || [ "$READY" -lt 1 ]; then
    echo "UNHEALTHY: ${COMPONENT} has ${READY:-0}/${DESIRED:-?} ready replicas"
    ALL_HEALTHY=false
  else
    echo "OK: ${COMPONENT} ${READY}/${DESIRED} ready"
  fi
done

if [ "$ALL_HEALTHY" = true ]; then
  echo "All Volcano components healthy"
  curl -s "$HEARTBEAT_URL"
else
  echo "One or more Volcano components are unhealthy"
  exit 1
fi

Also monitor the Volcano scheduler's metrics endpoint for direct readiness:

# Volcano scheduler exposes Prometheus metrics at :8080/metrics when running
SCHEDULER_POD=$(kubectl get pod -n volcano-system -l app=volcano-scheduler -o jsonpath='{.items[0].metadata.name}' 2>/dev/null)
if kubectl exec -n volcano-system "$SCHEDULER_POD" -- \
  wget -qO- http://localhost:8080/metrics 2>/dev/null | grep -q "volcano_scheduler"; then
  echo "Volcano scheduler metrics endpoint healthy"
fi

Step 2: Monitor Queue Depth and Job Backlog

Volcano's queue model organizes jobs by team or project with configurable weights for fair-share scheduling. A healthy queue has jobs entering, being scheduled, completing, and clearing. A queue whose depth only grows — because the cluster is over-subscribed, resource requests are too high, or affinity constraints are unsatisfiable — means jobs silently accumulate with no completion. Monitor queue state directly:

#!/bin/bash
# /usr/local/bin/volcano-queue-check.sh

HEARTBEAT_URL="https://vigilmon.online/heartbeat/def456"
MAX_PENDING_JOBS=50        # Alert if any queue has more than 50 pending jobs
MAX_QUEUE_AGE_HOURS=6      # Alert if oldest pending job has waited more than 6 hours

# Get all queues and their current job counts
QUEUES=$(kubectl get queues.scheduling.volcano.sh -o json 2>/dev/null)

if [ -z "$QUEUES" ]; then
  echo "Could not fetch Volcano queues — is Volcano installed?"
  exit 1
fi

QUEUE_STATUS=$(echo "$QUEUES" | python3 -c "
import sys, json

data = json.load(sys.stdin)
items = data.get('items', [])
issues = []

for q in items:
    name = q['metadata']['name']
    status = q.get('status', {})
    pending = status.get('pending', 0)
    running = status.get('running', 0)
    state = status.get('state', 'Unknown')

    print(f'Queue {name}: state={state}, pending={pending}, running={running}')

    if state not in ['Open']:
        issues.append(f'Queue {name} is in state {state} (expected Open)')
    if pending > ${MAX_PENDING_JOBS}:
        issues.append(f'Queue {name} has {pending} pending jobs (max: ${MAX_PENDING_JOBS})')

for issue in issues:
    print(f'ISSUE: {issue}', file=sys.stderr)

sys.exit(1 if issues else 0)
" 2>&1)

QUEUE_EXIT=$?

if [ "$QUEUE_EXIT" -eq 0 ]; then
  echo "Volcano queues healthy"
  curl -s "$HEARTBEAT_URL"
else
  echo "Volcano queue issues detected:"
  echo "$QUEUE_STATUS"
  exit 1
fi

For job age monitoring, check the age of the oldest pending VcJob:

#!/bin/bash
# Check for VcJobs stuck in pending for too long

MAX_PENDING_MINUTES=120   # 2 hours

kubectl get vcjob -A -o json 2>/dev/null | python3 -c "
import sys, json
from datetime import datetime, timezone

data = json.load(sys.stdin)
items = data.get('items', [])
now = datetime.now(timezone.utc)
stuck = []

for job in items:
    phase = job.get('status', {}).get('state', {}).get('phase', '')
    if phase in ['Pending']:
        created = job['metadata'].get('creationTimestamp', '')
        if created:
            try:
                ct = datetime.fromisoformat(created.replace('Z', '+00:00'))
                age_min = (now - ct).total_seconds() / 60
                if age_min > ${MAX_PENDING_MINUTES}:
                    name = job['metadata']['name']
                    ns = job['metadata']['namespace']
                    stuck.append(f'{ns}/{name} (pending {age_min:.0f} min)')
            except:
                pass

if stuck:
    print('Stuck jobs: ' + ', '.join(stuck))
    sys.exit(1)
else:
    print('No stuck pending jobs')
    sys.exit(0)
"

Step 3: Track Job Completion Rate

Volcano jobs complete (or fail) as a batch — the entire gang succeeds or fails together. A job stuck in Running phase where some pods have exited but the VcJob hasn't transitioned to Completed or Failed indicates a Volcano controller issue. Monitor job completion rate to detect pipeline stalls:

#!/bin/bash
# /usr/local/bin/volcano-job-completion.sh

HEARTBEAT_URL="https://vigilmon.online/heartbeat/ghi789"
NAMESPACE="ml-workloads"     # Adjust to your workload namespace
MAX_STALLED_MINUTES=30       # Alert if a job's Running phase stalls without pod changes

# Get recently submitted jobs and their completion status
JOBS=$(kubectl get vcjob -n "$NAMESPACE" -o json 2>/dev/null)

STALLED=$(echo "$JOBS" | python3 -c "
import sys, json
from datetime import datetime, timezone

data = json.load(sys.stdin)
items = data.get('items', [])
now = datetime.now(timezone.utc)
stalled = 0

for job in items:
    phase = job.get('status', {}).get('state', {}).get('phase', '')
    name = job['metadata']['name']

    # Jobs in terminal states are fine
    if phase in ['Completed', 'Failed', 'Terminated', 'Aborted']:
        continue

    # Check for Running jobs where lastTransitionTime is very old
    conditions = job.get('status', {}).get('conditions', [])
    for cond in conditions:
        if cond.get('type') == 'Running' and cond.get('status') == 'True':
            last_time = cond.get('lastTransitionTime', '')
            if last_time:
                try:
                    lt = datetime.fromisoformat(last_time.replace('Z', '+00:00'))
                    age_min = (now - lt).total_seconds() / 60
                    if age_min > ${MAX_STALLED_MINUTES}:
                        succeeded = job.get('status', {}).get('succeeded', 0)
                        total = job.get('spec', {}).get('tasks', [{}])[0].get('replicas', 1)
                        if succeeded < total:
                            print(f'Stalled job: {name} (Running for {age_min:.0f} min, {succeeded}/{total} pods succeeded)', file=sys.stderr)
                            stalled += 1
                except:
                    pass

print(stalled)
" 2>&1 | tail -1)

if [ "${STALLED:-0}" -eq 0 ]; then
  echo "Volcano job completion healthy — no stalled jobs"
  curl -s "$HEARTBEAT_URL"
else
  echo "Found ${STALLED} stalled Volcano job(s)"
  exit 1
fi

Step 4: Monitor Gang Scheduling Success Rate

Volcano's gang scheduling is all-or-nothing: a job only starts when minAvailable pods can be scheduled simultaneously. When cluster fragmentation prevents gang scheduling from succeeding — e.g., a large training job requiring 16 GPUs across 2 nodes when only single-GPU gaps remain — the job stays in Pending indefinitely. Track gang scheduling failures:

#!/bin/bash
# /usr/local/bin/volcano-gang-scheduling.sh

HEARTBEAT_URL="https://vigilmon.online/heartbeat/jkl012"
MAX_UNSCHEDULABLE_MINUTES=45   # Alert if gang can't be scheduled for 45+ minutes

# Check PodGroups (Volcano's gang scheduling primitive)
kubectl get podgroups.scheduling.volcano.sh -A -o json 2>/dev/null | python3 -c "
import sys, json
from datetime import datetime, timezone

data = json.load(sys.stdin)
items = data.get('items', [])
now = datetime.now(timezone.utc)
unschedulable = []

for pg in items:
    phase = pg.get('status', {}).get('phase', '')
    name = pg['metadata']['name']
    ns = pg['metadata']['namespace']

    if phase == 'Pending':
        conditions = pg.get('status', {}).get('conditions', [])
        for cond in conditions:
            if cond.get('reason') in ['Unschedulable', 'NotEnoughResources']:
                created = pg['metadata'].get('creationTimestamp', '')
                if created:
                    try:
                        ct = datetime.fromisoformat(created.replace('Z', '+00:00'))
                        age_min = (now - ct).total_seconds() / 60
                        if age_min > ${MAX_UNSCHEDULABLE_MINUTES}:
                            min_avail = pg.get('spec', {}).get('minMember', '?')
                            unschedulable.append(f'{ns}/{name} (unschedulable for {age_min:.0f} min, minMember={min_avail})')
                    except:
                        pass

if unschedulable:
    print('Unschedulable PodGroups: ' + '; '.join(unschedulable))
    sys.exit(1)
else:
    print('All PodGroups schedulable')
    sys.exit(0)
" && curl -s "$HEARTBEAT_URL"

A PodGroup that stays Unschedulable for longer than your maximum job wait time almost always indicates one of: cluster resource exhaustion (request kubectl describe nodes to verify), GPU fragmentation (too many small allocations blocking large contiguous requests), or a mis-set minAvailable that exceeds cluster capacity.

Step 5: Monitor Resource Quota Fair-Share Drift

Volcano's queue weights determine how cluster resources are allocated when demand exceeds supply. Over time, teams may accumulate running jobs that consume far more than their fair share — a long-running training job started when the cluster was idle can hold resources for hours while other queues starve. Monitor queue resource consumption against their weights:

#!/bin/bash
# /usr/local/bin/volcano-fairness-check.sh

HEARTBEAT_URL="https://vigilmon.online/heartbeat/mno345"
MAX_FAIRNESS_RATIO=3.0   # Alert if any queue consumes >3x its fair share

kubectl get queues.scheduling.volcano.sh -o json 2>/dev/null | python3 -c "
import sys, json

data = json.load(sys.stdin)
items = data.get('items', [])

total_weight = sum(q.get('spec', {}).get('weight', 1) for q in items)
if total_weight == 0:
    print('No queues found')
    sys.exit(0)

issues = []
for q in items:
    name = q['metadata']['name']
    weight = q.get('spec', {}).get('weight', 1)
    fair_share = weight / total_weight

    allocated = q.get('status', {}).get('allocated', {})
    # Use CPU as proxy for overall allocation
    cpu_allocated = allocated.get('cpu', '0')
    # (simplified — in production, compare to cluster total CPU)
    print(f'Queue {name}: weight={weight}, fair_share={fair_share:.2%}, allocated_cpu={cpu_allocated}')

if issues:
    print('Fair-share issues: ' + '; '.join(issues))
    sys.exit(1)
else:
    sys.exit(0)
" && curl -s "$HEARTBEAT_URL"

Step 6: Set Up Alert Channels

Configure Vigilmon to route Volcano alerts to the right teams:

  1. Go to Alert Channels in Vigilmon.
  2. Add a Slack webhook to #ml-platform for all Volcano monitors — batch job failures affect all ML teams.
  3. Add PagerDuty for the component availability monitor — a crashed Volcano scheduler means no new training jobs start.
  4. Add email for queue depth and stuck job monitors — these are serious but not immediate on-call incidents.
  5. Set Consecutive failures before alert to 2 for queue depth monitors — brief scheduling waves during burst periods are normal.

Include job context in Slack notifications:

ALERT_MSG=":rotating_light: Volcano: *${STALLED}* training job(s) stalled in namespace ${NAMESPACE}. Check: kubectl get vcjob -n ${NAMESPACE}"

curl -X POST https://hooks.slack.com/services/YOUR/WEBHOOK/URL \
  -H 'Content-type: application/json' \
  --data "{\"text\": \"$ALERT_MSG\", \"channel\": \"#ml-platform\"}"

Add maintenance windows during Volcano upgrades:

curl -X POST https://vigilmon.online/api/maintenance \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -d '{
    "monitor_id": "VOLCANO_SCHEDULER_MONITOR_ID",
    "duration_minutes": 20,
    "reason": "Volcano upgrade to 1.9.x — scheduler restart required"
  }'

Summary

| Monitor | Target | What It Catches | |---|---|---| | Cron heartbeat (components) | Deployment readyReplicas | Scheduler/controller crash, admission webhook failure | | Cron heartbeat (queues) | Queue state and pending depth | Queue deadlock, resource starvation, over-submission | | Cron heartbeat (jobs) | VcJob phase transition staleness | Controller hang, pod completion not propagated | | Cron heartbeat (gang scheduling) | PodGroup unschedulable age | Cluster fragmentation, GPU exhaustion, capacity issues | | Cron heartbeat (fair-share) | Queue weight vs allocation | Resource hoarding, fair-share drift, queue starvation |

Volcano's value — gang scheduling, queue fairness, batch lifecycle management — is entirely contingent on its control plane staying healthy. When the scheduler or controller goes down, training jobs silently queue; when fair-share drifts, teams experience inexplicable slowdowns. Vigilmon gives you external visibility into every operational dimension of Volcano so ML platform incidents surface quickly rather than through frustrated messages in #why-is-my-training-job-stuck.

Start monitoring for free at vigilmon.online.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles