tutorial

Monitoring Tdarr (Distributed Video Transcoding) with Vigilmon

Tdarr manages distributed video transcoding across worker nodes — but web server crashes, worker node disconnections, GPU driver failures, and plugin execution errors silently stall your transcode queue. Here's how to monitor Tdarr web uptime, worker health, queue depth, resource utilization, and plugin execution with Vigilmon.

Tdarr is a distributed video transcoding platform consisting of a central server (Node.js, port 8265 web dashboard + port 8266 API) and one or more worker nodes that execute FFmpeg-based transcoding jobs using CPU, GPU, or both. When the Tdarr server goes down, workers disconnect and queue processing halts. When a worker node crashes mid-transcode, its in-progress job is lost and re-queued automatically — but if all workers crash, the queue grows indefinitely without any alert. GPU driver failures cause workers to silently fall back to CPU (if configured) or fail jobs entirely, often at much slower speeds. Plugin execution errors abort individual items with no re-queue, meaning affected files stay in an error state until manually cleared. Without external monitoring, a Tdarr deployment can be "running" while hundreds of transcodes are backed up or failing. Vigilmon gives you coverage for Tdarr web server availability, worker node connectivity, queue depth, resource saturation, and plugin error rates.

What You'll Set Up

  • Tdarr web server and API uptime (ports 8265 and 8266)
  • Worker node health monitoring
  • Transcode queue depth tracking
  • GPU and CPU resource utilization
  • Plugin execution error alerting

Prerequisites

  • Tdarr server running (Node.js, accessible on ports 8265/8266)
  • One or more Tdarr worker nodes connected to the server
  • Tdarr API accessible (HTTP on port 8266 by default)
  • A free Vigilmon account

Step 1: Monitor Tdarr Web Server and API Uptime

Tdarr exposes two HTTP interfaces: the web dashboard (port 8265) and the server API (port 8266). Both must be running for workers to connect and for you to manage the queue. Add direct HTTP monitors in Vigilmon for both:

Web dashboard monitor:

  1. Click Add Monitor → HTTP(S).
  2. Set URL to http://your-tdarr-host:8265/.
  3. Set check interval to 2 minutes.
  4. Set expected response to HTTP 200.

Server API monitor:

  1. Click Add Monitor → HTTP(S).
  2. Set URL to http://your-tdarr-host:8266/api/v2/status.
  3. Set check interval to 2 minutes.
  4. Set expected response to HTTP 200.

For script-based monitoring (useful when Vigilmon can't reach your Tdarr host directly):

#!/bin/bash
# /usr/local/bin/tdarr-uptime-check.sh

WEB_HEARTBEAT_URL="https://vigilmon.online/heartbeat/abc123"
API_HEARTBEAT_URL="https://vigilmon.online/heartbeat/def456"
TDARR_HOST="localhost"
TIMEOUT=10

# Check web dashboard
WEB_CODE=$(curl -s -o /dev/null -w "%{http_code}" \
  --max-time "$TIMEOUT" \
  "http://${TDARR_HOST}:8265/")
if [ "$WEB_CODE" = "200" ] || [ "$WEB_CODE" = "302" ]; then
  echo "Tdarr web UI OK (HTTP $WEB_CODE)"
  curl -s "$WEB_HEARTBEAT_URL"
else
  echo "Tdarr web UI down (HTTP $WEB_CODE)"
fi

# Check server API
API_RESP=$(curl -s \
  --max-time "$TIMEOUT" \
  "http://${TDARR_HOST}:8266/api/v2/status")
if echo "$API_RESP" | python3 -c "import sys,json; d=json.load(sys.stdin); sys.exit(0 if d.get('status') else 1)" 2>/dev/null; then
  echo "Tdarr API OK"
  curl -s "$API_HEARTBEAT_URL"
else
  echo "Tdarr API down or returning unexpected response"
fi

Run this every 5 minutes. Tdarr server restarts take 15–30 seconds, so two consecutive failures confirms a real outage rather than a restart blip.

Step 2: Monitor Worker Node Health

Worker nodes connect to the Tdarr server via WebSocket and register themselves as available for jobs. When a worker disconnects (container restart, host crash, network partition), jobs assigned to it are eventually re-queued but this can take several minutes and may produce duplicate output files. Monitor worker connectivity through the Tdarr API:

#!/bin/bash
# /usr/local/bin/tdarr-workers-check.sh

HEARTBEAT_URL="https://vigilmon.online/heartbeat/ghi789"
TDARR_API="http://localhost:8266"
MIN_WORKERS=1       # Alert if fewer than this many workers are connected
TIMEOUT=15

# Get worker list from Tdarr API
WORKERS_RESP=$(curl -s \
  --max-time "$TIMEOUT" \
  "${TDARR_API}/api/v2/get-nodes")

if [ -z "$WORKERS_RESP" ]; then
  echo "No response from Tdarr API"
  exit 1
fi

WORKER_RESULT=$(echo "$WORKERS_RESP" | python3 -c "
import sys, json

try:
    data = json.load(sys.stdin)
except Exception as e:
    print(f'JSON parse error: {e}')
    sys.exit(1)

# Tdarr API returns nodes as an object keyed by node ID
nodes = data if isinstance(data, dict) else {}
connected = []
disconnected = []

for node_id, node_info in nodes.items():
    if isinstance(node_info, dict):
        status = node_info.get('nodePaused', True)
        name = node_info.get('nodeName', node_id)
        if not status:
            connected.append(name)
        else:
            disconnected.append(name)

print(f'connected:{len(connected)}')
print(f'disconnected:{len(disconnected)}')
for n in disconnected:
    print(f'offline:{n}')
" 2>/dev/null)

if [ -z "$WORKER_RESULT" ]; then
  echo "Failed to parse worker status"
  exit 1
fi

CONNECTED=$(echo "$WORKER_RESULT" | grep "^connected:" | cut -d: -f2)
CONNECTED="${CONNECTED:-0}"

if [ "$CONNECTED" -lt "$MIN_WORKERS" ]; then
  echo "Worker count below threshold: ${CONNECTED} connected (min: ${MIN_WORKERS})"
  echo "$WORKER_RESULT" | grep "^offline:" | sed 's/offline:/  Offline: /'
  exit 1
fi

echo "Workers OK: ${CONNECTED} connected"
echo "$WORKER_RESULT" | grep "^offline:" | sed 's/offline:/  Offline: /'
curl -s "$HEARTBEAT_URL"

Set the Vigilmon heartbeat to 5 minutes. Set MIN_WORKERS to the minimum number of workers required for your queue to make progress — typically 1 for a home setup, higher for a multi-node deployment.

Step 3: Monitor Transcode Queue Depth

The Tdarr queue accumulates files waiting for transcoding. When workers are too slow, too few, or down entirely, the queue depth grows. A growing queue signals capacity problems before individual workers start timing out. Track queue depth through the API:

#!/bin/bash
# /usr/local/bin/tdarr-queue-check.sh

HEARTBEAT_URL="https://vigilmon.online/heartbeat/jkl012"
TDARR_API="http://localhost:8266"
MAX_QUEUE_DEPTH=500    # Alert if more than 500 items are queued
MAX_ERROR_COUNT=50     # Alert if more than 50 items are in error state
TIMEOUT=15

STATS_RESP=$(curl -s \
  --max-time "$TIMEOUT" \
  "${TDARR_API}/api/v2/cruddb" \
  -X POST \
  -H "Content-Type: application/json" \
  -d '{
    "data": {
      "collection": "StatisticsJSONDB",
      "mode": "getAll",
      "docID": "statistics"
    }
  }')

if [ -z "$STATS_RESP" ]; then
  echo "No response from Tdarr statistics API"
  exit 1
fi

QUEUE_RESULT=$(echo "$STATS_RESP" | python3 -c "
import sys, json

try:
    data = json.load(sys.stdin)
except:
    sys.exit(1)

# Extract queue statistics
doc = data if isinstance(data, dict) else {}

queued = doc.get('totalTranscodeCount', 0) + doc.get('totalHealthCheckCount', 0)
errors = doc.get('totalErrorCount', 0)
processing = doc.get('totalTranscodeCount', 0)

print(f'queued:{queued}')
print(f'errors:{errors}')
print(f'processing:{processing}')
" 2>/dev/null)

QUEUED=$(echo "$QUEUE_RESULT" | grep "^queued:" | cut -d: -f2)
ERRORS=$(echo "$QUEUE_RESULT" | grep "^errors:" | cut -d: -f2)
QUEUED="${QUEUED:-0}"
ERRORS="${ERRORS:-0}"

FAILED=0

if [ "$QUEUED" -gt "$MAX_QUEUE_DEPTH" ]; then
  echo "Queue backed up: ${QUEUED} items waiting (max: ${MAX_QUEUE_DEPTH})"
  FAILED=1
fi

if [ "$ERRORS" -gt "$MAX_ERROR_COUNT" ]; then
  echo "Error count high: ${ERRORS} items in error state (max: ${MAX_ERROR_COUNT})"
  FAILED=1
fi

if [ "$FAILED" -eq 0 ]; then
  echo "Queue OK: ${QUEUED} queued, ${ERRORS} errors"
  curl -s "$HEARTBEAT_URL"
else
  exit 1
fi

Run this every 10 minutes. Queue depth spikes that don't recover within 30 minutes indicate a systemic issue — all workers down, a blocking plugin error, or a misconfigured flow that keeps re-queuing the same files.

Step 4: Monitor GPU and CPU Resource Utilization

Tdarr workers can transcode using CPU (FFmpeg software encoding), GPU (NVIDIA NVENC/NVDEC, Intel QSV, AMD VCE), or a mixture. GPU driver crashes, CUDA out-of-memory errors, and hardware encoder saturation cause workers to fail jobs silently or fall back to software encoding at much lower throughput. Monitor resource utilization on worker nodes:

#!/bin/bash
# /usr/local/bin/tdarr-resources-check.sh

HEARTBEAT_URL="https://vigilmon.online/heartbeat/mno345"
CPU_THRESHOLD=95      # percent
GPU_MEM_THRESHOLD=90  # percent VRAM usage
DISK_THRESHOLD=90     # percent disk used (transcode temp directory)
TEMP_DIR="/mnt/transcode-cache"   # Adjust for your temp dir

FAILED=0

# CPU utilization
CPU_IDLE=$(vmstat 1 2 | tail -1 | awk '{print $15}')
CPU_USED=$(( 100 - CPU_IDLE ))

if [ "$CPU_USED" -ge "$CPU_THRESHOLD" ]; then
  echo "CPU saturated: ${CPU_USED}% (threshold: ${CPU_THRESHOLD}%)"
  FAILED=1
else
  echo "CPU OK: ${CPU_USED}%"
fi

# GPU utilization (NVIDIA)
if command -v nvidia-smi &>/dev/null; then
  GPU_MEM_USED=$(nvidia-smi --query-gpu=memory.used --format=csv,noheader,nounits | head -1 | tr -d ' ')
  GPU_MEM_TOTAL=$(nvidia-smi --query-gpu=memory.total --format=csv,noheader,nounits | head -1 | tr -d ' ')
  GPU_MEM_PCT=$(( GPU_MEM_USED * 100 / GPU_MEM_TOTAL ))

  GPU_ENC=$(nvidia-smi --query-gpu=encoder.stats.sessionCount --format=csv,noheader,nounits 2>/dev/null | head -1 | tr -d ' ')
  GPU_DEC=$(nvidia-smi --query-gpu=decoder.stats.sessionCount --format=csv,noheader,nounits 2>/dev/null | head -1 | tr -d ' ')

  if [ "$GPU_MEM_PCT" -ge "$GPU_MEM_THRESHOLD" ]; then
    echo "GPU VRAM pressure: ${GPU_MEM_PCT}% used (${GPU_MEM_USED}MB/${GPU_MEM_TOTAL}MB)"
    FAILED=1
  else
    echo "GPU OK: ${GPU_MEM_PCT}% VRAM, ${GPU_ENC:-0} enc sessions, ${GPU_DEC:-0} dec sessions"
  fi

  # Check GPU driver error state
  GPU_STATE=$(nvidia-smi --query-gpu=ecc.errors.uncorrected.volatile.total --format=csv,noheader,nounits 2>/dev/null | head -1 | tr -d ' ')
  if [ -n "$GPU_STATE" ] && [ "$GPU_STATE" != "N/A" ] && [ "$GPU_STATE" -gt 0 ]; then
    echo "GPU ECC errors detected: ${GPU_STATE} uncorrected errors — hardware issue"
    FAILED=1
  fi
fi

# Disk utilization (transcode temp directory)
if [ -d "$TEMP_DIR" ]; then
  DISK_PCT=$(df "$TEMP_DIR" | awk 'NR==2 {gsub(/%/,""); print $5}')
  if [ "$DISK_PCT" -ge "$DISK_THRESHOLD" ]; then
    echo "Transcode temp disk full: ${DISK_PCT}% (threshold: ${DISK_THRESHOLD}%)"
    FAILED=1
  else
    echo "Disk OK: ${DISK_PCT}% used at $TEMP_DIR"
  fi
fi

if [ "$FAILED" -eq 0 ]; then
  curl -s "$HEARTBEAT_URL"
else
  exit 1
fi

Install this on each worker node with a 5 minute cron interval. Disk saturation in the transcode temp directory is the most common cause of transcode job failures — FFmpeg cannot write output files and fails the entire job.

Step 5: Monitor Plugin Execution Errors

Tdarr uses a plugin-based flow system to decide when and how to transcode each file. When a plugin crashes, produces invalid FFmpeg arguments, or encounters a codec it can't handle, the affected file enters an error state and is not automatically retried. A growing error count means files are being permanently skipped:

#!/bin/bash
# /usr/local/bin/tdarr-errors-check.sh

HEARTBEAT_URL="https://vigilmon.online/heartbeat/pqr678"
TDARR_API="http://localhost:8266"
MAX_NEW_ERRORS=10     # Alert if more than 10 new errors since last check
STATE_FILE="/var/run/tdarr-error-count.txt"
TIMEOUT=15

# Get current error count
CURRENT_ERRORS=$(curl -s \
  --max-time "$TIMEOUT" \
  "${TDARR_API}/api/v2/cruddb" \
  -X POST \
  -H "Content-Type: application/json" \
  -d '{
    "data": {
      "collection": "StatisticsJSONDB",
      "mode": "getAll",
      "docID": "statistics"
    }
  }' | python3 -c "
import sys, json
data = json.load(sys.stdin)
print(data.get('totalErrorCount', 0))
" 2>/dev/null || echo 0)

CURRENT_ERRORS="${CURRENT_ERRORS:-0}"

# Compare with previous count
if [ -f "$STATE_FILE" ]; then
  PREV_ERRORS=$(cat "$STATE_FILE")
  NEW_ERRORS=$(( CURRENT_ERRORS - PREV_ERRORS ))

  if [ "$NEW_ERRORS" -gt "$MAX_NEW_ERRORS" ]; then
    echo "Plugin error spike: ${NEW_ERRORS} new errors since last check (total: ${CURRENT_ERRORS})"
    echo "$CURRENT_ERRORS" > "$STATE_FILE"
    exit 1
  else
    echo "Error rate OK: ${NEW_ERRORS} new errors, ${CURRENT_ERRORS} total"
  fi
else
  echo "First run: current error count is ${CURRENT_ERRORS}"
fi

echo "$CURRENT_ERRORS" > "$STATE_FILE"
curl -s "$HEARTBEAT_URL"

Run this every 30 minutes. A sudden spike in plugin errors (10+ new errors in one check interval) typically indicates a bad plugin update or a batch of files in an unsupported format that the flow doesn't handle gracefully.

Step 6: Set Up Alert Channels

Configure Vigilmon alert routing for Tdarr:

  1. Go to Alert Channels in Vigilmon.
  2. Add a Slack webhook to #media-ops for server downtime and worker disconnection alerts — these stop all transcoding immediately.
  3. Add email notification for queue depth and error rate alerts — these are capacity and quality issues, not emergencies.
  4. Add PagerDuty only for the web server monitor if Tdarr is a production media delivery pipeline (not required for home use).
  5. Set Consecutive failures before alert to 2 for resource monitors — brief GPU memory spikes during encode session startup are normal.

Add maintenance windows during Tdarr upgrades:

curl -X POST https://vigilmon.online/api/maintenance \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -d '{
    "monitor_id": "TDARR_WEB_MONITOR_ID",
    "duration_minutes": 10,
    "reason": "Tdarr server upgrade"
  }'

docker compose pull tdarr && docker compose up -d tdarr

Summary

| Monitor | Target | What It Catches | |---|---|---| | HTTP monitor (web) | Port 8265 | Web server crash, Node.js OOM | | HTTP monitor (API) | Port 8266 /api/v2/status | API process failure, crash loop | | Cron heartbeat (workers) | /api/v2/get-nodes | Worker disconnection, node crash | | Cron heartbeat (queue) | Statistics API queue depth | Capacity exhaustion, blocked queue | | Cron heartbeat (resources) | CPU / GPU VRAM / disk | Hardware saturation, driver errors, temp disk full | | Cron heartbeat (errors) | Error count delta | Plugin crashes, codec incompatibilities, flow bugs |

Tdarr's failure mode is a frozen queue — the dashboard shows workers as "connected" and the server as "running" while the transcode backlog grows because all workers are stuck on hanging FFmpeg processes or GPU driver errors. Vigilmon's external monitoring surfaces these failures within minutes, before your queue depth becomes a multi-day recovery project.

Start monitoring for free at vigilmon.online.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles