Monitoring Your Restreamer Instance with Vigilmon

Restreamer is a self-hosted live streaming server: receive an RTMP or SRT stream and simultaneously push it to YouTube, Twitch, Facebook Live, and any other RTMP destination. It runs on Go + FFmpeg and exposes a web UI and API on port 8080, with RTMP ingest on port 1935 and SRT ingest on port 6000.

Live streaming has zero tolerance for silent failures. If the RTMP ingest server goes down, your streaming software can't connect and the stream simply doesn't start — no error to the viewer, just silence. If FFmpeg transcoding stalls, the stream freezes. If an output destination loses connectivity, you're broadcasting to nowhere.

This tutorial sets up production monitoring for Restreamer using Vigilmon:

• RTMP and SRT ingest server availability
• Web UI and API server health
• FFmpeg transcoding process health
• Active stream stall detection
• Output destination connectivity
• Stream metadata API monitoring
• Disk space for recording storage
• Scheduled stream start/stop job heartbeats
• TLS certificate expiry
• Slack alerting

Step 1: Verify the web API is healthy

Restreamer exposes a REST API on port 8080. This is both the management interface and the health surface you'll monitor. Check the API status endpoint:

curl http://localhost:8080/api/v3/ping
# {"message":"pong"}

Restreamer v2+ exposes /api/v3/ping — a lightweight liveness endpoint. If you're running an older version, check the API docs for your version's equivalent.

For a richer health check that includes the FFmpeg process status and active streams:

# Get the list of processes (each stream is a "process" in Restreamer's model)
curl -s http://localhost:8080/api/v3/process \
  -H "Authorization: Bearer $RESTREAMER_TOKEN"

Step 2: Monitor the web UI and API server

Point Vigilmon at the ping endpoint:

1. Sign up at vigilmon.online
2. Click New Monitor → HTTP
3. URL: http://restreamer.yourdomain.com:8080/api/v3/ping (or https:// if behind a TLS-terminating proxy)
4. Check interval: 1 minute
5. Expected status: 200
6. Keyword check: pong
7. Save

The keyword check on pong ensures you're getting the actual API response, not a proxy error page that happens to return 200.

Step 3: Monitor RTMP ingest server availability (port 1935)

The RTMP ingest port (1935) is what your streaming software connects to. If it's unreachable, the stream can't start.

In Vigilmon:

1. Click New Monitor → TCP
2. Host: restreamer.yourdomain.com
3. Port: 1935
4. Check interval: 1 minute
5. Save

A TCP monitor sends a connection attempt and verifies the port accepts connections. It doesn't verify that Restreamer is processing streams — just that the RTMP server is listening. Combine this with the FFmpeg process monitor in Step 5 for full coverage.

Step 4: Monitor SRT ingest server availability (port 6000)

SRT (Secure Reliable Transport) is the alternative ingest protocol — lower latency and more reliable on unstable networks than RTMP. If you accept SRT streams, monitor port 6000:

In Vigilmon:

1. Click New Monitor → TCP
2. Host: restreamer.yourdomain.com
3. Port: 6000
4. Check interval: 1 minute
5. Save

Note: SRT uses UDP, and most TCP monitors use TCP connection attempts. If your Vigilmon plan supports UDP monitoring, use that instead. Otherwise, monitor the SRT port via a synthetic probe script that uses an SRT client library.

For a UDP-aware probe:

#!/bin/bash
# probe-srt-port.sh
# Use socat to attempt a UDP connection and check response
timeout 5 socat -u UDP:restreamer.yourdomain.com:6000,connect-timeout=3 /dev/null 2>/dev/null
if [ $? -ne 124 ]; then  # 124 = timeout, anything else = port responded
  curl -s "$VIGILMON_SRT_HEARTBEAT_URL" > /dev/null
fi

Schedule this every 2 minutes and pair it with a Vigilmon heartbeat monitor.

Step 5: Monitor FFmpeg transcoding process health

FFmpeg is the core engine — it does the actual re-encoding and stream pushing. If FFmpeg dies or stalls, the visual stream freezes even though all the server ports remain open.

Restreamer tracks each stream as a "process" with a state. Use the process API to check FFmpeg status:

#!/bin/bash
# probe-ffmpeg-health.sh
RESPONSE=$(curl -s \
  -H "Authorization: Bearer $RESTREAMER_TOKEN" \
  http://localhost:8080/api/v3/process)

# Check that the active stream process is in 'running' state
STATE=$(echo "$RESPONSE" | jq -r '.[0].state.exec // "unknown"')

if [ "$STATE" = "running" ]; then
  curl -s "$VIGILMON_FFMPEG_HEARTBEAT_URL" > /dev/null
else
  echo "FFmpeg process state: $STATE — not pinging heartbeat"
fi

In Vigilmon:

1. Click New Monitor → Heartbeat
2. Name: Restreamer FFmpeg Process
3. Expected interval: 2 minutes
4. Grace period: 3 minutes (allow a brief stall during stream transitions)
5. Copy the ping URL → set as VIGILMON_FFMPEG_HEARTBEAT_URL
6. Save

Schedule the probe script every 2 minutes via cron on the Restreamer host.

Step 6: Active stream health — stall detection

A stream can be technically "running" in Restreamer's process model but stalled — the input is frozen (no new video frames arriving) but FFmpeg hasn't detected it as an error yet. This is the worst user experience: viewers see a frozen frame with no indication that anything is wrong.

Restreamer's process API exposes frame counters. Monitor them for changes:

#!/usr/bin/env python3
# probe-stream-activity.py
import requests
import os
import time
import json

RESTREAMER_URL = os.environ["RESTREAMER_URL"]
TOKEN = os.environ["RESTREAMER_TOKEN"]
HEARTBEAT_URL = os.environ["VIGILMON_STREAM_ACTIVITY_HEARTBEAT_URL"]
STATE_FILE = "/tmp/restreamer-probe-state.json"

headers = {"Authorization": f"Bearer {TOKEN}"}

try:
    resp = requests.get(f"{RESTREAMER_URL}/api/v3/process", headers=headers, timeout=10)
    processes = resp.json()

    if not processes:
        # No active streams — don't alert (might be off-air by design)
        exit(0)

    process = processes[0]
    current_frames = process.get("state", {}).get("progress", {}).get("frame", 0)

    # Compare with previous reading
    state = {}
    if os.path.exists(STATE_FILE):
        with open(STATE_FILE) as f:
            state = json.load(f)

    prev_frames = state.get("frames", -1)

    if prev_frames == -1 or current_frames > prev_frames:
        # Frames are advancing — stream is live
        requests.get(HEARTBEAT_URL, timeout=5)

    # Store current frame count
    with open(STATE_FILE, "w") as f:
        json.dump({"frames": current_frames, "ts": time.time()}, f)

except Exception as e:
    print(f"Probe error: {e}")
    exit(1)

In Vigilmon:

1. Click New Monitor → Heartbeat
2. Name: Restreamer Stream Activity
3. Expected interval: 2 minutes
4. Grace period: 4 minutes (allow a 2-probe miss before alerting)
5. Save

Schedule the probe every 2 minutes. If frames stop advancing, the heartbeat misses and you get an alert — within 6 minutes of the stall.

Step 7: Monitor output destination connectivity

Each configured output (YouTube, Twitch, custom RTMP) is a separate potential failure point. Restreamer can be receiving input fine while one or more outputs are failing.

For YouTube and Twitch, monitor their RTMP ingest endpoints:

YouTube RTMP ingest:

In Vigilmon:

1. Click New Monitor → TCP
2. Host: a.rtmp.youtube.com
3. Port: 1935
4. Save

Twitch RTMP ingest:

1. Click New Monitor → TCP
2. Host: live.twitch.tv
3. Port: 1935
4. Save

For custom RTMP destinations, add a TCP monitor for each configured output server.

You can also use Restreamer's process API to check the output states:

#!/bin/bash
# probe-outputs.sh
OUTPUTS=$(curl -s \
  -H "Authorization: Bearer $RESTREAMER_TOKEN" \
  http://localhost:8080/api/v3/process | \
  jq -r '.[0].state.progress.streams[].codec_type // empty')

# If we have video and audio codec types, output is streaming
if echo "$OUTPUTS" | grep -q "video"; then
  curl -s "$VIGILMON_OUTPUTS_HEARTBEAT_URL" > /dev/null
fi

Step 8: Monitor the stream metadata API

The /api/v3/process endpoint is Restreamer's primary operational API — your monitoring scripts, dashboards, and any automation tools use it. Monitor it directly:

In Vigilmon:

1. Click New Monitor → HTTP
2. URL: https://restreamer.yourdomain.com/api/v3/process
3. Expected status: 200 or 401 (if unauthenticated request is rejected)
4. Check interval: 1 minute
5. Save

A 401 confirms the API is alive and auth middleware is running. A 200 from an unauthed request would indicate misconfiguration. Either way, 5xx or timeout means the API layer has failed.

Step 9: Disk space monitoring for recording storage

If local recording is enabled, Restreamer writes video files to disk. A full disk silently stops the recording — and may crash FFmpeg if it can't write output.

Add a disk space probe:

#!/bin/bash
# probe-disk-space.sh
RECORDING_DIR="${RESTREAMER_RECORDING_DIR:-/data/restreamer/recordings}"
THRESHOLD_PCT=85

USED=$(df -h "$RECORDING_DIR" | awk 'NR==2 {print $5}' | tr -d '%')

if [ "$USED" -lt "$THRESHOLD_PCT" ]; then
  curl -s "$VIGILMON_DISK_HEARTBEAT_URL" > /dev/null
else
  echo "Disk usage at ${USED}% — above threshold, not pinging heartbeat"
fi

In Vigilmon:

1. Click New Monitor → Heartbeat
2. Name: Restreamer Disk Space
3. Expected interval: 15 minutes
4. Grace period: 5 minutes
5. Set VIGILMON_DISK_HEARTBEAT_URL env variable
6. Save

Schedule the probe every 15 minutes. If disk usage exceeds 85%, the heartbeat stops and you get an alert before the disk fills entirely.

Step 10: Heartbeat monitoring for scheduled stream start/stop jobs

If you use scheduled stream start/stop (cron-based automation to start FFmpeg at a specific time), add a heartbeat to each schedule trigger:

#!/bin/bash
# scheduled-stream-start.sh
# Trigger Restreamer to start a specific process
curl -s -X PUT \
  -H "Authorization: Bearer $RESTREAMER_TOKEN" \
  -H "Content-Type: application/json" \
  http://localhost:8080/api/v3/process/your-stream-id/command \
  -d '{"command":"start"}'

# Verify it started (check state)
sleep 5
STATE=$(curl -s \
  -H "Authorization: Bearer $RESTREAMER_TOKEN" \
  http://localhost:8080/api/v3/process/your-stream-id | \
  jq -r '.state.exec')

if [ "$STATE" = "running" ]; then
  curl -s "$VIGILMON_STREAM_START_HEARTBEAT_URL" > /dev/null
fi

In Vigilmon:

1. Click New Monitor → Heartbeat
2. Name: Restreamer Scheduled Stream Start
3. Expected interval: match your schedule (e.g. 24 hours for a daily stream)
4. Copy the ping URL → set as VIGILMON_STREAM_START_HEARTBEAT_URL
5. Save

Step 11: TLS certificate expiry monitoring

If you expose Restreamer through a TLS-terminating reverse proxy (nginx/Caddy), monitor the certificate:

In Vigilmon:

1. Go to New Monitor → SSL Certificate
2. Domain: restreamer.yourdomain.com
3. Alert thresholds: 30 days (warning) and 7 days (critical)
4. Save

An expired certificate will break the web UI and API access, preventing you from managing streams.

Step 12: Slack alerts

In Vigilmon, go to Notifications → New Channel → Slack and paste your Slack incoming webhook URL.

Enable Slack on all Restreamer monitors. The most critical alert patterns during a live stream:

🔴 TCP DOWN: restreamer.yourdomain.com:1935
RTMP ingest port unreachable
Detected: 1 minute ago

🔴 MISSED HEARTBEAT: Restreamer Stream Activity
Last ping: 8m ago (expected every 2m) — stream may be stalled

🔴 MISSED HEARTBEAT: Restreamer Disk Space
Last ping: 25m ago — recording disk may be full

The stream activity heartbeat miss is the most time-sensitive — a frozen stream during a live event needs immediate response.

What you've built

| What | How | |------|-----| | Web UI / API server | Vigilmon HTTP monitor → /api/v3/ping | | RTMP ingest (port 1935) | Vigilmon TCP monitor | | SRT ingest (port 6000) | Vigilmon TCP monitor + UDP probe heartbeat | | FFmpeg transcoding | Process API probe + heartbeat (2-minute interval) | | Stream stall detection | Frame counter probe + heartbeat (2-minute interval) | | YouTube RTMP output | TCP monitor → a.rtmp.youtube.com:1935 | | Twitch RTMP output | TCP monitor → live.twitch.tv:1935 | | Stream metadata API | HTTP monitor → /api/v3/process expecting 200/401 | | Recording disk space | Disk probe + heartbeat (15-minute interval) | | Scheduled stream start | Heartbeat per schedule event | | TLS certificate | SSL monitor, 30-day + 7-day alert | | Slack alerts | Vigilmon Slack notification channel |

Live streaming tolerates no silent failures. With this setup, Vigilmon catches RTMP port failures, FFmpeg stalls, output disconnections, and disk problems — all within minutes.

Next steps

• Add Vigilmon response time history to track API latency trends — spikes often precede FFmpeg crashes
• Set up a Vigilmon status page to share stream health with your production team during live events
• If you broadcast to multiple outputs simultaneously, add a per-destination TCP monitor for each RTMP endpoint
• Consider a Vigilmon keyword check on the process API that validates output count — catching cases where one destination drops while others remain active

Get started free at vigilmon.online.