tutorial

Monitoring Restate (Distributed Application Runtime) with Vigilmon

Restate is a durable execution runtime for distributed applications — but a crashed server, stalled invocation journal, or overwhelmed handler pool silently breaks the durable guarantees your application depends on. Here's how to monitor Restate server health, invocation queue depth, journal storage, handler latency, and cluster connectivity with Vigilmon.

Restate is an open-source durable execution runtime that gives distributed applications transactional semantics without distributed transactions. It persists every service invocation to a journal, retries failed calls automatically, and provides exactly-once execution guarantees across service boundaries. Applications register handlers with the Restate server, which orchestrates invocations, manages state, and drives execution to completion even through process crashes. When the Restate server becomes unavailable, journal writes fail and new invocations cannot be accepted; when the journal storage fills, durability guarantees break silently; when handlers are stuck in long retries, the invocation queue grows without bound. Vigilmon gives you external monitoring for Restate server availability, invocation queue depth, journal storage health, handler execution latency, and cluster node connectivity so you catch durability infrastructure failures before your application's guarantees are compromised.

What You'll Set Up

  • Restate server health via HTTP endpoint probes and cron heartbeats
  • Invocation queue depth and retry storm monitoring
  • Journal storage utilization tracking
  • Handler execution latency measurement
  • Cluster node health and replication status

Prerequisites

  • Restate 1.0+ running as a server process or in Kubernetes
  • Access to the Restate admin API (default port: 9070)
  • Access to the Restate ingress port (default: 8080)
  • A free Vigilmon account

Step 1: Monitor Restate Server Health

Restate exposes an admin API at port 9070 and an ingress API at port 8080. When the server process crashes or the HTTP listener stops responding, all new invocations are rejected and durable execution halts for any service that relies on Restate for orchestration. The admin API's /health endpoint provides an immediate liveness check:

  1. In Vigilmon, click Add Monitor → HTTP(S).
  2. Enter the URL: http://restate.internal:9070/health
  3. Set Check interval to 1 minute.
  4. Set Expected HTTP status to 200.

For environments where the admin API is not externally reachable, use a cron heartbeat with a local probe:

#!/bin/bash
# /usr/local/bin/restate-health-check.sh

HEARTBEAT_URL="https://vigilmon.online/heartbeat/abc123"
RESTATE_ADMIN_URL="http://localhost:9070"
RESTATE_INGRESS_URL="http://localhost:8080"
TIMEOUT=10

# Check admin API health
ADMIN_STATUS=$(curl -s -o /dev/null -w "%{http_code}" \
  --max-time "$TIMEOUT" \
  "${RESTATE_ADMIN_URL}/health")

if [ "$ADMIN_STATUS" != "200" ]; then
  echo "Restate admin API unhealthy: HTTP $ADMIN_STATUS"
  exit 1
fi

# Check ingress is accepting connections
INGRESS_STATUS=$(curl -s -o /dev/null -w "%{http_code}" \
  --max-time "$TIMEOUT" \
  "${RESTATE_INGRESS_URL}/restate/health")

if [ "$INGRESS_STATUS" != "200" ] && [ "$INGRESS_STATUS" != "404" ]; then
  echo "Restate ingress unhealthy: HTTP $INGRESS_STATUS"
  exit 1
fi

echo "Restate server healthy (admin: $ADMIN_STATUS, ingress: $INGRESS_STATUS)"
curl -s "$HEARTBEAT_URL"

Set the Vigilmon cron heartbeat to 2 minutes. Restate's health endpoint responds in milliseconds when healthy — a timeout or non-200 response indicates the server process has died or the HTTP listener has deadlocked.

Step 2: Monitor Invocation Queue Depth

Restate queues invocations when handlers are busy, retrying, or when downstream services are unavailable. A growing queue is the first sign of a retry storm (handlers repeatedly failing and being retried with exponential backoff) or a downstream service outage. Monitor queue depth via the Restate admin API:

#!/bin/bash
# /usr/local/bin/restate-queue-check.sh

HEARTBEAT_URL="https://vigilmon.online/heartbeat/def456"
RESTATE_ADMIN_URL="http://localhost:9070"
MAX_PENDING=100   # Alert if more than 100 invocations are pending
MAX_RETRYING=20   # Alert if more than 20 invocations are in retry loops
TIMEOUT=15

# Get invocation state counts from the admin API
INVOCATIONS=$(curl -s \
  --max-time "$TIMEOUT" \
  "${RESTATE_ADMIN_URL}/invocations?status=pending&limit=1")

PENDING_COUNT=$(echo "$INVOCATIONS" | python3 -c "
import sys, json
try:
    data = json.load(sys.stdin)
    print(data.get('total', 0))
except:
    print(0)
" 2>/dev/null || echo 0)

RETRYING_INVOCATIONS=$(curl -s \
  --max-time "$TIMEOUT" \
  "${RESTATE_ADMIN_URL}/invocations?status=retrying&limit=1")

RETRYING_COUNT=$(echo "$RETRYING_INVOCATIONS" | python3 -c "
import sys, json
try:
    data = json.load(sys.stdin)
    print(data.get('total', 0))
except:
    print(0)
" 2>/dev/null || echo 0)

PENDING_COUNT="${PENDING_COUNT:-0}"
RETRYING_COUNT="${RETRYING_COUNT:-0}"

FAILED=0

if [ "$PENDING_COUNT" -ge "$MAX_PENDING" ]; then
  echo "Invocation queue backed up: ${PENDING_COUNT} pending (max: ${MAX_PENDING})"
  FAILED=1
fi

if [ "$RETRYING_COUNT" -ge "$MAX_RETRYING" ]; then
  echo "Retry storm detected: ${RETRYING_COUNT} retrying (max: ${MAX_RETRYING})"
  FAILED=1
fi

if [ "$FAILED" -eq 0 ]; then
  echo "Invocation queue OK: ${PENDING_COUNT} pending, ${RETRYING_COUNT} retrying"
  curl -s "$HEARTBEAT_URL"
else
  exit 1
fi

Create a Vigilmon cron heartbeat with a 5 minute expected interval. A retry storm — many invocations stuck in exponential backoff — indicates a downstream service outage or a systematic handler bug. Examine the specific invocations with curl http://localhost:9070/invocations?status=retrying to identify the affected service and handler.

Step 3: Monitor Journal Storage Health

Restate persists every invocation's state to its journal — a RocksDB-backed store. The journal is the source of truth for durable execution: if journal writes fail due to disk exhaustion, file descriptor limits, or RocksDB corruption, Restate can no longer guarantee exactly-once execution. Monitor journal storage utilization before it becomes critical:

#!/bin/bash
# /usr/local/bin/restate-journal-check.sh

HEARTBEAT_URL="https://vigilmon.online/heartbeat/ghi789"
RESTATE_DATA_DIR="${RESTATE_DATA_DIR:-/restate-data}"
MAX_DISK_PCT=80   # Alert if journal disk exceeds 80% full
MAX_JOURNAL_GB=100  # Alert if journal exceeds 100 GB

# Check disk where journal is stored
JOURNAL_DISK_PCT=$(df "$RESTATE_DATA_DIR" 2>/dev/null | \
  awk 'NR==2 {gsub(/%/,""); print $5}' || echo 0)

# Measure journal size directly
JOURNAL_GB=$(du -s --block-size=1G "${RESTATE_DATA_DIR}/db" 2>/dev/null | \
  awk '{print $1}' || echo 0)

# Check for RocksDB compaction errors via log scan
ROCKSDB_ERRORS=$(find "${RESTATE_DATA_DIR}/logs" -name "*.log" -newer /tmp/restate-journal-last-check \
  -exec grep -l "Corruption\|IO error\|RocksDB error" {} \; 2>/dev/null | wc -l)

touch /tmp/restate-journal-last-check

JOURNAL_DISK_PCT="${JOURNAL_DISK_PCT:-0}"
JOURNAL_GB="${JOURNAL_GB:-0}"
ROCKSDB_ERRORS="${ROCKSDB_ERRORS:-0}"

FAILED=0

if [ "$JOURNAL_DISK_PCT" -ge "$MAX_DISK_PCT" ]; then
  echo "Journal disk full: ${JOURNAL_DISK_PCT}% (threshold: ${MAX_DISK_PCT}%)"
  FAILED=1
fi

if [ "$JOURNAL_GB" -ge "$MAX_JOURNAL_GB" ]; then
  echo "Journal size large: ${JOURNAL_GB}GB (threshold: ${MAX_JOURNAL_GB}GB)"
  FAILED=1
fi

if [ "$ROCKSDB_ERRORS" -gt 0 ]; then
  echo "RocksDB errors detected in recent logs: ${ROCKSDB_ERRORS} files with errors"
  FAILED=1
fi

if [ "$FAILED" -eq 0 ]; then
  echo "Journal storage OK: ${JOURNAL_DISK_PCT}% disk, ${JOURNAL_GB}GB size"
  curl -s "$HEARTBEAT_URL"
else
  exit 1
fi

Set the Vigilmon heartbeat to 10 minutes. Journal growth is proportional to invocation volume and the number of retrying invocations — a retry storm that persists for hours can fill the journal unexpectedly. Configure Restate's worker.cleanup-interval and worker.invocation-state-ttl settings to prune completed invocations from the journal automatically.

Step 4: Track Handler Execution Latency

Handler latency measures the time between Restate dispatching an invocation to a registered handler and the handler returning a response. When handlers are slow — due to downstream database latency, complex state machine transitions, or resource contention in the handler service — the invocation queue grows and the user-facing latency for all operations that depend on those handlers increases proportionally. Track handler latency using a synthetic probe invocation:

#!/bin/bash
# /usr/local/bin/restate-latency-check.sh

HEARTBEAT_URL="https://vigilmon.online/heartbeat/jkl012"
RESTATE_INGRESS_URL="http://localhost:8080"
MAX_LATENCY_MS=2000   # Alert if handler latency exceeds 2 seconds
PROBE_SERVICE="HealthProbe"   # A minimal handler registered for latency testing
PROBE_HANDLER="ping"
TIMEOUT=30

INVOKE_START=$(date +%s%3N)

# Invoke the probe handler synchronously
HTTP_CODE=$(curl -s -o /tmp/restate-latency-response -w "%{http_code}" \
  --max-time "$TIMEOUT" \
  -X POST \
  -H "Content-Type: application/json" \
  -d '{}' \
  "${RESTATE_INGRESS_URL}/${PROBE_SERVICE}/${PROBE_HANDLER}")

INVOKE_END=$(date +%s%3N)
LATENCY_MS=$(( INVOKE_END - INVOKE_START ))

if [ "$HTTP_CODE" != "200" ]; then
  echo "Restate probe handler failed: HTTP $HTTP_CODE"
  exit 1
fi

if [ "$LATENCY_MS" -le "$MAX_LATENCY_MS" ]; then
  echo "Handler latency OK: ${LATENCY_MS}ms (max: ${MAX_LATENCY_MS}ms)"
  curl -s "$HEARTBEAT_URL"
else
  echo "Handler latency high: ${LATENCY_MS}ms (max: ${MAX_LATENCY_MS}ms)"
  exit 1
fi

Register a minimal HealthProbe handler in your Restate application for this probe. Run the latency check every 5 minutes with a Vigilmon heartbeat at 10 minute expected interval. High probe latency that does not correlate with application traffic usually indicates Restate's internal scheduler is under pressure from a large number of concurrent invocations or journal compaction activity.

Step 5: Monitor Cluster Node Connectivity

In Restate cluster deployments, multiple server nodes coordinate via a consensus protocol to maintain journal replication. When a node loses connectivity to its peers — due to network partition, a failed node, or misconfigured discovery — the cluster can lose quorum and stop accepting writes. Monitor cluster node health via the admin API:

#!/bin/bash
# /usr/local/bin/restate-cluster-check.sh

HEARTBEAT_URL="https://vigilmon.online/heartbeat/mno345"
RESTATE_ADMIN_URL="http://localhost:9070"
MIN_HEALTHY_NODES=2   # Minimum number of nodes that must be healthy
TIMEOUT=15

# Get cluster node status
CLUSTER_STATUS=$(curl -s \
  --max-time "$TIMEOUT" \
  "${RESTATE_ADMIN_URL}/cluster")

if [ -z "$CLUSTER_STATUS" ]; then
  echo "Could not reach Restate admin API for cluster status"
  exit 1
fi

HEALTHY_NODES=$(echo "$CLUSTER_STATUS" | python3 -c "
import sys, json
try:
    data = json.load(sys.stdin)
    nodes = data.get('nodes', [])
    healthy = [n for n in nodes if n.get('status') in ('alive', 'healthy')]
    print(len(healthy))
except:
    # Single-node deployment — treat as healthy if API responds
    print(1)
" 2>/dev/null || echo 1)

TOTAL_NODES=$(echo "$CLUSTER_STATUS" | python3 -c "
import sys, json
try:
    data = json.load(sys.stdin)
    print(len(data.get('nodes', [])))
except:
    print(1)
" 2>/dev/null || echo 1)

HEALTHY_NODES="${HEALTHY_NODES:-0}"

if [ "$HEALTHY_NODES" -ge "$MIN_HEALTHY_NODES" ]; then
  echo "Cluster OK: ${HEALTHY_NODES}/${TOTAL_NODES} nodes healthy"
  curl -s "$HEARTBEAT_URL"
else
  echo "Cluster degraded: only ${HEALTHY_NODES}/${TOTAL_NODES} nodes healthy (min: ${MIN_HEALTHY_NODES})"
  exit 1
fi

Set the Vigilmon heartbeat to 5 minutes. For single-node Restate deployments, skip this check — it only applies to multi-node clusters where quorum matters. A cluster that drops below the minimum healthy node count cannot guarantee durability until the failed node rejoins or is replaced.

Step 6: Set Up Alert Channels

Configure Vigilmon to route Restate alerts to your platform and application teams:

  1. Go to Alert Channels in Vigilmon.
  2. Add a Slack webhook to #platform-alerts — Restate failures break durable execution guarantees across all dependent services.
  3. Add PagerDuty for the server health and journal storage monitors — these indicate complete durability infrastructure failure.
  4. Add email notification for queue depth and latency monitors (performance degradation that needs attention, not an outage).
  5. Set Consecutive failures before alert to 2 for queue depth monitors — brief spikes during deployments are normal as handlers restart.

Add maintenance windows during Restate upgrades:

curl -X POST https://vigilmon.online/api/maintenance \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -d '{
    "monitor_id": "RESTATE_MONITOR_ID",
    "duration_minutes": 30,
    "reason": "Restate server upgrade and journal migration"
  }'

Summary

| Monitor | Target | What It Catches | |---|---|---| | HTTP monitor (health) | /health admin endpoint | Server crash, HTTP listener deadlock | | Cron heartbeat (queue) | Pending + retrying invocation count | Retry storm, downstream service outage | | Cron heartbeat (journal) | Journal disk usage + RocksDB errors | Storage exhaustion, journal corruption | | Cron heartbeat (latency) | Synthetic probe invocation round-trip | Handler slowness, scheduler backpressure | | Cron heartbeat (cluster) | Admin API node health list | Network partition, node failure, quorum loss |

Restate's failure modes are particularly dangerous because they silently violate the durability guarantees that your application is built on top of. A stalled journal write doesn't immediately crash the application — it causes subtle at-least-once or lost-execution bugs that only surface during failures. Vigilmon's external monitoring catches infrastructure problems before they compromise your application's correctness guarantees.

Start monitoring for free at vigilmon.online.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles