Monitoring gVisor (runsc) with Vigilmon

gVisor is Google's open-source container sandbox runtime that intercepts Linux syscalls in user space via its Sentry kernel and proxies filesystem access through the Gofer process. It provides strong isolation between containers and the host kernel — but that isolation layer also introduces new failure modes: the Sentry process can crash, the Gofer file proxy can stall, and syscall intercept latency can spike under load, all without any signal visible to the container orchestrator. Vigilmon gives you external monitoring for gVisor's Sentry and Gofer process health, sandbox integrity, and syscall latency so you catch runtime degradation before your sandboxed workloads fail silently.

What You'll Set Up

• Sentry and Gofer process health via cron heartbeats
• Sandbox container responsiveness monitoring
• Syscall intercept latency tracking
• Security boundary integrity checks
• Automated alerting on sandbox crashes

Prerequisites

• gVisor installed and configured (runsc available on PATH)
• Containerd or Docker configured to use runsc as a runtime
• A sandboxed container running for testing
• A free Vigilmon account

Step 1: Monitor Sentry and Gofer Process Health

Each gVisor sandbox runs two key processes: the Sentry (the user-space kernel that intercepts syscalls) and the Gofer (the file proxy that mediates filesystem access). When either crashes, the sandbox container dies without a clean exit code. Monitor both as a pair:

1. In Vigilmon, click Add Monitor → Cron Heartbeat.
2. Set Expected ping interval to 2 minutes.
3. Copy the heartbeat URL (e.g., https://vigilmon.online/heartbeat/abc123).

Create the process health script:

#!/bin/bash
# /usr/local/bin/gvisor-process-check.sh

HEARTBEAT_URL="https://vigilmon.online/heartbeat/abc123"

# Count runsc processes (Sentry runs as "runsc" or "runsc-sandbox")
SENTRY_COUNT=$(pgrep -c "runsc" 2>/dev/null || echo 0)

# Count gofer processes  
GOFER_COUNT=$(pgrep -c "runsc-gofer" 2>/dev/null || echo 0)

# We expect sandboxes to have paired Sentry+Gofer processes
# Sanity check: if we have Sentry but no Gofer, something is wrong
if [ "$SENTRY_COUNT" -gt 0 ] && [ "$GOFER_COUNT" -eq 0 ]; then
  echo "Sentry/Gofer mismatch: ${SENTRY_COUNT} Sentry, ${GOFER_COUNT} Gofer"
  exit 1
fi

echo "gVisor processes OK: ${SENTRY_COUNT} Sentry, ${GOFER_COUNT} Gofer"
curl -s "$HEARTBEAT_URL"

Install the cron job:

chmod +x /usr/local/bin/gvisor-process-check.sh
(crontab -l 2>/dev/null; echo "*/2 * * * * /usr/local/bin/gvisor-process-check.sh >> /var/log/gvisor-health.log 2>&1") | crontab -

A Sentry/Gofer mismatch is one of the most common gVisor failure patterns — it indicates a partial crash where one component exited while the other is wedged in a cleanup loop.

Step 2: Monitor Sandbox Container Responsiveness

Running containers inside gVisor should respond to health checks just like non-sandboxed containers. Add an HTTP health check to any sandboxed service to verify the entire stack — Sentry syscall interception, Gofer filesystem proxy, and the application itself — is working end-to-end:

1. In Vigilmon, click Add Monitor → HTTP / HTTPS.
2. Set the URL to your sandboxed service's health endpoint, e.g. https://your-server.example.com/health.
3. Set check interval to 1 minute.
4. Set expected status code to 200.
5. Save the monitor.

If you don't have an HTTP service, add a heartbeat from inside the sandbox:

# Run inside the gVisor sandbox container
cat > /entrypoint-healthcheck.sh << 'EOF'
#!/bin/sh
while true; do
  # Ping Vigilmon every 2 minutes
  curl -s https://vigilmon.online/heartbeat/def456
  sleep 120
done
EOF
chmod +x /entrypoint-healthcheck.sh

Or add it as a background task in your container's entrypoint. This approach catches the full failure chain: if Sentry stalls syscall processing, curl inside the container will hang and the heartbeat will stop.

Step 3: Track Syscall Intercept Latency

gVisor's Sentry intercepts every Linux syscall and translates it into safe operations — this adds latency compared to native containers. Under load or resource pressure, this latency can spike and make I/O-bound workloads unresponsive. Measure syscall overhead periodically:

#!/bin/bash
# /usr/local/bin/gvisor-latency-check.sh

HEARTBEAT_URL="https://vigilmon.online/heartbeat/ghi789"
LATENCY_THRESHOLD_MS=100  # Alert if basic I/O takes more than 100ms extra vs native

# Run a simple I/O benchmark inside a gVisor sandbox
# Using "runsc do" for a one-shot sandboxed command
START=$(date +%s%N)

# Time a simple syscall-heavy operation inside a sandbox
runsc do -- /bin/sh -c "for i in \$(seq 1 100); do echo test > /tmp/gvisor-test-\$i; rm /tmp/gvisor-test-\$i; done" 2>/dev/null

END=$(date +%s%N)
ELAPSED_MS=$(( (END - START) / 1000000 ))

if [ "$ELAPSED_MS" -lt "$LATENCY_THRESHOLD_MS" ]; then
  echo "gVisor latency OK: ${ELAPSED_MS}ms for 100 I/O operations"
  curl -s "$HEARTBEAT_URL"
else
  echo "gVisor latency high: ${ELAPSED_MS}ms (threshold: ${LATENCY_THRESHOLD_MS}ms)"
  exit 1
fi

Set the Vigilmon heartbeat interval to 10 minutes. Syscall latency spikes in gVisor are typically caused by Sentry GC pressure (it's a Go application), resource contention on the host, or KVM virtualization overhead in KVM mode. Catching the spike early gives you time to drain workloads before user-visible timeouts.

Step 4: Verify Security Boundary Integrity

gVisor's security guarantee depends on the Sentry intercepting all syscalls — a misconfigured sandbox that accidentally runs in native mode (bypassing gVisor) provides no isolation. Verify that workloads are actually running inside gVisor sandboxes as expected:

#!/bin/bash
# /usr/local/bin/gvisor-sandbox-verify.sh

HEARTBEAT_URL="https://vigilmon.online/heartbeat/jkl012"

# List running Docker containers that should be using gVisor
# Adjust the runtime name to match your configuration (runsc, runsc-kvm, etc.)
GVISOR_CONTAINERS=$(docker ps --filter "label=com.docker.runtime=runsc" --format "{{.ID}}" 2>/dev/null | wc -l)
RUNSC_RUNTIME_CONTAINERS=$(docker ps --format '{{.ID}}' 2>/dev/null | xargs -I{} docker inspect {} --format '{{.ID}} {{.HostConfig.Runtime}}' 2>/dev/null | grep -c "runsc" || echo 0)

# Check that expected containers are using the gVisor runtime
if [ "$RUNSC_RUNTIME_CONTAINERS" -gt 0 ]; then
  echo "gVisor boundary OK: ${RUNSC_RUNTIME_CONTAINERS} containers using runsc runtime"
  curl -s "$HEARTBEAT_URL"
else
  echo "WARNING: No containers found using runsc runtime — verify sandbox configuration"
  exit 1
fi

Run this check every 15 minutes and set the Vigilmon heartbeat interval to 20 minutes. If containers are accidentally started with the runc runtime instead of runsc, they run without gVisor isolation — a silent security regression that this check catches immediately.

Step 5: Monitor runsc State Files and Logs

gVisor writes state files and logs under /var/run/runsc (or your configured root). Stale state files from crashed sandboxes can prevent new sandboxes from starting. Monitor the state directory for accumulation:

#!/bin/bash
# /usr/local/bin/gvisor-state-check.sh

HEARTBEAT_URL="https://vigilmon.online/heartbeat/mno345"
RUNSC_ROOT="/var/run/runsc"
STATE_THRESHOLD=50  # Alert if more than 50 state entries accumulate

if [ ! -d "$RUNSC_ROOT" ]; then
  # Directory doesn't exist — gVisor not in use or rootless setup
  curl -s "$HEARTBEAT_URL"
  exit 0
fi

STATE_COUNT=$(find "$RUNSC_ROOT" -name "state.json" 2>/dev/null | wc -l)
RUNNING_SANDBOXES=$(pgrep -c "runsc" 2>/dev/null || echo 0)

# Stale state files = state entries without matching processes
STALE_ESTIMATE=$(( STATE_COUNT - RUNNING_SANDBOXES ))

if [ "$STALE_ESTIMATE" -gt "$STATE_THRESHOLD" ]; then
  echo "Stale runsc state files: ~${STALE_ESTIMATE} stale entries"
  exit 1
fi

echo "runsc state OK: ${STATE_COUNT} entries, ~${STALE_ESTIMATE} possibly stale"
curl -s "$HEARTBEAT_URL"

Create a Vigilmon heartbeat with a 15 minute expected interval. gVisor does not automatically clean up state files from killed sandboxes — stale state accumulates on hosts that kill containers rather than stopping them cleanly, and can eventually block new container creation.

Step 6: Configure Alert Channels

Set up Vigilmon to notify the right people when gVisor health degrades:

1. Go to Alert Channels in Vigilmon.
2. Add a Slack webhook to your #security-infra or #platform channel — gVisor issues are often security-relevant.
3. For the security boundary integrity monitor, add a PagerDuty integration so a sandbox misconfiguration wakes someone up.
4. Set Consecutive failures before alert to 2 for process-count heartbeats to avoid alerts from transient cron timing.

For planned gVisor upgrades, pause monitors first:

# Pause during upgrade
curl -X POST https://vigilmon.online/api/maintenance \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -d '{
    "monitor_id": "MONITOR_ID",
    "duration_minutes": 15,
    "reason": "gVisor runsc upgrade"
  }'

# Upgrade runsc binary
systemctl stop containerd
cp runsc-new /usr/bin/runsc
systemctl start containerd

Summary

| Monitor | Target | What It Catches | |---|---|---| | Cron heartbeat (processes) | Sentry + Gofer process pair count | Sentry crash, Gofer stall, partial sandbox failure | | HTTP or heartbeat (in-sandbox) | Health endpoint or in-container curl | Full stack failure: Sentry stall, Gofer block | | Cron heartbeat (latency) | Timed I/O benchmark inside sandbox | Syscall intercept slowdown, GC pressure, KVM overhead | | Cron heartbeat (boundary) | Runtime label check on containers | Accidental runc fallback, sandbox misconfiguration | | Cron heartbeat (state files) | Stale state file accumulation | Zombie state entries blocking new container creation |

gVisor's security model depends on every component working correctly — a crashed Sentry, a stalled Gofer, or a misconfigured runtime silently breaks the isolation boundary. Vigilmon's external checks give you visibility into the gVisor stack that the container orchestrator cannot provide, so security regressions and performance degradation surface as alerts rather than mystery failures.

Start monitoring for free at vigilmon.online.