Monitoring Sysbox with Vigilmon

Sysbox enables system-level workloads — Docker-in-Docker, Kubernetes-in-Docker, systemd — inside unprivileged containers by virtualizing Linux user namespaces. When the sysbox-mgr daemon crashes or the UID/GID pool is exhausted, new Sysbox containers fail to start and existing workflows break silently. Vigilmon gives you continuous monitoring of the Sysbox daemon, container start success rates, the user namespace pool, and Docker integration health.

What You'll Set Up

sysbox-mgr daemon process health via heartbeat
Container start success rate monitoring
User namespace UID/GID pool availability alerts
Docker integration health (sysbox-runc in docker info)
Sysbox-in-Kubernetes DaemonSet pod health
Kernel compatibility verification

Prerequisites

Sysbox 0.6+ installed (via apt or Sysbox binaries)
Docker configured with sysbox-runc runtime, or Kubernetes with sysbox-deploy-k8s DaemonSet
Linux kernel 5.5+ with user namespace support enabled
A free Vigilmon account

Step 1: Monitor sysbox-mgr Daemon Health

sysbox-mgr manages Sysbox-specific state and the UID/GID range pool. If it crashes, no new Sysbox containers can start — existing containers are unaffected but unmanageable.

Monitor daemon health via a process-liveness heartbeat:

Create /usr/local/bin/sysbox-mgr-check.sh:

#!/bin/bash
# Check sysbox-mgr is running and its unix socket is present
if systemctl is-active --quiet sysbox-mgr && \
   [ -S /run/sysbox/sysbox-mgr.sock ]; then
  curl -s "https://vigilmon.online/heartbeat/$SYSBOX_MGR_TOKEN"
fi

Add to cron every minute:

* * * * * /usr/local/bin/sysbox-mgr-check.sh

In Vigilmon: Add Monitor → Type: Heartbeat.
Set Expected interval to 2 minutes.
Paste the heartbeat URL into SYSBOX_MGR_TOKEN in your script.

A missed heartbeat means either sysbox-mgr is down or its Unix socket is unavailable — both block new container starts.

If you expose a lightweight status server from sysbox-mgr (via systemd socket activation), add a direct HTTP monitor instead:

Add Monitor → Type: HTTP / HTTPS.
URL: http://localhost:9102/healthz (or your chosen port).
Check interval: 1 minute, Expected status: 200.

Step 2: Monitor Container Start Success Rate

Every Sysbox container start invokes sysbox-runc. A spike in container start failures indicates runtime errors, kernel incompatibilities, or pool exhaustion.

Wrap your container start calls with a success/failure reporter:

#!/bin/bash
CONTAINER_NAME="$1"
IMAGE="$2"

docker run --runtime=sysbox-runc --name "$CONTAINER_NAME" "$IMAGE" &>/tmp/sysbox-start.log
EXIT_CODE=$?

if [ "$EXIT_CODE" -eq 0 ]; then
  curl -s "https://vigilmon.online/heartbeat/$START_SUCCESS_TOKEN"
else
  # Log the failure for later analysis
  echo "Sysbox container start failed: $CONTAINER_NAME" >> /var/log/sysbox-failures.log
fi

For automation pipelines, monitor the docker events stream for Sysbox container start failures:

#!/bin/bash
docker events --filter "type=container" --filter "event=die" --filter "label=com.docker.runtime=sysbox-runc" \
  --format '{{.Actor.Attributes.name}} died at {{.Time}}' >> /var/log/sysbox-deaths.log &

Set a Vigilmon heartbeat for the container-start-success check with a 5-minute expected interval when run periodically from your CI pipeline.

Step 3: Monitor User Namespace UID/GID Pool

Sysbox allocates a distinct UID/GID range per container from a finite pool. Pool exhaustion prevents new containers from starting with the error no UID/GID available.

Check pool availability:

#!/bin/bash
# sysbox-mgr logs pool state; parse current allocation count
# Alternatively, count running Sysbox containers vs. configured max

RUNNING=$(docker ps --filter "label=com.docker.runtime=sysbox-runc" --format "{{.ID}}" 2>/dev/null | wc -l)
# Sysbox default max is 268435455 / 65536 ≈ 4096 containers per system
# Check if we're below 80% of a configured limit
MAX_CONTAINERS="${SYSBOX_MAX_CONTAINERS:-100}"
THRESHOLD=$((MAX_CONTAINERS * 80 / 100))

if [ "$RUNNING" -lt "$THRESHOLD" ]; then
  curl -s "https://vigilmon.online/heartbeat/$POOL_TOKEN"
fi

Run every 5 minutes. Set the Vigilmon heartbeat expected interval to 10 minutes. When running containers approach the pool limit, the heartbeat goes silent and you're alerted before new starts fail.

Check pool exhaustion in sysbox-mgr logs if you need the exact count:

journalctl -u sysbox-mgr --since "5 minutes ago" | grep -i "uid.* pool\|no.*uid"

Step 4: Monitor Docker Integration Health

Sysbox registers sysbox-runc as a Docker runtime. If the runtime disappears from Docker's runtime list (after a Docker daemon restart or Sysbox reinstall), all --runtime=sysbox-runc container launches silently fail.

Add a Docker integration health check:

#!/bin/bash
if docker info 2>/dev/null | grep -q "sysbox-runc"; then
  curl -s "https://vigilmon.online/heartbeat/$DOCKER_INTEGRATION_TOKEN"
fi

Schedule every 5 minutes via cron. Set Vigilmon heartbeat interval to 10 minutes.

This alert catches the common "Sysbox was reinstalled but Docker wasn't reloaded" failure mode that removes sysbox-runc from Docker's runtimes list.

Step 5: Monitor Sysbox-in-Kubernetes DaemonSet Health

If you're running Sysbox in Kubernetes via the sysbox-deploy-k8s DaemonSet, every node in the cluster must have the Sysbox DaemonSet pod running. A missing pod on any node means Sysbox containers cannot start on that node.

Monitor DaemonSet health from a cron job on the Kubernetes control plane:

#!/bin/bash
DESIRED=$(kubectl get daemonset sysbox-deploy-k8s -n kube-system -o jsonpath='{.status.desiredNumberScheduled}' 2>/dev/null)
READY=$(kubectl get daemonset sysbox-deploy-k8s -n kube-system -o jsonpath='{.status.numberReady}' 2>/dev/null)

if [ "$DESIRED" -gt 0 ] && [ "$READY" -eq "$DESIRED" ]; then
  curl -s "https://vigilmon.online/heartbeat/$K8S_DAEMONSET_TOKEN"
fi

Schedule every 5 minutes. Set Vigilmon heartbeat interval to 10 minutes. When any node is missing the Sysbox pod, READY < DESIRED and the heartbeat stops.

Alternatively, add an HTTP monitor on kube-state-metrics if available:

Add Monitor → Type: HTTP / HTTPS.
URL: http://your-kube-state-metrics:8080/metrics
Keyword: kube_daemonset_status_number_ready{daemonset="sysbox-deploy-k8s"}

Step 6: Verify Kernel Compatibility

Sysbox requires Linux kernel 5.5+ with user namespace support. A kernel downgrade (e.g., from an unintended package update) breaks Sysbox without any obvious error until a new container is started.

Add a daily kernel version check:

#!/bin/bash
KERNEL=$(uname -r | cut -d. -f1,2)
MAJOR=$(echo "$KERNEL" | cut -d. -f1)
MINOR=$(echo "$KERNEL" | cut -d. -f2)

if [ "$MAJOR" -gt 5 ] || ([ "$MAJOR" -eq 5 ] && [ "$MINOR" -ge 5 ]); then
  curl -s "https://vigilmon.online/heartbeat/$KERNEL_TOKEN"
fi

Schedule daily. Set Vigilmon heartbeat expected interval to 25 hours. A kernel downgrade triggers an immediate missed heartbeat alert.

Step 7: Monitor Sysbox Log Error Rate

A spike in error-level log messages from sysbox-mgr or sysbox-runc often precedes systemic failures. Catch it before containers start failing:

#!/bin/bash
ERROR_COUNT=$(journalctl -u sysbox-mgr --since "5 minutes ago" -p err -q 2>/dev/null | wc -l)
RUNC_ERRORS=$(grep -c "level=error" /var/log/sysbox/sysbox-runc.log 2>/dev/null || echo 0)

if [ "$ERROR_COUNT" -eq 0 ] && [ "$RUNC_ERRORS" -eq 0 ]; then
  curl -s "https://vigilmon.online/heartbeat/$LOG_TOKEN"
fi

Run every 5 minutes. Set heartbeat expected interval to 10 minutes. Any errors in the last 5 minutes stop the heartbeat, alerting you to investigate before the errors accumulate into outages.

Step 8: Configure Alerting

In Vigilmon, navigate to Alert Channels and connect your preferred notification method:

Email — for async alerting on non-urgent checks (kernel version, pool availability)
Slack or Discord webhook — real-time team notification
PagerDuty — for production Kubernetes clusters where Sysbox outage blocks CI/CD pipelines

Recommended alert priorities:

| Monitor | Condition | Priority | |---|---|---| | sysbox-mgr daemon | Heartbeat missed | Page immediately | | Docker runtime integration | Heartbeat missed | Alert within 5 min | | UID/GID pool | Heartbeat missed | Alert within 10 min | | Container start success | Heartbeat missed | Alert within 10 min | | K8s DaemonSet | Heartbeat missed | Alert within 5 min | | Log error rate | Heartbeat missed | Alert within 10 min | | Kernel compatibility | Heartbeat missed | Alert within 24 hr |

Conclusion

Sysbox's security model — user namespace virtualization without --privileged — is powerful, but it adds a daemon dependency (sysbox-mgr) and a resource pool (UID/GID ranges) that standard Docker monitoring doesn't cover. With Vigilmon heartbeat monitors for daemon liveness, pool availability, Docker integration health, DaemonSet coverage, and log error rates, you have early warning before any of these failure modes impact production workloads. Add kernel compatibility checks and wire up your alert channels for full Sysbox observability.