Monitoring crun with Vigilmon: Keeping Your OCI Container Runtime Healthy

crun is a lightweight, fast OCI container runtime written in C that serves as the default runtime engine for Podman and CRI-O. Unlike runc (written in Go), crun has a minimal footprint, starts containers in microseconds, and provides full cgroup v2 support without shim processes. Platform teams running crun-based stacks depend on it for every container start, stop, and exec — when crun's integration with CRI-O or Podman breaks, workloads stop scheduling silently. Vigilmon gives you external monitoring that watches the runtime integration layer and validates that container lifecycle operations are completing on schedule.

What You'll Build

• Vigilmon HTTP monitors for the CRI-O or Podman API socket health
• A heartbeat monitor validating that container lifecycle operations complete end-to-end
• A cgroup v2 health heartbeat to catch silent resource accounting failures
• Alert channels routed to your platform engineering team

Prerequisites

• crun installed and configured as the OCI runtime for Podman or CRI-O
• Linux system with cgroup v2 enabled (recommended)
• A free account at vigilmon.online

Why Monitoring crun Matters

crun operates at the lowest layer of the container stack. Its failure modes are silent and often misdiagnosed as application or scheduler problems:

Runtime binary failures prevent all container starts. If the crun binary is missing, corrupt, or incompatible with the host kernel, every podman run or pod schedule through CRI-O returns an opaque error. Engineers typically debug the wrong layer — checking images, network policies, or scheduler state — before realizing the runtime itself is broken.

cgroup v2 accounting failures are the hardest to detect. crun uses the cgroup v2 unified hierarchy for memory, CPU, and I/O limits. A cgroup hierarchy that is mounted incorrectly or partially broken still allows containers to start — but resource limits are silently unenforced. A container you believe is memory-limited is actually running without limits, risking node OOM.

OCI hook failures crash container starts without useful error messages. crun supports OCI lifecycle hooks (prestart, poststart, poststop) used by CNI plugins, seccomp loaders, and SELinux relabelers. A broken hook silently prevents pod creation in Kubernetes, manifesting only as CrashLoopBackOff or CreateContainerError at the pod level.

CRI-O runtime integration drift occurs when a CRI-O upgrade changes the expected crun invocation flags or socket protocol. External monitoring that validates the CRI-O socket and runs a container lifecycle probe catches this before node registration fails at cluster scale.

Step 1: Monitor the CRI-O API Socket

CRI-O exposes a gRPC API over a Unix socket, but you can expose it via HTTP for external monitoring using a lightweight proxy or by checking its readiness endpoint:

# Check CRI-O readiness (requires crictl configured)
crictl info | grep -q runtimeVersion
echo $?  # 0 = healthy

# Or check via the systemd service
systemctl is-active --quiet crio && echo "healthy"

To enable HTTP health checking for external monitoring, run a simple proxy:

# Add a systemd drop-in that exposes CRI-O runtime version via HTTP
cat > /etc/systemd/system/crio-health-proxy.service << 'EOF'
[Unit]
Description=CRI-O health proxy
After=crio.service

[Service]
ExecStart=/bin/bash -c 'while true; do echo -e "HTTP/1.1 200 OK\r\nContent-Type: text/plain\r\n\r\n$(crictl info 2>/dev/null | grep -c runtimeVersion)" | nc -l -p 9101 -q 1; done'
Restart=always

[Install]
WantedBy=multi-user.target
EOF
systemctl daemon-reload && systemctl enable --now crio-health-proxy

In the Vigilmon dashboard:

1. Go to Add Monitor → HTTP
2. Configure:

| Field | Value | |---|---| | Name | CRI-O runtime health | | URL | http://your-node-ip:9101 | | Method | GET | | Expected status | 200 | | Response body contains | 1 | | Check interval | 1 minute | | Timeout | 10 seconds | | Alert after | 2 consecutive failures |

If you use Podman's REST API instead of CRI-O, monitor the Podman socket service at port 8080 (default podman system service --time=0 tcp:8080).

Step 2: Monitor the Podman REST API (Podman-based deployments)

If your stack uses Podman with crun as the runtime, Podman's REST API provides a direct health endpoint:

# Start Podman socket service
systemctl enable --now podman.socket

# Test the Podman REST API
curl -s --unix-socket /run/podman/podman.sock http://d/v4.0.0/libpod/info | grep -c '"OCI Runtime"'

Expose Podman's API over TCP for external monitoring:

podman system service --time=0 tcp:127.0.0.1:8080 &

In Vigilmon:

| Field | Value | |---|---| | Name | Podman REST API | | URL | http://your-node-ip:8080/v4.0.0/libpod/info | | Method | GET | | Expected status | 200 | | Response body contains | crun | | Check interval | 2 minutes | | Timeout | 10 seconds | | Alert after | 2 consecutive failures |

The crun string in the response body confirms that crun — not another OCI runtime — is the active engine. A runtime switch (for example, back to runc after a failed crun upgrade) triggers this alert.

Step 3: Heartbeat Monitor for Container Lifecycle Operations

The most reliable way to validate crun is working is to run a complete container lifecycle and send a heartbeat on success. Create a systemd timer:

# /usr/local/bin/crun-lifecycle-probe.sh
#!/bin/bash
set -euo pipefail

HEARTBEAT_URL="${CRUN_HEARTBEAT_URL}"
RUNTIME="crun"

# Run a minimal container with explicit crun runtime
if podman run --rm --runtime "$RUNTIME" --timeout 30 \
   alpine:latest /bin/true 2>/dev/null; then
  curl -fsS --max-time 10 "$HEARTBEAT_URL" -o /dev/null
  echo "$(date): crun lifecycle probe OK, heartbeat sent"
else
  echo "$(date): ERROR — crun lifecycle probe failed" >&2
  exit 1
fi

# /etc/systemd/system/crun-probe.service
[Unit]
Description=crun container lifecycle probe
After=podman.service

[Service]
Type=oneshot
EnvironmentFile=/etc/crun-probe.env
ExecStart=/usr/local/bin/crun-lifecycle-probe.sh

# /etc/systemd/system/crun-probe.timer
[Unit]
Description=Run crun lifecycle probe every 5 minutes

[Timer]
OnBootSec=2min
OnUnitActiveSec=5min
Unit=crun-probe.service

[Install]
WantedBy=timers.target

# /etc/crun-probe.env
CRUN_HEARTBEAT_URL=https://vigilmon.online/heartbeat/YOUR_LIFECYCLE_TOKEN

Enable it:

chmod +x /usr/local/bin/crun-lifecycle-probe.sh
systemctl daemon-reload
systemctl enable --now crun-probe.timer

Create the heartbeat monitor in Vigilmon:

1. Add Monitor → Heartbeat
2. Name: crun container lifecycle
3. Expected interval: 5 minutes
4. Grace period: 3 minutes

Step 4: Heartbeat Monitor for cgroup v2 Resource Accounting

Validate that cgroup v2 resource limits are actually being enforced. This probe creates a container with a memory limit and verifies the cgroup hierarchy is configured correctly:

# /usr/local/bin/cgroup-probe.sh
#!/bin/bash
set -euo pipefail

HEARTBEAT_URL="${CGROUP_HEARTBEAT_URL}"

# Run container with 64MB memory limit and check cgroup was created
CONTAINER_ID=$(podman run -d --rm --memory 64m --runtime crun \
  alpine:latest sleep 5 2>/dev/null)

if [ -z "$CONTAINER_ID" ]; then
  echo "ERROR: Container failed to start" >&2
  exit 1
fi

# Verify cgroup memory limit was applied
CGROUP_PATH=$(podman inspect "$CONTAINER_ID" \
  --format '{{.CgroupPath}}' 2>/dev/null)

if [ -n "$CGROUP_PATH" ] && \
   [ -f "/sys/fs/cgroup/${CGROUP_PATH}/memory.max" ]; then
  LIMIT=$(cat "/sys/fs/cgroup/${CGROUP_PATH}/memory.max")
  podman stop "$CONTAINER_ID" >/dev/null 2>&1 || true
  echo "cgroup v2 memory limit: $LIMIT bytes"
  curl -fsS --max-time 10 "$HEARTBEAT_URL" -o /dev/null
  echo "cgroup probe heartbeat sent"
else
  podman stop "$CONTAINER_ID" >/dev/null 2>&1 || true
  echo "ERROR: cgroup v2 memory limit not found at expected path" >&2
  exit 1
fi

# /etc/systemd/system/cgroup-probe.timer
[Unit]
Description=Run cgroup v2 accounting probe every 15 minutes

[Timer]
OnBootSec=3min
OnUnitActiveSec=15min
Unit=cgroup-probe.service

[Install]
WantedBy=timers.target

Create the heartbeat monitor in Vigilmon:

1. Add Monitor → Heartbeat
2. Name: crun cgroup v2 accounting
3. Expected interval: 15 minutes
4. Grace period: 5 minutes

Step 5: Configure Alert Channels

Route crun alerts to your infrastructure team:

1. In Vigilmon: Alert Channels → Add Channel → Slack Webhook
2. Add your #infrastructure or #platform-engineering webhook URL
3. Assign to all crun monitors

For cgroup failures (unenforced resource limits are a reliability and security concern):

1. Alert Channels → Add Channel → PagerDuty
2. Route specifically the crun cgroup v2 accounting heartbeat to on-call
3. This failure warrants immediate response — a node with broken cgroup accounting has no container resource isolation

What You're Now Monitoring

| Component | Monitor Type | What It Detects | |---|---|---| | CRI-O runtime health | HTTP GET | CRI-O down, crun binary missing or broken | | Podman REST API | HTTP GET | Podman service down, runtime switched from crun | | Container lifecycle | Heartbeat (5 min) | crun fails to start/stop containers end-to-end | | cgroup v2 accounting | Heartbeat (15 min) | Resource limits unenforced, cgroup hierarchy broken |

Conclusion

crun's strength — its minimal footprint and absence of persistent processes — also makes it invisible to most observability stacks. There is no crun daemon to scrape metrics from, no log aggregation target, and no health endpoint to poll. External monitoring with Vigilmon fills that gap by watching the runtime integration layer and validating that container lifecycle operations complete on time. A broken crun binary is caught within 5 minutes; a cgroup hierarchy failure is caught within 15 minutes — before either problem cascades into workload scheduling failures at scale.

Get started free at vigilmon.online. The Podman REST API probe and lifecycle heartbeat together give you the two most critical crun signals in under ten minutes: is the runtime accepting invocations, and can it run a container end-to-end?