tutorial

Monitoring CRI-O with Vigilmon

CRI-O is the lightweight OCI-compliant container runtime purpose-built for Kubernetes — but CRI endpoint hangs, image pull failures, and pod sandbox creation stalls can silently break your cluster. Here's how to monitor CRI-O health, image pull performance, and pod sandbox latency with Vigilmon.

CRI-O is the Kubernetes-native container runtime that implements only what the Container Runtime Interface requires — nothing more. Designed to be minimal, stable, and auditable, it ships with RHEL-based Kubernetes distributions like OpenShift and is popular anywhere operators want a runtime that does exactly one job. But that minimalism cuts both ways: when CRI-O's CRI socket hangs, image pull goroutines deadlock, or storage driver health degrades, Kubernetes kubelets quietly stall pod creation with ContainerCreating states and no clear signal. Vigilmon gives you external monitoring for CRI-O socket health, image pull latency, pod sandbox creation performance, and storage driver status so you catch runtime failures before they cascade into scheduling backlogs.

What You'll Set Up

  • CRI-O process and socket health via cron heartbeats
  • CRI endpoint responsiveness monitoring
  • Image pull performance tracking
  • Pod sandbox creation latency checks
  • Storage driver health monitoring

Prerequisites

  • CRI-O 1.26+ running on Linux (systemctl status crio)
  • crictl CLI configured to use CRI-O (/etc/crictl.yaml)
  • crio-status or crio CLI available
  • A free Vigilmon account

Step 1: Monitor CRI-O Process and Socket Health

CRI-O runs as a systemd service and exposes a Unix socket at /var/run/crio/crio.sock. A heartbeat that probes the actual CRI socket — not just the systemd unit state — catches the failure modes that matter: daemon freeze, socket removal, and CRI API deadlock:

  1. In Vigilmon, click Add Monitor → Cron Heartbeat.
  2. Set Expected ping interval to 2 minutes.
  3. Copy the heartbeat URL (e.g., https://vigilmon.online/heartbeat/abc123).

Create the health check script:

#!/bin/bash
# /usr/local/bin/crio-health-check.sh

SOCKET="/var/run/crio/crio.sock"
HEARTBEAT_URL="https://vigilmon.online/heartbeat/abc123"

# Verify the socket file exists and is a socket
if [ ! -S "$SOCKET" ]; then
  echo "CRI-O socket missing: $SOCKET"
  exit 1
fi

# Probe the CRI API — this exercises the actual RPC path
if crictl --runtime-endpoint "unix://$SOCKET" --timeout 5s info >/dev/null 2>&1; then
  curl -s "$HEARTBEAT_URL"
else
  echo "CRI-O CRI API unresponsive on $SOCKET"
  exit 1
fi

Install and enable the cron job:

chmod +x /usr/local/bin/crio-health-check.sh

# Add to root crontab (CRI-O socket requires root)
(crontab -l 2>/dev/null; echo "*/2 * * * * /usr/local/bin/crio-health-check.sh >> /var/log/crio-health.log 2>&1") | crontab -

If the heartbeat stops firing, Vigilmon alerts after 2 minutes — a direct signal that the CRI-O daemon has frozen or its socket has been removed, well before the Kubernetes scheduler marks affected nodes as NotReady.

Step 2: Monitor CRI Endpoint Responsiveness

CRI-O's gRPC endpoint can become sluggish under storage pressure or when the image garbage collector holds internal locks. Kubelets poll this endpoint frequently; even a 3-second response delay causes kubelet-reported warnings and slows pod scheduling. Track CRI latency independently of process liveness:

#!/bin/bash
# /usr/local/bin/crio-cri-latency.sh

HEARTBEAT_URL="https://vigilmon.online/heartbeat/def456"
LATENCY_THRESHOLD_MS=3000  # Alert if CRI responds slower than 3 seconds

START=$(date +%s%N)
if crictl --runtime-endpoint unix:///var/run/crio/crio.sock \
    --timeout 10s info >/dev/null 2>&1; then
  END=$(date +%s%N)
  ELAPSED_MS=$(( (END - START) / 1000000 ))

  if [ "$ELAPSED_MS" -lt "$LATENCY_THRESHOLD_MS" ]; then
    echo "CRI OK: ${ELAPSED_MS}ms"
    curl -s "$HEARTBEAT_URL"
  else
    echo "CRI slow: ${ELAPSED_MS}ms (threshold: ${LATENCY_THRESHOLD_MS}ms)"
    exit 1
  fi
else
  echo "CRI API call failed"
  exit 1
fi

Set the Vigilmon heartbeat interval to 5 minutes. CRI latency spikes above 3 seconds typically indicate overlayfs mount contention or image store lock contention during garbage collection — catching this early lets you pause scheduling on the node before kubelet retries accumulate.

Step 3: Track Image Pull Performance

CRI-O delegates image pulls to containers/image, which handles layer decompression, signature verification, and overlay application. Pull failures and slowdowns manifest as pod ErrImagePull events in Kubernetes — often without a clear cause in kubelet logs. Monitor pull performance proactively:

#!/bin/bash
# /usr/local/bin/crio-pull-latency.sh

HEARTBEAT_URL="https://vigilmon.online/heartbeat/ghi789"
TEST_IMAGE="registry.k8s.io/pause:3.9"
LATENCY_THRESHOLD=45  # seconds

# Remove the cached image to force a real network pull
crictl --runtime-endpoint unix:///var/run/crio/crio.sock \
  rmi "$TEST_IMAGE" >/dev/null 2>&1

START=$(date +%s)
if crictl --runtime-endpoint unix:///var/run/crio/crio.sock \
    pull "$TEST_IMAGE" >/dev/null 2>&1; then
  END=$(date +%s)
  ELAPSED=$(( END - START ))

  if [ "$ELAPSED" -lt "$LATENCY_THRESHOLD" ]; then
    echo "Pull OK: ${ELAPSED}s for $TEST_IMAGE"
    curl -s "$HEARTBEAT_URL"
  else
    echo "Pull slow: ${ELAPSED}s (threshold: ${LATENCY_THRESHOLD}s)"
    exit 1
  fi
else
  echo "Pull failed for $TEST_IMAGE"
  exit 1
fi

Schedule this check every 15 minutes with a matching Vigilmon heartbeat interval. Using pause:3.9 (under 700 KB) means pull time reflects network and storage overhead rather than image size. If pulls to your internal registry start taking more than 45 seconds, the problem is typically registry connectivity, DNS resolution, or overlay mount time on a storage-pressured node.

To check what CRI-O currently has cached:

crictl --runtime-endpoint unix:///var/run/crio/crio.sock images

Step 4: Monitor Pod Sandbox Creation Latency

Pod sandbox creation is CRI-O's most complex operation — it involves creating a network namespace, calling CNI plugins, applying seccomp/AppArmor profiles, and setting up the pause container. Sandbox creation latency above 2 seconds typically signals CNI plugin timeout, SELinux label application bottleneck, or namespace setup pressure under high pod churn:

#!/bin/bash
# /usr/local/bin/crio-sandbox-latency.sh

HEARTBEAT_URL="https://vigilmon.online/heartbeat/jkl012"
LATENCY_THRESHOLD=5  # seconds for sandbox creation
SANDBOX_CONFIG="/tmp/crio-sandbox-test.json"

# Create a minimal sandbox config for testing
cat > "$SANDBOX_CONFIG" << 'EOF'
{
  "metadata": {
    "name": "vigilmon-sandbox-test",
    "namespace": "default",
    "attempt": 0,
    "uid": "vigil-test-uid-001"
  },
  "hostname": "sandbox-test",
  "log_directory": "/tmp",
  "linux": {}
}
EOF

START=$(date +%s)
SANDBOX_ID=$(crictl --runtime-endpoint unix:///var/run/crio/crio.sock \
  runp "$SANDBOX_CONFIG" 2>/dev/null)
END=$(date +%s)
ELAPSED=$(( END - START ))

# Clean up the test sandbox immediately
if [ -n "$SANDBOX_ID" ]; then
  crictl --runtime-endpoint unix:///var/run/crio/crio.sock \
    stopp "$SANDBOX_ID" >/dev/null 2>&1
  crictl --runtime-endpoint unix:///var/run/crio/crio.sock \
    rmp "$SANDBOX_ID" >/dev/null 2>&1
fi

if [ -n "$SANDBOX_ID" ] && [ "$ELAPSED" -lt "$LATENCY_THRESHOLD" ]; then
  echo "Sandbox creation OK: ${ELAPSED}s"
  curl -s "$HEARTBEAT_URL"
else
  echo "Sandbox creation slow or failed: ${ELAPSED}s (threshold: ${LATENCY_THRESHOLD}s)"
  exit 1
fi

rm -f "$SANDBOX_CONFIG"

Run this check every 10 minutes. Sandbox creation latency is one of the clearest early indicators of CNI plugin health — network namespace setup dominates the timing, so a 5-second spike usually means your CNI (Calico, Flannel, Cilium) is experiencing connectivity or configuration issues before pods start failing cluster-wide.

Step 5: Monitor CRI-O Storage Driver Health

CRI-O uses either overlay or vfs as its storage driver, configured in /etc/containers/storage.conf. Storage driver failures are silent — CRI-O continues to respond to CRI API calls but container image layers cannot be mounted. Validate the storage driver separately from process health:

#!/bin/bash
# /usr/local/bin/crio-storage-check.sh

HEARTBEAT_URL="https://vigilmon.online/heartbeat/mno345"
STORAGE_ROOT="/var/lib/containers/storage"
DISK_THRESHOLD=82  # percent

# Check disk usage at the CRI-O storage root
DISK_USED=$(df "$STORAGE_ROOT" 2>/dev/null | awk 'NR==2 {gsub(/%/, ""); print $5}')

if [ -z "$DISK_USED" ]; then
  echo "Storage root not mounted: $STORAGE_ROOT"
  exit 1
fi

if [ "$DISK_USED" -ge "$DISK_THRESHOLD" ]; then
  echo "Storage critical: ${DISK_USED}% used at $STORAGE_ROOT"
  exit 1
fi

# Verify overlay mounts are working via a lightweight test
OVERLAY_DIR="$STORAGE_ROOT/overlay"
if [ -d "$OVERLAY_DIR" ]; then
  LAYER_COUNT=$(ls "$OVERLAY_DIR" 2>/dev/null | wc -l)
  echo "Storage OK: ${DISK_USED}% used, ${LAYER_COUNT} overlay layers"
else
  echo "Overlay directory missing — storage driver may have changed"
  exit 1
fi

curl -s "$HEARTBEAT_URL"

Set the Vigilmon heartbeat to 10 minutes. CRI-O's garbage collector can reclaim image layers when disk pressure is detected, but this runs asynchronously and can lag behind rapid disk fill. If disk usage climbs above 82%, new container starts will fail with obscure errors until layers can be written.

To inspect CRI-O storage utilization directly:

# List images and their sizes
crictl images --output json | jq '.images[] | {id: .id, size: .size, repoTags: .repoTags}'

# Check total storage driver usage
du -sh /var/lib/containers/storage/overlay/
du -sh /var/lib/containers/storage/overlay-containers/

Step 6: Set Up Alert Channels

Route CRI-O alerts to the right team with minimal noise:

  1. Go to Alert Channels in Vigilmon.
  2. Add a Slack webhook pointing to your #infra-alerts channel.
  3. Add PagerDuty for critical monitors (socket health, CRI latency).
  4. Set Consecutive failures before alert to 2 for heartbeat monitors — a single missed ping on a loaded node is usually a cron scheduling delay, not a real CRI-O failure.

Add maintenance windows for CRI-O upgrades:

# Pause monitoring before upgrade
curl -X POST https://vigilmon.online/api/maintenance \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -d '{
    "monitor_id": "MONITOR_ID",
    "duration_minutes": 15,
    "reason": "cri-o upgrade"
  }'

# Drain the node first
kubectl drain NODE_NAME --ignore-daemonsets --delete-emptydir-data

# Perform the upgrade
systemctl stop crio
dnf upgrade -y crio  # or apt-get on Debian-based systems
systemctl start crio

# Uncordon after verification
kubectl uncordon NODE_NAME

Summary

| Monitor | Target | What It Catches | |---|---|---| | Cron heartbeat (process) | Socket probe via crictl info | CRI-O crash, socket removal, daemon freeze | | Cron heartbeat (CRI latency) | Timed crictl info call | CRI endpoint slowdown, GC lock contention | | Cron heartbeat (image pull) | Timed pull of pause:3.9 | Registry connectivity, overlay mount latency | | Cron heartbeat (sandbox) | Timed crictl runp + cleanup | CNI plugin timeout, namespace setup pressure | | Cron heartbeat (storage) | Disk usage + overlay dir check | Disk full, storage driver misconfiguration |

CRI-O's deliberate minimalism means fewer moving parts but also fewer built-in observability hooks. Vigilmon closes that gap with external checks that validate the full path from CRI socket to image pull to pod sandbox — so you know about CRI-O problems before Kubernetes starts reporting ContainerCreating across your node fleet.

Start monitoring for free at vigilmon.online.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles