tutorial

Monitoring Containerd with Vigilmon

Containerd is the industry-standard container runtime underneath Kubernetes and Docker — but shim crashes, snapshot store failures, and CRI API timeouts can silently break your entire container workload. Here's how to monitor containerd health, image pull latency, and storage utilization with Vigilmon.

Containerd is the CNCF-graduated container runtime that powers Kubernetes nodes, Docker Desktop, and most modern container platforms. It sits between your orchestrator and the Linux kernel — managing image pulls, container lifecycle via shims, snapshot storage, and the CRI gRPC API. When containerd stalls, every container on that node silently stops starting, and Kubernetes marks pods as ContainerCreating indefinitely. Vigilmon gives you external monitoring for containerd health, shim availability, storage utilization, and CRI responsiveness so you catch runtime failures before they cascade into cluster-wide outages.

What You'll Set Up

  • Containerd process and socket health via cron heartbeats
  • CRI API responsiveness monitoring
  • Snapshot storage utilization alerts
  • Image pull latency tracking
  • Shim process health checks

Prerequisites

  • Containerd 1.6+ running on Linux (systemctl status containerd)
  • ctr CLI available (containerd package)
  • crictl CLI available (for CRI checks)
  • A free Vigilmon account

Step 1: Monitor Containerd Process Health with a Heartbeat

Containerd runs as a systemd service and exposes a Unix socket at /run/containerd/containerd.sock. The most reliable first signal is a heartbeat that tests actual socket connectivity — not just whether the process is in the systemd running state:

  1. In Vigilmon, click Add Monitor → Cron Heartbeat.
  2. Set Expected ping interval to 2 minutes.
  3. Copy the heartbeat URL (e.g., https://vigilmon.online/heartbeat/abc123).

Create the health check script:

#!/bin/bash
# /usr/local/bin/containerd-health-check.sh

SOCKET="/run/containerd/containerd.sock"
HEARTBEAT_URL="https://vigilmon.online/heartbeat/abc123"

# Check that containerd socket exists and is a socket file
if [ ! -S "$SOCKET" ]; then
  echo "containerd socket missing: $SOCKET"
  exit 1
fi

# Test CRI API responds via crictl
if crictl --runtime-endpoint "unix://$SOCKET" info >/dev/null 2>&1; then
  curl -s "$HEARTBEAT_URL"
else
  echo "containerd socket exists but CRI API is unresponsive"
  exit 1
fi

Install and enable the cron job:

chmod +x /usr/local/bin/containerd-health-check.sh

# Add to root crontab (containerd socket requires root or containerd group)
(crontab -l 2>/dev/null; echo "*/2 * * * * /usr/local/bin/containerd-health-check.sh >> /var/log/containerd-health.log 2>&1") | crontab -

If the heartbeat stops firing, Vigilmon alerts after 2 minutes — you'll know containerd has frozen or crashed well before your orchestrator's pod backoff timers give up.

Step 2: Monitor the CRI gRPC API Responsiveness

Containerd exposes its CRI API over a Unix gRPC socket used by Kubernetes kubelets. Even when the containerd process is running, the CRI API can become unresponsive under heavy snapshot I/O or plugin failures. Track CRI latency separately from process liveness:

#!/bin/bash
# /usr/local/bin/containerd-cri-latency.sh

HEARTBEAT_URL="https://vigilmon.online/heartbeat/def456"
LATENCY_THRESHOLD_MS=2000  # Alert if CRI takes more than 2 seconds

START=$(date +%s%N)
if crictl --runtime-endpoint unix:///run/containerd/containerd.sock \
    --timeout 5s info >/dev/null 2>&1; then
  END=$(date +%s%N)
  ELAPSED_MS=$(( (END - START) / 1000000 ))

  if [ "$ELAPSED_MS" -lt "$LATENCY_THRESHOLD_MS" ]; then
    curl -s "$HEARTBEAT_URL"
  else
    echo "CRI API slow: ${ELAPSED_MS}ms (threshold: ${LATENCY_THRESHOLD_MS}ms)"
    exit 1
  fi
else
  echo "CRI API call failed"
  exit 1
fi

Create a Vigilmon heartbeat with a 5 minute expected interval for this check. Slow CRI responses (>2s) typically indicate snapshot store I/O pressure or overlayfs mount issues — catching the slowdown early lets you drain the node before pods fail to schedule.

Step 3: Track Snapshot Storage Utilization

Containerd stores container layers and image data in its snapshot store — typically at /var/lib/containerd. When this fills up, image pulls silently fail and new containers cannot start. Monitor storage before it becomes critical:

#!/bin/bash
# /usr/local/bin/containerd-storage-check.sh

HEARTBEAT_URL="https://vigilmon.online/heartbeat/ghi789"
STORAGE_PATH="/var/lib/containerd"
DISK_THRESHOLD=80  # percent

DISK_USED=$(df "$STORAGE_PATH" | awk 'NR==2 {gsub(/%/, ""); print $5}')

if [ "$DISK_USED" -lt "$DISK_THRESHOLD" ]; then
  # Also check containerd's own view of snapshots
  SNAPSHOT_COUNT=$(ctr snapshots ls 2>/dev/null | wc -l)
  echo "Storage OK: ${DISK_USED}% used, ${SNAPSHOT_COUNT} snapshots"
  curl -s "$HEARTBEAT_URL"
else
  echo "Storage critical: ${DISK_USED}% used at $STORAGE_PATH"
  exit 1
fi

Run this check every 10 minutes and set the Vigilmon heartbeat interval to 15 minutes. Containerd's snapshot garbage collection only runs when images are explicitly deleted — disk fill is a common failure mode on nodes that pull many image versions over time.

To inspect what's consuming the most space:

# List image sizes
ctr images ls | awk '{print $1, $4}' | sort -k2 -rh | head -20

# Check snapshot store size directly
du -sh /var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/

Step 4: Monitor Image Pull Latency

Slow image pulls are often the first sign of registry connectivity issues, overlay filesystem pressure, or snapshotter plugin degradation. Add a periodic image pull test to detect latency before it blocks pod startup:

#!/bin/bash
# /usr/local/bin/containerd-pull-latency.sh

HEARTBEAT_URL="https://vigilmon.online/heartbeat/jkl012"
TEST_IMAGE="registry.k8s.io/pause:3.9"  # Small, official image
LATENCY_THRESHOLD=30  # seconds

# Remove any existing copy to force a real pull
ctr images rm "$TEST_IMAGE" >/dev/null 2>&1

START=$(date +%s)
if ctr images pull "$TEST_IMAGE" >/dev/null 2>&1; then
  END=$(date +%s)
  ELAPSED=$(( END - START ))

  if [ "$ELAPSED" -lt "$LATENCY_THRESHOLD" ]; then
    echo "Pull OK: ${ELAPSED}s for $TEST_IMAGE"
    curl -s "$HEARTBEAT_URL"
  else
    echo "Pull slow: ${ELAPSED}s (threshold: ${LATENCY_THRESHOLD}s)"
    exit 1
  fi
else
  echo "Pull failed for $TEST_IMAGE"
  exit 1
fi

# Clean up after the test
ctr images rm "$TEST_IMAGE" >/dev/null 2>&1

Create a Vigilmon heartbeat with a 15 minute expected interval. Use a small well-known image like pause or busybox so pull time reflects infrastructure latency rather than image size. If pulls start taking more than 30 seconds for a tiny image, investigate registry connectivity and snapshot I/O before pods begin failing to schedule.

Step 5: Monitor Containerd Shim Health

Each running container has a containerd-shim process (containerd-shim-runc-v2) that manages its lifecycle independently of the main containerd daemon. Shim crashes orphan containers — they disappear from ctr containers ls but leave behind zombie processes. Track the shim count against your expected container count:

#!/bin/bash
# /usr/local/bin/containerd-shim-check.sh

HEARTBEAT_URL="https://vigilmon.online/heartbeat/mno345"
MIN_EXPECTED_SHIMS=0  # Set to your minimum expected container count

SHIM_COUNT=$(pgrep -c containerd-shim 2>/dev/null || echo 0)
CONTAINER_COUNT=$(ctr containers ls 2>/dev/null | tail -n +2 | wc -l)

# Check for orphaned shims (shims > containers)
if [ "$SHIM_COUNT" -gt "$((CONTAINER_COUNT + 5))" ]; then
  echo "Orphaned shims detected: ${SHIM_COUNT} shims, ${CONTAINER_COUNT} containers"
  exit 1
fi

echo "Shims OK: ${SHIM_COUNT} shims for ${CONTAINER_COUNT} containers"
curl -s "$HEARTBEAT_URL"

Run this check every 5 minutes. Orphaned shims indicate failed container cleanup — usually caused by a brief containerd freeze or a shim crash during container exit. Left unchecked, orphaned shims accumulate and consume PIDs until fork calls start failing system-wide.

Step 6: Set Up Alert Channels

Configure Vigilmon to deliver alerts where your on-call team will actually see them:

  1. Go to Alert Channels in Vigilmon.
  2. Add a Slack webhook pointing to your #infra-alerts or #on-call channel.
  3. Add a PagerDuty or email integration for critical monitors (process health, CRI responsiveness).
  4. Set Consecutive failures before alert to 2 for heartbeat monitors — transient cron delays can cause a single missed ping without a real containerd failure.

Add maintenance windows before planned containerd upgrades:

# Pause monitoring during upgrade
curl -X POST https://vigilmon.online/api/maintenance \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -d '{
    "monitor_id": "MONITOR_ID",
    "duration_minutes": 10,
    "reason": "containerd upgrade"
  }'

# Perform the upgrade
systemctl stop containerd
apt-get install --only-upgrade containerd.io
systemctl start containerd

# Maintenance window expires automatically after 10 minutes

Summary

| Monitor | Target | What It Catches | |---|---|---| | Cron heartbeat (process) | Socket connectivity via crictl info | Containerd crash, socket removal, daemon freeze | | Cron heartbeat (CRI latency) | Timed crictl info call | CRI API slowdown, snapshot I/O pressure | | Cron heartbeat (storage) | /var/lib/containerd disk usage | Disk full, snapshot store growth | | Cron heartbeat (pull latency) | Timed image pull of pause | Registry connectivity, overlay I/O degradation | | Cron heartbeat (shims) | Shim vs container count delta | Orphaned shims, failed container cleanup |

Containerd is deliberately low-visibility by design — it does its job silently and only surfaces errors when something asks the CRI API a question. Vigilmon closes that visibility gap with lightweight external checks that catch frozen daemons, full disks, and CRI slowdowns before Kubernetes starts reporting ContainerCreating across your entire node.

Start monitoring for free at vigilmon.online.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles