tutorial

Monitoring Nerdctl and Containerd-Backed Services with Vigilmon

Nerdctl brings Docker-compatible workflows to containerd — but containerd socket failures, compose service drift, snapshot store issues, and build pipeline stalls can leave your services silently broken. Here's how to monitor nerdctl service health and containerd socket availability with Vigilmon.

Nerdctl is a Docker-compatible CLI for containerd that gives you familiar docker-style commands — nerdctl run, nerdctl compose up, nerdctl build — while running directly against the containerd runtime. It's the default CLI for Lima-based local container environments on macOS and a popular choice on Linux servers where Docker daemon overhead is unwanted. When containerd's socket becomes unavailable, nerdctl compose services drift out of sync, or the build cache snapshot store fills up, nerdctl commands fail silently with confusing errors and running services become unobservable. Vigilmon gives you external monitoring for the containerd socket nerdctl depends on, compose service health, and snapshot store metrics so you catch infrastructure failures before your container workflows break.

What You'll Set Up

  • Containerd socket availability monitoring (nerdctl's dependency)
  • Compose service health checks for nerdctl-managed services
  • Snapshot store utilization alerts
  • Build pipeline connectivity checks
  • Namespace isolation health verification

Prerequisites

  • Nerdctl installed (nerdctl --version)
  • Containerd running (systemctl status containerd or equivalent)
  • At least one service running with nerdctl compose up
  • A free Vigilmon account

Step 1: Monitor the Containerd Socket Nerdctl Depends On

Every nerdctl command connects to containerd via a Unix socket — by default /run/containerd/containerd.sock. When this socket goes away (containerd crash, permissions change, apparmor policy), every nerdctl command fails immediately. Monitor socket availability as the foundational health signal:

  1. In Vigilmon, click Add Monitor → Cron Heartbeat.
  2. Set Expected ping interval to 2 minutes.
  3. Copy the heartbeat URL (e.g., https://vigilmon.online/heartbeat/abc123).

Create the socket health script:

#!/bin/bash
# /usr/local/bin/nerdctl-socket-check.sh

HEARTBEAT_URL="https://vigilmon.online/heartbeat/abc123"

# Default containerd socket location
SOCKET="/run/containerd/containerd.sock"

# For rootless nerdctl, use the user-scoped socket
# SOCKET="$XDG_RUNTIME_DIR/containerd/containerd.sock"

if [ ! -S "$SOCKET" ]; then
  echo "containerd socket missing: $SOCKET"
  exit 1
fi

# Test that nerdctl can actually reach containerd (not just socket existence)
if nerdctl --address "$SOCKET" info >/dev/null 2>&1; then
  echo "nerdctl socket OK: $SOCKET"
  curl -s "$HEARTBEAT_URL"
else
  echo "containerd socket exists but nerdctl cannot connect"
  exit 1
fi

Install the cron job:

chmod +x /usr/local/bin/nerdctl-socket-check.sh
(crontab -l 2>/dev/null; echo "*/2 * * * * /usr/local/bin/nerdctl-socket-check.sh >> /var/log/nerdctl-health.log 2>&1") | crontab -

For rootless nerdctl running as a specific user, run this cron job under that user's crontab rather than root's, so the $XDG_RUNTIME_DIR resolves correctly.

Step 2: Monitor Compose Service Health

nerdctl compose manages multi-container applications with the same compose file format as Docker Compose. When individual services crash or restart-loop, nerdctl considers them "running" (at the compose orchestration level) even as traffic fails. Add HTTP health monitoring for each compose service that exposes an endpoint:

  1. In Vigilmon, go to Add Monitor → HTTP / HTTPS.
  2. Set the URL to your service's health endpoint, e.g. https://your-server.example.com/health.
  3. Set check interval to 1 minute.
  4. Set expected status code to 200.
  5. Save the monitor.

Also add a heartbeat from inside each critical compose service to detect restart loops:

# docker-compose.yml (works with nerdctl compose)
services:
  web:
    image: your-app:latest
    ports:
      - "3000:3000"
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:3000/health"]
      interval: 30s
      timeout: 5s
      retries: 3
    restart: unless-stopped

  worker:
    image: your-worker:latest
    environment:
      VIGILMON_HEARTBEAT: "https://vigilmon.online/heartbeat/def456"
    command: >
      sh -c "while true; do
        ./worker-task
        curl -s $$VIGILMON_HEARTBEAT
        sleep 60
      done"

A heartbeat from inside the worker means you'll know within 2 minutes if it stops processing — regardless of whether the container appears healthy at the compose level.

Step 3: Track Snapshot Store Utilization

Nerdctl stores container images and layers in containerd's snapshot store — by default under /var/lib/containerd for root mode or ~/.local/share/containerd for rootless. When this fills up, nerdctl pull and nerdctl build fail with obscure storage errors. Monitor utilization proactively:

#!/bin/bash
# /usr/local/bin/nerdctl-storage-check.sh

HEARTBEAT_URL="https://vigilmon.online/heartbeat/ghi789"
DISK_THRESHOLD=80  # percent

# Detect rootless vs root mode
if [ "$(id -u)" -eq 0 ]; then
  STORAGE_PATH="/var/lib/containerd"
else
  STORAGE_PATH="${XDG_DATA_HOME:-$HOME/.local/share}/containerd"
fi

if [ ! -d "$STORAGE_PATH" ]; then
  echo "Snapshot store not found at $STORAGE_PATH"
  exit 1
fi

DISK_USED=$(df "$STORAGE_PATH" | awk 'NR==2 {gsub(/%/, ""); print $5}')
IMAGE_COUNT=$(nerdctl images -q 2>/dev/null | wc -l)

if [ "$DISK_USED" -lt "$DISK_THRESHOLD" ]; then
  echo "nerdctl storage OK: ${DISK_USED}% used, ${IMAGE_COUNT} images"
  curl -s "$HEARTBEAT_URL"
else
  echo "nerdctl storage critical: ${DISK_USED}% used at $STORAGE_PATH"
  exit 1
fi

Set the Vigilmon heartbeat interval to 15 minutes. Build caches accumulate faster with nerdctl than with Docker because containerd's garbage collection is more conservative — dangling build layers persist until you run nerdctl system prune. When storage fills, nerdctl build starts failing mid-layer with confusing "write: no space left on device" errors inside the buildkit daemon.

To identify what's consuming space:

# Show image sizes
nerdctl images --format "table {{.Repository}}:{{.Tag}}\t{{.Size}}" | sort -k2 -rh | head -20

# Show container disk usage
nerdctl system df

# Clean up dangling images and build cache
nerdctl system prune --volumes

Step 4: Monitor Build Pipeline Connectivity

Nerdctl uses BuildKit for image builds (nerdctl build). BuildKit runs as a separate daemon (buildkitd) with its own socket. When BuildKit is unavailable, nerdctl build fails immediately — but this is separate from the containerd socket failure. Track build pipeline health independently:

#!/bin/bash
# /usr/local/bin/nerdctl-buildkit-check.sh

HEARTBEAT_URL="https://vigilmon.online/heartbeat/jkl012"

# Check BuildKit socket
BUILDKIT_SOCKET="/run/buildkit/buildkitd.sock"

# For rootless mode
if [ "$(id -u)" -ne 0 ]; then
  BUILDKIT_SOCKET="${XDG_RUNTIME_DIR}/buildkit/buildkitd.sock"
fi

if [ ! -S "$BUILDKIT_SOCKET" ]; then
  # BuildKit not running — nerdctl build will fail
  echo "BuildKit socket missing: $BUILDKIT_SOCKET"
  exit 1
fi

# Test with a minimal build to verify end-to-end functionality
BUILD_DIR=$(mktemp -d)
cat > "$BUILD_DIR/Dockerfile" << 'EOF'
FROM scratch
COPY /dev/null /
EOF

if nerdctl build --no-cache -t vigilmon-build-test:latest "$BUILD_DIR" >/dev/null 2>&1; then
  nerdctl rmi vigilmon-build-test:latest >/dev/null 2>&1
  rm -rf "$BUILD_DIR"
  echo "nerdctl build pipeline OK"
  curl -s "$HEARTBEAT_URL"
else
  rm -rf "$BUILD_DIR"
  echo "nerdctl build pipeline failed"
  exit 1
fi

Create a Vigilmon heartbeat with a 15 minute expected interval for this check. BuildKit daemon crashes are more common than containerd crashes because BuildKit has no automatic restart — if it dies, nerdctl build fails until an operator restarts buildkitd manually.

Step 5: Verify Namespace Isolation Health

Nerdctl uses containerd namespaces to isolate resources — the default namespace is default for root mode and nerdctl-1000 (based on UID) for rootless. Namespace corruption or unexpected namespace mixing can cause containers from different projects to be visible to each other or for services to disappear from nerdctl ps while still running:

#!/bin/bash
# /usr/local/bin/nerdctl-namespace-check.sh

HEARTBEAT_URL="https://vigilmon.online/heartbeat/mno345"

# List all containerd namespaces
NAMESPACE_COUNT=$(nerdctl namespace ls 2>/dev/null | tail -n +2 | wc -l)

if [ "$NAMESPACE_COUNT" -eq 0 ]; then
  echo "No namespaces found — containerd may be unreachable"
  exit 1
fi

# Check that default namespace is present and healthy
DEFAULT_NS=$(nerdctl namespace ls 2>/dev/null | grep "^default" || echo "")

if [ -z "$DEFAULT_NS" ]; then
  echo "Default namespace missing from containerd"
  exit 1
fi

# Cross-check: containers in nerdctl ps should match containers in containerd namespace
NERDCTL_CONTAINERS=$(nerdctl ps -q 2>/dev/null | wc -l)
CONTAINERD_CONTAINERS=$(ctr --namespace default containers ls 2>/dev/null | tail -n +2 | wc -l)

echo "Namespace OK: ${NAMESPACE_COUNT} namespaces, ${NERDCTL_CONTAINERS} nerdctl containers"
curl -s "$HEARTBEAT_URL"

Set the Vigilmon heartbeat interval to 10 minutes. Namespace consistency issues in containerd are uncommon but they do occur after abrupt host shutdowns, and the symptom — containers that exist in one view but not another — is extremely confusing without this cross-check.

Step 6: Configure Alert Channels

Configure alerts that match the severity of each nerdctl monitoring check:

  1. Go to Alert Channels in Vigilmon.
  2. Add a Slack webhook to your #dev-infra or #deploys channel for compose service health failures.
  3. Add email for storage utilization alerts — these are slow-burn issues you want to notice before they become urgent.
  4. Set Consecutive failures before alert to 2 for socket checks to avoid spurious alerts from cron timing.

Pause monitoring during planned maintenance:

# Pause before restarting containerd
curl -X POST https://vigilmon.online/api/maintenance \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -d '{"monitor_id": "MONITOR_ID", "duration_minutes": 5}'

# Restart containerd (and by extension, pause any nerdctl compose services)
systemctl restart containerd

# Bring compose services back up
nerdctl compose up -d

For rootless nerdctl, ensure the cron jobs run as the correct user and that $XDG_RUNTIME_DIR is set (rootless containerd sockets are user-scoped, so root's crontab cannot reach them).

Summary

| Monitor | Target | What It Catches | |---|---|---| | Cron heartbeat (socket) | nerdctl info connectivity | Containerd crash, socket permission change | | HTTP monitor or heartbeat | Compose service health endpoint | Service crash, restart loop, dependency failure | | Cron heartbeat (storage) | /var/lib/containerd disk usage | Snapshot store full, image cache growth | | Cron heartbeat (buildkit) | BuildKit socket + test build | buildkitd crash, build pipeline failure | | Cron heartbeat (namespace) | Namespace list consistency | Namespace corruption, container visibility issues |

Nerdctl brings Docker ergonomics to containerd — but it inherits containerd's failure modes and adds a few of its own via BuildKit and namespace management. Vigilmon's external checks give you visibility into the full stack that nerdctl ps and compose status commands cannot provide, so socket failures, full disks, and dead build pipelines surface as alerts rather than mysterious command-line errors.

Start monitoring for free at vigilmon.online.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles