Monitoring OverlayBD with Vigilmon: Keeping Block-Level Lazy Container Pulling Healthy

OverlayBD is a block-device based container image format and snapshotter from Alibaba Cloud that enables on-demand loading of container layers at the block level. Unlike file-based lazy pulling solutions, OverlayBD presents a virtual block device for each container layer, allowing the guest OS to page in only the blocks actually read at runtime. This dramatically reduces container cold-start times for large images while maintaining full POSIX filesystem semantics. Teams use OverlayBD in cloud-native environments where startup latency directly affects auto-scaling responsiveness and job throughput. When OverlayBD fails — the snapshotter daemon crashes, the block device backend becomes unavailable, or containerd loses its connection to the snapshotter — containers either fail to start or fall back to full-layer downloading. Vigilmon gives you external monitoring that watches OverlayBD health and alerts before cold-start regressions reach end users.

What You'll Build

• A heartbeat monitor validating the OverlayBD snapshotter daemon is running
• Vigilmon HTTP monitors for the registry endpoints OverlayBD pulls blocks from
• A block device probe that validates the snapshotter can provision devices
• Alert channels routed to your infrastructure team

Prerequisites

• OverlayBD installed with overlaybd-tcmu daemon and overlaybd-snapshotter plugin configured
• A registry serving OverlayBD-format images (typically with overlaybd-snapshotter-convertor)
• containerd configured to use the OverlayBD snapshotter
• A free account at vigilmon.online

Why Monitoring OverlayBD Matters

OverlayBD introduces a unique infrastructure dependency: the kernel TCMU (Target Core Module Userspace) subsystem, which provides the virtual block device interface. This dependency stack is longer than file-based snapshotters, creating more failure points:

overlaybd-tcmu daemon crash is the most common failure mode. Without the TCMU daemon, the block device backend is unavailable and containerd cannot provision OverlayBD snapshots. Containers using OverlayBD images fail to start with cryptic block device errors, while the connection to containerd was lost silently.

TCMU kernel module unavailability on certain kernel versions or after kernel upgrades causes the overlaybd-tcmu daemon to fail at startup. The daemon may appear to run but fail to register the TCMU handler, causing all block device operations to fail.

Registry connectivity loss prevents block range fetches. Unlike traditional pulls, OverlayBD fetches specific block ranges as containers read them — if the registry becomes unavailable mid-execution, running containers can experience I/O errors, not just startup failures.

Block device provisioning failures can occur when the system runs low on loop device slots, has insufficient tmpfs space for block device metadata, or when the snapshotter's working directory fills up.

External monitoring with Vigilmon catches both daemon-level failures and the silent performance regressions that happen when block fetching degrades.

Step 1: Monitor the OCI Registry Endpoint

OverlayBD fetches layer blocks from an OCI-compliant registry. Monitor the registry API and range request support — critical for OverlayBD's block-level fetching:

# Verify registry /v2/ is alive
curl -I https://registry.yourdomain.com/v2/
# Expected: 200 OK or 401 Unauthorized

# Verify range request support (OverlayBD requires this for block fetching)
curl -I -H "Range: bytes=0-4095" \
  https://registry.yourdomain.com/v2/myorg/myimage/blobs/sha256:LAYER_DIGEST
# Expected: HTTP/1.1 206 Partial Content

In the Vigilmon dashboard:

1. Go to Add Monitor → HTTP
2. Configure:

| Field | Value | |---|---| | Name | OverlayBD registry /v2/ | | URL | https://registry.yourdomain.com/v2/ | | Method | GET | | Expected status | 200, 401 | | Check interval | 1 minute | | Timeout | 10 seconds | | Alert after | 2 consecutive failures |

Step 2: Heartbeat Monitor for overlaybd-tcmu Daemon

The overlaybd-tcmu daemon is the core process. Probe it and send a heartbeat to Vigilmon:

#!/bin/bash
# overlaybd-health-check.sh
# Run on the host via systemd timer

HEARTBEAT_URL="${VIGILMON_OVERLAYBD_HEARTBEAT_URL}"

# Check 1: overlaybd-tcmu daemon is running
if ! systemctl is-active --quiet overlaybd-tcmu; then
    echo "ERROR: overlaybd-tcmu daemon is not active" >&2
    exit 1
fi

# Check 2: overlaybd-snapshotter socket exists
SOCKET_PATH="/run/overlaybd-snapshotter/overlaybd-snapshotter.sock"
if [ ! -S "$SOCKET_PATH" ]; then
    echo "ERROR: overlaybd-snapshotter socket missing at $SOCKET_PATH" >&2
    exit 1
fi

# Check 3: TCMU subsystem is loaded
if ! lsmod | grep -q "target_core_user"; then
    echo "ERROR: target_core_user kernel module not loaded (TCMU unavailable)" >&2
    exit 1
fi

# Check 4: containerd recognizes the snapshotter
SNAPSHOTTER_STATUS=$(ctr plugins ls 2>/dev/null | grep "overlaybd" | awk '{print $NF}')
if [ -z "$SNAPSHOTTER_STATUS" ]; then
    echo "ERROR: overlaybd snapshotter not visible in containerd plugins" >&2
    exit 1
fi

if [ "$SNAPSHOTTER_STATUS" != "ok" ]; then
    echo "ERROR: overlaybd snapshotter status: $SNAPSHOTTER_STATUS" >&2
    exit 1
fi

curl -fsS --max-time 10 "$HEARTBEAT_URL" > /dev/null
echo "OverlayBD health check passed, heartbeat sent"

Deploy as a systemd timer on each node:

# /etc/systemd/system/overlaybd-health-check.service
[Unit]
Description=OverlayBD health check for Vigilmon
After=overlaybd-tcmu.service

[Service]
Type=oneshot
ExecStart=/usr/local/bin/overlaybd-health-check.sh
Environment=VIGILMON_OVERLAYBD_HEARTBEAT_URL=https://vigilmon.online/heartbeat/YOUR_TOKEN

# /etc/systemd/system/overlaybd-health-check.timer
[Unit]
Description=Run OverlayBD health check every 5 minutes

[Timer]
OnBootSec=3min
OnUnitActiveSec=5min
Unit=overlaybd-health-check.service

[Install]
WantedBy=timers.target

Enable:

systemctl daemon-reload
systemctl enable --now overlaybd-health-check.timer

Create the heartbeat monitor in Vigilmon:

1. Add Monitor → Heartbeat
2. Name: overlaybd-tcmu daemon health
3. Expected interval: 5 minutes
4. Grace period: 5 minutes

Step 3: Block Device Provisioning Probe

The deepest health signal for OverlayBD is whether the snapshotter can actually provision a block device. Add a probe that creates a test snapshot and validates block device provisioning:

# overlaybd-provision-probe.yaml
apiVersion: batch/v1
kind: CronJob
metadata:
  name: overlaybd-provision-probe
  namespace: monitoring
spec:
  schedule: "*/10 * * * *"
  jobTemplate:
    spec:
      template:
        spec:
          restartPolicy: OnFailure
          hostPID: true
          tolerations:
            - operator: Exists
          containers:
            - name: provision-probe
              image: alpine:latest
              securityContext:
                privileged: true
              env:
                - name: HEARTBEAT_URL
                  valueFrom:
                    secretKeyRef:
                      name: vigilmon-secrets
                      key: overlaybd-provision-heartbeat-url
                - name: TEST_IMAGE
                  value: "registry.yourdomain.com/monitoring/overlaybd-probe:latest"
              command:
                - /bin/sh
                - -c
                - |
                  apk add --quiet curl

                  START=$(date +%s)

                  # Pull a small OverlayBD test image using the overlaybd snapshotter
                  if ! ctr images pull --snapshotter overlaybd "$TEST_IMAGE" 2>&1; then
                    echo "ERROR: OverlayBD test image pull failed" >&2
                    exit 1
                  fi

                  END=$(date +%s)
                  ELAPSED=$((END - START))

                  # Verify the image is accessible
                  if ! ctr images check "$TEST_IMAGE" 2>/dev/null | grep -q "complete"; then
                    echo "ERROR: OverlayBD image check incomplete" >&2
                    ctr images rm "$TEST_IMAGE" 2>/dev/null || true
                    exit 1
                  fi

                  # Clean up
                  ctr images rm "$TEST_IMAGE" 2>/dev/null || true

                  echo "OverlayBD provisioning OK in ${ELAPSED}s"
                  curl -fsS --max-time 10 "$HEARTBEAT_URL" > /dev/null
                  echo "Provisioning heartbeat sent"
          volumes:
            - name: run
              hostPath:
                path: /run

Create the heartbeat monitor in Vigilmon:

1. Add Monitor → Heartbeat
2. Name: overlaybd block device provisioning
3. Expected interval: 10 minutes
4. Grace period: 5 minutes

Step 4: Monitor OverlayBD Working Directory Space

OverlayBD stores block device metadata and layer indexes in a working directory (typically /var/lib/overlaybd). If this fills up, new containers cannot start:

#!/bin/bash
# overlaybd-disk-check.sh
HEARTBEAT_URL="${VIGILMON_OVERLAYBD_DISK_HEARTBEAT_URL}"
OVERLAYBD_DIR="/var/lib/overlaybd"
THRESHOLD_PERCENT=85

USAGE=$(df "$OVERLAYBD_DIR" | awk 'NR==2 {print $5}' | tr -d '%')

if [ "$USAGE" -gt "$THRESHOLD_PERCENT" ]; then
    echo "ERROR: OverlayBD working directory usage at ${USAGE}% (threshold: ${THRESHOLD_PERCENT}%)" >&2
    exit 1
fi

echo "OverlayBD working directory at ${USAGE}% utilization"
curl -fsS --max-time 10 "$HEARTBEAT_URL" > /dev/null
echo "Disk health heartbeat sent"

Add to the systemd service or run as a separate timer with 15-minute interval. Create the heartbeat monitor in Vigilmon:

1. Add Monitor → Heartbeat
2. Name: overlaybd working directory space
3. Expected interval: 15 minutes
4. Grace period: 10 minutes

Step 5: DaemonSet-Based Node Coverage

Deploy a DaemonSet to provide per-node monitoring across all worker nodes running OverlayBD:

# overlaybd-node-probe.yaml
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: overlaybd-node-probe
  namespace: monitoring
spec:
  selector:
    matchLabels:
      app: overlaybd-node-probe
  template:
    metadata:
      labels:
        app: overlaybd-node-probe
    spec:
      tolerations:
        - operator: Exists
      containers:
        - name: probe
          image: alpine:latest
          command:
            - /bin/sh
            - -c
            - |
              apk add --quiet curl

              while true; do
                SOCKET="/run/overlaybd-snapshotter/overlaybd-snapshotter.sock"
                OVERLAYBD_PID=$(pgrep overlaybd-tcmu 2>/dev/null || true)

                if [ -S "$SOCKET" ] && [ -n "$OVERLAYBD_PID" ]; then
                  echo "$(date): OverlayBD healthy on node $NODE_NAME"
                  curl -fsS --max-time 10 "$VIGILMON_HEARTBEAT_URL" > /dev/null && \
                    echo "$(date): heartbeat sent"
                else
                  echo "$(date): ERROR - overlaybd-tcmu not running on $NODE_NAME"
                fi
                sleep 300
              done
          env:
            - name: VIGILMON_HEARTBEAT_URL
              valueFrom:
                secretKeyRef:
                  name: vigilmon-secrets
                  key: overlaybd-heartbeat-url
            - name: NODE_NAME
              valueFrom:
                fieldRef:
                  fieldPath: spec.nodeName
          volumeMounts:
            - name: run
              mountPath: /run
              readOnly: true
      volumes:
        - name: run
          hostPath:
            path: /run

Step 6: Configure Alert Channels

Route OverlayBD alerts to your infrastructure team:

1. In Vigilmon: Alert Channels → Add Channel → Slack Webhook
2. Add your #infrastructure webhook URL
3. Assign to all OverlayBD monitors

For daemon crashes and provisioning failures (direct container start failures):

1. Alert Channels → Add Channel → PagerDuty or email
2. Add your platform on-call rotation
3. Assign to overlaybd-tcmu daemon health and overlaybd block device provisioning heartbeats

Create the Kubernetes secrets:

kubectl create secret generic vigilmon-secrets \
  --from-literal=overlaybd-heartbeat-url='https://vigilmon.online/heartbeat/YOUR_DAEMON_TOKEN' \
  --from-literal=overlaybd-provision-heartbeat-url='https://vigilmon.online/heartbeat/YOUR_PROVISION_TOKEN' \
  --from-literal=overlaybd-disk-heartbeat-url='https://vigilmon.online/heartbeat/YOUR_DISK_TOKEN' \
  -n monitoring

What You're Now Monitoring

| Component | Monitor Type | What It Detects | |---|---|---| | OCI registry | HTTP GET /v2/ | Registry down, block range fetches will fail | | overlaybd-tcmu daemon | Heartbeat (5 min) | Daemon crashed, TCMU module missing, socket unavailable | | Block device provisioning | Heartbeat (10 min) | Snapshotter cannot create devices (full stack validation) | | Working directory space | Heartbeat (15 min) | Disk pressure blocking new container starts | | Per-node DaemonSet | Heartbeat (5 min) | Node-level daemon health across all workers |

Conclusion

OverlayBD's block-device abstraction makes it uniquely powerful for large-image workloads, but the dependency on the TCMU kernel subsystem means failure modes are lower-level and harder to debug than file-based snapshotters. The most dangerous scenario — the overlaybd-tcmu daemon crashing silently while containerd appears healthy — is exactly what external monitoring catches. By validating the full stack (daemon health, socket availability, TCMU kernel module, and actual provisioning), Vigilmon ensures you know about OverlayBD failures before they cascade into container scheduling failures.

Get started free at vigilmon.online. The daemon heartbeat and registry probe together give you the two most critical OverlayBD signals in under fifteen minutes: is the TCMU daemon running, and can the registry serve block ranges when containers need them?