Monitoring Kata Containers with Vigilmon

Kata Containers provide VM-level isolation for container workloads by launching each pod in a dedicated lightweight virtual machine — using QEMU, Cloud Hypervisor, or Firecracker as the hypervisor and the Kata agent for container management inside the VM. The isolation is strong, but the failure surface is wider than a standard runc container: the hypervisor can crash, the microVM can fail to boot within its timeout, the Kata agent inside the VM can become unreachable, or resource overhead can accumulate and starve the host. Vigilmon gives you external monitoring for Kata runtime health, microVM boot latency, hypervisor availability, and agent responsiveness so you catch VM-based container failures before they block your workloads.

What You'll Set Up

Kata runtime process health via cron heartbeats
MicroVM boot latency monitoring
Hypervisor health and crash detection
Kata agent responsiveness checks
Resource overhead tracking (CPU, memory)

Prerequisites

Kata Containers 3.x installed (kata-runtime available on PATH)
Containerd or Kubernetes configured to use the Kata runtime class
At least one workload running via Kata for testing
A free Vigilmon account

Step 1: Monitor Kata Runtime Process Health

Kata Containers launch a hypervisor process and a kata-agent for each running sandbox. When a hypervisor crashes, the container appears running at the Kubernetes level but is completely unresponsive. Track the hypervisor and shim processes as a health signal:

In Vigilmon, click Add Monitor → Cron Heartbeat.
Set Expected ping interval to 2 minutes.
Copy the heartbeat URL (e.g., https://vigilmon.online/heartbeat/abc123).

Create the process health script:

#!/bin/bash
# /usr/local/bin/kata-process-check.sh

HEARTBEAT_URL="https://vigilmon.online/heartbeat/abc123"

# Count containerd-shim-kata processes (one per running Kata sandbox)
KATA_SHIM_COUNT=$(pgrep -c "containerd-shim-kata" 2>/dev/null || echo 0)

# Count QEMU processes (adjust for Firecracker/Cloud Hypervisor if used)
# For QEMU: "qemu-system-x86_64" or "qemu-kvm"
HYPERVISOR_COUNT=$(pgrep -c "qemu" 2>/dev/null || echo 0)

# Check that shim and hypervisor counts are roughly matched
# Each Kata sandbox = 1 shim + 1 hypervisor
if [ "$KATA_SHIM_COUNT" -gt 0 ] && [ "$HYPERVISOR_COUNT" -eq 0 ]; then
  echo "Hypervisor/shim mismatch: ${KATA_SHIM_COUNT} shims, ${HYPERVISOR_COUNT} hypervisors"
  exit 1
fi

echo "Kata processes OK: ${KATA_SHIM_COUNT} shims, ${HYPERVISOR_COUNT} hypervisors"
curl -s "$HEARTBEAT_URL"

Install the cron job:

chmod +x /usr/local/bin/kata-process-check.sh
(crontab -l 2>/dev/null; echo "*/2 * * * * /usr/local/bin/kata-process-check.sh >> /var/log/kata-health.log 2>&1") | crontab -

A shim-without-hypervisor condition is the clearest signal that QEMU exited unexpectedly, leaving the containerd shim in a wedged state. Kubernetes will not reschedule the pod until this is detected and the shim is killed.

Step 2: Monitor MicroVM Boot Latency

Kata Containers must boot a full VM before a container can start — this adds boot latency ranging from 500ms (Cloud Hypervisor with pre-built snapshots) to several seconds (QEMU cold boot). When boot latency spikes, Kubernetes pod startup times degrade and readiness probes time out before the container is even running:

#!/bin/bash
# /usr/local/bin/kata-boot-latency.sh

HEARTBEAT_URL="https://vigilmon.online/heartbeat/def456"
BOOT_THRESHOLD=5  # Alert if boot takes more than 5 seconds

# Time a Kata container creation via crictl
START=$(date +%s)

# Create a minimal pod with the Kata runtime class
POD_ID=$(crictl runp - << 'EOF' 2>/dev/null
{
  "metadata": {"name": "kata-latency-test", "namespace": "default"},
  "log_directory": "/tmp",
  "linux": {}
}
EOF
)

if [ -z "$POD_ID" ]; then
  echo "Failed to create Kata sandbox for latency test"
  exit 1
fi

END=$(date +%s)
ELAPSED=$(( END - START ))

# Clean up the test sandbox
crictl stopp "$POD_ID" >/dev/null 2>&1
crictl rmp "$POD_ID" >/dev/null 2>&1

if [ "$ELAPSED" -lt "$BOOT_THRESHOLD" ]; then
  echo "Kata boot latency OK: ${ELAPSED}s"
  curl -s "$HEARTBEAT_URL"
else
  echo "Kata boot latency high: ${ELAPSED}s (threshold: ${BOOT_THRESHOLD}s)"
  exit 1
fi

Set the Vigilmon heartbeat interval to 10 minutes. Boot latency spikes typically indicate KVM virtualization degradation (nested virt, host resource pressure), QEMU binary I/O slowness, or kernel module issues. Detecting the spike before pod startup timeouts makes triage much easier.

Step 3: Monitor Hypervisor Health

The hypervisor is the most critical component in the Kata stack — if QEMU or Firecracker crashes, all containers in that sandbox die immediately. Track hypervisor crashes using systemd journal analysis:

#!/bin/bash
# /usr/local/bin/kata-hypervisor-health.sh

HEARTBEAT_URL="https://vigilmon.online/heartbeat/ghi789"
LOG_FILE="/var/log/kata-hypervisor-crashes.log"
TOUCH_FILE="/tmp/kata-last-crash-check"

# Look for QEMU crash signatures in the kernel log
# QEMU crashes appear as "KVM: entry failed" or segfault entries
CRASH_COUNT=$(journalctl --since="5 minutes ago" -k 2>/dev/null | \
  grep -c "qemu\|KVM: entry failed\|qemu.*segfault" || echo 0)

# Check for containerd shim panics related to Kata
SHIM_PANIC=$(journalctl --since="5 minutes ago" -u containerd 2>/dev/null | \
  grep -c "kata.*panic\|shim.*died\|hypervisor.*exit" || echo 0)

TOTAL_FAILURES=$(( CRASH_COUNT + SHIM_PANIC ))

if [ "$TOTAL_FAILURES" -gt 0 ]; then
  echo "Hypervisor failures detected in last 5 minutes: ${TOTAL_FAILURES}"
  echo "$(date): ${TOTAL_FAILURES} failures" >> "$LOG_FILE"
  exit 1
fi

echo "Hypervisor health OK: no crashes in last 5 minutes"
curl -s "$HEARTBEAT_URL"

Create a Vigilmon heartbeat with a 5 minute expected interval. Hypervisor crashes in Kata are usually caused by KVM hardware virtualization issues, memory corruption, or guests exceeding resource limits — catching them early via journal monitoring prevents silent workload loss.

Step 4: Track Resource Overhead

Kata Containers run every sandbox in a VM — this adds memory overhead (the guest kernel + kata-agent typically use 100–256 MB per sandbox) and CPU overhead from hypervisor emulation. On hosts running many Kata sandboxes, this overhead can exhaust host resources silently:

#!/bin/bash
# /usr/local/bin/kata-resource-overhead.sh

HEARTBEAT_URL="https://vigilmon.online/heartbeat/jkl012"
MEM_THRESHOLD_GB=2  # Alert if Kata overhead exceeds 2 GB
CPU_THRESHOLD=80    # Alert if Kata hypervisors use more than 80% CPU

# Total memory used by all QEMU processes
QEMU_MEM_MB=$(ps -eo pid,rss,comm 2>/dev/null | \
  awk '/qemu/ {total += $2} END {printf "%d", total/1024}')

# Total CPU used by QEMU processes (snap over 1 second)
QEMU_CPU=$(ps -eo pid,pcpu,comm 2>/dev/null | \
  awk '/qemu/ {total += $2} END {printf "%d", total}')

SANDBOX_COUNT=$(pgrep -c "containerd-shim-kata" 2>/dev/null || echo 0)

# Convert threshold to MB for comparison
MEM_THRESHOLD_MB=$(( MEM_THRESHOLD_GB * 1024 ))

if [ "$QEMU_MEM_MB" -gt "$MEM_THRESHOLD_MB" ] || [ "$QEMU_CPU" -gt "$CPU_THRESHOLD" ]; then
  echo "Kata overhead high: ${QEMU_MEM_MB}MB memory, ${QEMU_CPU}% CPU across ${SANDBOX_COUNT} sandboxes"
  exit 1
fi

echo "Kata overhead OK: ${QEMU_MEM_MB}MB, ${QEMU_CPU}% CPU, ${SANDBOX_COUNT} sandboxes"
curl -s "$HEARTBEAT_URL"

Run this check every 10 minutes and set the Vigilmon heartbeat interval to 15 minutes. Memory pressure from Kata overhead is especially dangerous on Kubernetes nodes — when the host runs low on memory, the kernel starts killing processes including QEMU instances, which silently terminates sandboxed containers.

Step 5: Verify Kata Agent Communication

The Kata agent runs inside each microVM and handles container lifecycle commands relayed from the containerd shim via a vsock or VHOST connection. When the agent becomes unreachable (network reset, vsock error, OOM kill inside the VM), the container appears running but cannot be exec'd into, and future lifecycle operations hang indefinitely:

#!/bin/bash
# /usr/local/bin/kata-agent-check.sh

HEARTBEAT_URL="https://vigilmon.online/heartbeat/mno345"

# Use kata-runtime to check the state of all running sandboxes
# "kata-runtime list" shows sandbox status including agent connectivity
if ! command -v kata-runtime >/dev/null 2>&1; then
  # Fall back to checking via crictl
  SANDBOX_LIST=$(crictl pods --state Ready 2>/dev/null | tail -n +2 | wc -l)
  echo "kata-runtime not available, falling back to crictl: ${SANDBOX_LIST} ready pods"
  curl -s "$HEARTBEAT_URL"
  exit 0
fi

# Get sandboxes and their states
RUNNING_COUNT=$(kata-runtime list 2>/dev/null | grep -c "running" || echo 0)
TOTAL_COUNT=$(kata-runtime list 2>/dev/null | tail -n +2 | wc -l || echo 0)

# If we have sandboxes that aren't in running state, flag it
PROBLEM_COUNT=$(( TOTAL_COUNT - RUNNING_COUNT ))

if [ "$PROBLEM_COUNT" -gt 0 ]; then
  echo "Kata agent issue: ${PROBLEM_COUNT} of ${TOTAL_COUNT} sandboxes not in running state"
  exit 1
fi

echo "Kata agents OK: ${RUNNING_COUNT} sandboxes in running state"
curl -s "$HEARTBEAT_URL"

Set the Vigilmon heartbeat interval to 5 minutes. Agent communication failures are silent — Kubernetes considers the pod healthy while all exec operations against it hang. The only way to detect this without active probing is via the kata-runtime state API or by testing exec responsiveness.

Step 6: Configure Alert Channels

For Kata Containers running security-sensitive workloads, alert configuration should reflect the severity:

Go to Alert Channels in Vigilmon.
Add a Slack webhook to your #platform-ops or #security-infra channel.
Add a PagerDuty or OpsGenie integration for hypervisor crash and agent connectivity monitors — these indicate active workload failure.
Add email for resource overhead alerts — these are usually slower-burning issues.
Set Consecutive failures before alert to 1 for hypervisor crash detection and 2 for latency monitors.

Add a maintenance window before Kata or kernel upgrades:

curl -X POST https://vigilmon.online/api/maintenance \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -d '{
    "monitor_id": "MONITOR_ID",
    "duration_minutes": 20,
    "reason": "Kata Containers upgrade"
  }'

# Drain the node first
kubectl drain NODE_NAME --ignore-daemonsets

# Perform the upgrade
apt-get install --only-upgrade kata-containers
systemctl restart containerd

# Uncordon after testing
kubectl uncordon NODE_NAME

Summary

| Monitor | Target | What It Catches | |---|---|---| | Cron heartbeat (processes) | Shim + hypervisor process pair | Hypervisor crash, orphaned shim | | Cron heartbeat (boot latency) | Timed microVM creation | Slow boot, KVM degradation, QEMU I/O issues | | Cron heartbeat (hypervisor) | Journal scan for crash signatures | QEMU crash, KVM entry failure, shim panics | | Cron heartbeat (overhead) | QEMU memory and CPU consumption | Host memory pressure, hypervisor CPU saturation | | Cron heartbeat (agent) | kata-runtime sandbox state check | Agent unreachable, vsock disconnection |

Kata Containers trade container density for strong VM-level isolation — and that trade introduces a hypervisor layer that can fail in ways a standard container runtime never does. Vigilmon's external heartbeat checks give you visibility into the entire Kata stack without requiring changes to your workloads, so hypervisor crashes, boot slowdowns, and agent disconnections surface as alerts rather than silent failures.

Start monitoring for free at vigilmon.online.