tutorial

Monitoring Firecracker MicroVMs with Vigilmon

Firecracker powers serverless containers at AWS Lambda and Fargate — but VM boot latency spikes, vCPU throttling, and network device saturation can silently degrade your serverless workloads. Here's how to monitor Firecracker boot time, resource metrics, and snapshot restore performance with Vigilmon.

Firecracker is the open-source microVM monitor developed by AWS that underpins Lambda, Fargate, and Lightsail Containers. It uses KVM to launch stripped-down virtual machines in under 125ms with a minimal attack surface — no BIOS, no legacy devices, just a guest kernel, virtio network/block devices, and your workload. Its speed and isolation make it ideal for multi-tenant serverless environments, but that same minimalism means fewer built-in observability hooks. VM boot latency spikes, vCPU steal from a noisy neighbor, network device throughput degradation, and snapshot restore failures all look the same from the outside: functions that start slowly or time out. Vigilmon gives you external monitoring for Firecracker VM boot performance, vCPU and memory resource metrics, network device throughput, and snapshot restore time so you catch microVM infrastructure problems before they affect function latency SLAs.

What You'll Set Up

  • Firecracker API socket health via cron heartbeats
  • VM boot latency monitoring
  • vCPU and memory resource utilization tracking
  • Network device throughput monitoring
  • Snapshot restore performance checks

Prerequisites

  • Firecracker 1.4+ installed (/usr/bin/firecracker or /usr/local/bin/firecracker)
  • KVM available (ls /dev/kvm)
  • jailer binary available for production deployments
  • A Linux host with KVM support (egrep -c '(vmx|svm)' /proc/cpuinfo > 0)
  • A free Vigilmon account

Step 1: Monitor Firecracker API Socket Health

Each Firecracker process exposes a management API via a Unix socket. This API is how orchestration systems (like Lambda's internal fleet management or your own tooling) control VM lifecycle, load snapshots, and retrieve metrics. A non-responsive API socket means VM management is broken regardless of whether the guest kernel is running. Monitor the API socket for new VM slots:

  1. In Vigilmon, click Add Monitor → Cron Heartbeat.
  2. Set Expected ping interval to 2 minutes.
  3. Copy the heartbeat URL (e.g., https://vigilmon.online/heartbeat/abc123).

Create the API health check script:

#!/bin/bash
# /usr/local/bin/firecracker-api-check.sh

HEARTBEAT_URL="https://vigilmon.online/heartbeat/abc123"
FC_SOCKET="/tmp/firecracker-monitor.socket"
FC_BIN="${FC_BIN:-/usr/bin/firecracker}"
FC_PID_FILE="/tmp/firecracker-monitor.pid"

# Clean up any stale socket from a previous check
rm -f "$FC_SOCKET"

# Launch a minimal Firecracker instance for API testing
"$FC_BIN" --api-sock "$FC_SOCKET" --log-path /dev/null &
FC_PID=$!
echo $FC_PID > "$FC_PID_FILE"

# Wait briefly for the socket to appear
sleep 0.5

if [ ! -S "$FC_SOCKET" ]; then
  echo "Firecracker API socket failed to appear"
  kill $FC_PID 2>/dev/null
  exit 1
fi

# Query the API to verify it responds
RESPONSE=$(curl -s --unix-socket "$FC_SOCKET" \
  http://localhost/version 2>/dev/null)

if echo "$RESPONSE" | grep -q "firecracker_version"; then
  VERSION=$(echo "$RESPONSE" | grep -o '"firecracker_version":"[^"]*"' | cut -d'"' -f4)
  echo "Firecracker API OK: version=$VERSION"
  curl -s "$HEARTBEAT_URL"
else
  echo "Firecracker API unresponsive or unexpected response"
  kill $FC_PID 2>/dev/null
  exit 1
fi

# Clean up the test VM
kill $FC_PID 2>/dev/null
rm -f "$FC_SOCKET" "$FC_PID_FILE"

Install and enable the cron job:

chmod +x /usr/local/bin/firecracker-api-check.sh
(crontab -l 2>/dev/null; echo "*/2 * * * * /usr/local/bin/firecracker-api-check.sh >> /var/log/firecracker-health.log 2>&1") | crontab -

This check validates the full API path without launching a guest kernel — it catches binary corruption, KVM device permission issues, and socket creation failures.

Step 2: Monitor VM Boot Latency

Firecracker's sub-125ms boot time is one of its core value propositions. Boot latency above 300ms typically indicates KVM overhead (contended host CPUs), guest kernel bloat (too many compiled-in drivers), or virtio device initialization delays. Monitor boot latency continuously:

#!/bin/bash
# /usr/local/bin/firecracker-boot-latency.sh

HEARTBEAT_URL="https://vigilmon.online/heartbeat/def456"
FC_BIN="${FC_BIN:-/usr/bin/firecracker}"
FC_SOCKET="/tmp/fc-boot-test-$$.socket"
KERNEL_IMAGE="${FC_KERNEL:-/var/lib/firecracker/vmlinux}"
ROOTFS_IMAGE="${FC_ROOTFS:-/var/lib/firecracker/rootfs.ext4}"
BOOT_THRESHOLD_MS=300  # Alert if boot takes more than 300ms
LOG_FILE="/var/log/firecracker-boot-latency.log"

# Start Firecracker
"$FC_BIN" --api-sock "$FC_SOCKET" --log-path /dev/null &
FC_PID=$!
sleep 0.3

if [ ! -S "$FC_SOCKET" ]; then
  echo "Socket did not appear"
  kill $FC_PID 2>/dev/null
  exit 1
fi

# Configure the VM via the API
curl -s --unix-socket "$FC_SOCKET" \
  -X PUT http://localhost/boot-source \
  -H "Content-Type: application/json" \
  -d "{\"kernel_image_path\": \"$KERNEL_IMAGE\", \"boot_args\": \"console=ttyS0 noapic reboot=k panic=1 pci=off nomodules\"}" \
  >/dev/null

curl -s --unix-socket "$FC_SOCKET" \
  -X PUT http://localhost/drives/rootfs \
  -H "Content-Type: application/json" \
  -d "{\"drive_id\": \"rootfs\", \"path_on_host\": \"$ROOTFS_IMAGE\", \"is_root_device\": true, \"is_read_only\": false}" \
  >/dev/null

curl -s --unix-socket "$FC_SOCKET" \
  -X PUT http://localhost/machine-config \
  -H "Content-Type: application/json" \
  -d '{"vcpu_count": 1, "mem_size_mib": 128}' \
  >/dev/null

# Measure boot time
START_NS=$(date +%s%N)
curl -s --unix-socket "$FC_SOCKET" \
  -X PUT http://localhost/actions \
  -H "Content-Type: application/json" \
  -d '{"action_type": "InstanceStart"}' \
  >/dev/null
END_NS=$(date +%s%N)

ELAPSED_MS=$(( (END_NS - START_NS) / 1000000 ))

# Log for trend analysis
echo "$(date -u +%Y-%m-%dT%H:%M:%SZ) boot_latency_ms=$ELAPSED_MS" >> "$LOG_FILE"

# Tear down the VM
kill $FC_PID 2>/dev/null
rm -f "$FC_SOCKET"

if [ "$ELAPSED_MS" -lt "$BOOT_THRESHOLD_MS" ]; then
  echo "Boot OK: ${ELAPSED_MS}ms"
  curl -s "$HEARTBEAT_URL"
else
  echo "Boot slow: ${ELAPSED_MS}ms (threshold: ${BOOT_THRESHOLD_MS}ms)"
  exit 1
fi

Set the Vigilmon heartbeat to 10 minutes. Review the boot latency log file weekly — a gradual increase from 80ms to 250ms over a week typically indicates KVM steal time accumulation or host CPU frequency scaling issues (cpupower frequency-info).

Step 3: Track vCPU and Memory Resource Metrics

Firecracker exposes per-VM resource metrics via its API and a FIFO-based metrics pipe. vCPU steal time (time the guest vCPU spent waiting for a physical CPU) and memory balloon pressure (if the balloon device is configured) are the two most actionable resource signals. Monitor them for the VMs you care about most:

#!/bin/bash
# /usr/local/bin/firecracker-resource-metrics.sh

HEARTBEAT_URL="https://vigilmon.online/heartbeat/ghi789"
FC_SOCKET="${FC_SOCKET:-/tmp/firecracker.socket}"
METRICS_FIFO="/tmp/fc-metrics-$$.fifo"
VCPU_STEAL_THRESHOLD=10  # percent

if [ ! -S "$FC_SOCKET" ]; then
  echo "No Firecracker socket at $FC_SOCKET — is a VM running?"
  exit 1
fi

# Request metrics flush via the API
curl -s --unix-socket "$FC_SOCKET" \
  -X PATCH http://localhost/vm \
  -H "Content-Type: application/json" \
  -d '{"state": "Running"}' >/dev/null 2>&1

# Read the latest metrics from the Firecracker metrics endpoint
METRICS=$(curl -s --unix-socket "$FC_SOCKET" http://localhost/metrics 2>/dev/null)

if [ -z "$METRICS" ]; then
  # Metrics endpoint may not be configured; fall back to host-level KVM stats
  KVM_STEAL=$(cat /proc/stat | awk '/^cpu / {steal=$9; total=0; for(i=2;i<=NF;i++) total+=$i; printf "%.1f", steal/total*100}')
  echo "Host KVM steal: ${KVM_STEAL}%"

  STEAL_INT=$(echo "$KVM_STEAL" | cut -d'.' -f1)
  if [ "${STEAL_INT:-0}" -lt "$VCPU_STEAL_THRESHOLD" ]; then
    curl -s "$HEARTBEAT_URL"
  else
    echo "High KVM steal time: ${KVM_STEAL}% (threshold: ${VCPU_STEAL_THRESHOLD}%)"
    exit 1
  fi
else
  echo "Firecracker metrics retrieved"
  curl -s "$HEARTBEAT_URL"
fi

Run every 5 minutes. To get richer vCPU metrics, configure Firecracker with a metrics FIFO file at startup (--metrics-fifo /tmp/fc-metrics.fifo) and parse the JSON it writes — it includes vcpu_exit_io, vcpu_exit_mmio, and patch_error counters that reveal I/O-bound and misconfigured workloads.

Step 4: Monitor Network Device Throughput

Firecracker's network device (virtio-net) uses tap devices on the host. Network throughput degradation — from tap queue saturation, host NIC oversubscription, or tc filter chain misconfiguration — affects all VMs on the host simultaneously. Monitor throughput via tap device statistics:

#!/bin/bash
# /usr/local/bin/firecracker-network-check.sh

HEARTBEAT_URL="https://vigilmon.online/heartbeat/jkl012"
TAP_IFACE="${FC_TAP:-tap0}"  # The tap interface Firecracker uses
TX_THRESHOLD_MBPS=900   # Alert if TX throughput drops below this for a loaded VM
SAMPLE_INTERVAL=5        # Seconds to sample

if ! ip link show "$TAP_IFACE" >/dev/null 2>&1; then
  echo "Tap interface $TAP_IFACE not found — no VMs may be running"
  # Still ping heartbeat if this is expected (no VMs = no tap)
  curl -s "$HEARTBEAT_URL"
  exit 0
fi

# Sample RX/TX bytes before and after the interval
RX_BEFORE=$(cat /sys/class/net/$TAP_IFACE/statistics/rx_bytes 2>/dev/null || echo 0)
TX_BEFORE=$(cat /sys/class/net/$TAP_IFACE/statistics/tx_bytes 2>/dev/null || echo 0)

sleep $SAMPLE_INTERVAL

RX_AFTER=$(cat /sys/class/net/$TAP_IFACE/statistics/rx_bytes 2>/dev/null || echo 0)
TX_AFTER=$(cat /sys/class/net/$TAP_IFACE/statistics/tx_bytes 2>/dev/null || echo 0)

RX_BPS=$(( (RX_AFTER - RX_BEFORE) * 8 / SAMPLE_INTERVAL ))
TX_BPS=$(( (TX_AFTER - TX_BEFORE) * 8 / SAMPLE_INTERVAL ))
RX_MBPS=$(( RX_BPS / 1000000 ))
TX_MBPS=$(( TX_BPS / 1000000 ))

echo "Network: RX=${RX_MBPS}Mbps TX=${TX_MBPS}Mbps on $TAP_IFACE"

# Check for dropped packets which indicate queue saturation
RX_DROP=$(cat /sys/class/net/$TAP_IFACE/statistics/rx_dropped 2>/dev/null || echo 0)
TX_DROP=$(cat /sys/class/net/$TAP_IFACE/statistics/tx_dropped 2>/dev/null || echo 0)

if [ "$RX_DROP" -gt 1000 ] || [ "$TX_DROP" -gt 1000 ]; then
  echo "Network drops detected: rx_dropped=$RX_DROP tx_dropped=$TX_DROP"
  exit 1
fi

curl -s "$HEARTBEAT_URL"

Set the Vigilmon heartbeat to 5 minutes. Significant packet drops on the tap interface (>1000 in a measurement window) indicate the virtio-net queue depth is being exceeded — this degrades all VMs sharing that host's network and typically requires host-level ethtool -G $IFACE queue depth tuning.

Step 5: Monitor Snapshot Restore Performance

Firecracker snapshots (VMM state + memory snapshot) are the mechanism that enables Lambda-style fast function warm-starts. Snapshot restore latency above 50ms means your platform can't meet sub-100ms cold-start SLAs. Monitor restore performance continuously if you rely on snapshots for workload warm-up:

#!/bin/bash
# /usr/local/bin/firecracker-snapshot-restore.sh

HEARTBEAT_URL="https://vigilmon.online/heartbeat/mno345"
FC_BIN="${FC_BIN:-/usr/bin/firecracker}"
FC_SOCKET="/tmp/fc-snap-test-$$.socket"
SNAP_DIR="${FC_SNAP_DIR:-/var/lib/firecracker/snapshots}"
RESTORE_THRESHOLD_MS=100  # Alert if restore takes more than 100ms
LOG_FILE="/var/log/firecracker-snapshot-latency.log"

# This script assumes a snapshot already exists at $SNAP_DIR/test/
SNAP_VMSTATE="$SNAP_DIR/test/vmstate.snap"
SNAP_MEMORY="$SNAP_DIR/test/mem.snap"

if [ ! -f "$SNAP_VMSTATE" ] || [ ! -f "$SNAP_MEMORY" ]; then
  echo "No test snapshot at $SNAP_DIR/test/ — skipping restore check"
  # Ping heartbeat anyway so the monitor doesn't false-alarm
  curl -s "$HEARTBEAT_URL"
  exit 0
fi

"$FC_BIN" --api-sock "$FC_SOCKET" --log-path /dev/null &
FC_PID=$!
sleep 0.3

if [ ! -S "$FC_SOCKET" ]; then
  echo "Firecracker socket did not appear"
  kill $FC_PID 2>/dev/null
  exit 1
fi

START_NS=$(date +%s%N)
RESTORE_RESULT=$(curl -s --unix-socket "$FC_SOCKET" \
  -X PUT http://localhost/snapshot/load \
  -H "Content-Type: application/json" \
  -d "{
    \"snapshot_path\": \"$SNAP_VMSTATE\",
    \"mem_file_path\": \"$SNAP_MEMORY\",
    \"enable_diff_snapshots\": false,
    \"resume_vm\": false
  }")
END_NS=$(date +%s%N)
ELAPSED_MS=$(( (END_NS - START_NS) / 1000000 ))

echo "$(date -u +%Y-%m-%dT%H:%M:%SZ) snapshot_restore_ms=$ELAPSED_MS" >> "$LOG_FILE"

kill $FC_PID 2>/dev/null
rm -f "$FC_SOCKET"

if echo "$RESTORE_RESULT" | grep -q '"fault_message"'; then
  echo "Snapshot restore failed: $RESTORE_RESULT"
  exit 1
fi

if [ "$ELAPSED_MS" -lt "$RESTORE_THRESHOLD_MS" ]; then
  echo "Snapshot restore OK: ${ELAPSED_MS}ms"
  curl -s "$HEARTBEAT_URL"
else
  echo "Snapshot restore slow: ${ELAPSED_MS}ms (threshold: ${RESTORE_THRESHOLD_MS}ms)"
  exit 1
fi

Run every 15 minutes. Slow snapshot restores are often caused by memory snapshot files stored on slow storage (NFS, network-backed EBS), or by large memory snapshots that require I/O-intensive page remapping. If restore latency drifts above your threshold, move snapshots to local NVMe or investigate memory balloon usage that inflates snapshot size.

Step 6: Set Up Alert Channels

Route Firecracker alerts to the team responsible for your serverless or container platform:

  1. Go to Alert Channels in Vigilmon.
  2. Add a Slack webhook to your #platform-alerts or #sre channel.
  3. Add PagerDuty for the boot latency and API socket monitors — these directly impact function invocation latency.
  4. Set Consecutive failures before alert to 1 for snapshot restore monitors — even a single failure means warm-start is broken.

Add maintenance windows for host maintenance:

curl -X POST https://vigilmon.online/api/maintenance \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -d '{
    "monitor_id": "MONITOR_ID",
    "duration_minutes": 30,
    "reason": "host kernel upgrade and KVM module reload"
  }'

Summary

| Monitor | Target | What It Catches | |---|---|---| | Cron heartbeat (API socket) | GET /version via Unix socket | Binary corruption, KVM permission error, daemon crash | | Cron heartbeat (boot latency) | Timed InstanceStart API call | KVM steal time, guest kernel bloat, CPU frequency scaling | | Cron heartbeat (vCPU/memory) | Host steal time + metrics flush | CPU contention, noisy neighbor, resource overcommit | | Cron heartbeat (network) | Tap device byte/drop counters | virtio-net queue saturation, NIC oversubscription | | Cron heartbeat (snapshot restore) | Timed snapshot load API call | Slow storage, large memory snapshots, restore path bugs |

Firecracker is built for scale, but at scale its failure modes become infrastructure-level rather than application-level — host CPU steal, tap device saturation, and snapshot store I/O all affect every VM on the host simultaneously. Vigilmon's external checks give you early warning before a slow host degrades your entire function fleet.

Start monitoring for free at vigilmon.online.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles