tutorial

Monitoring Flatcar Container Linux with Vigilmon

Flatcar Container Linux is the immutable, auto-updating container OS that succeeded CoreOS. Here's how to monitor update-engine, OS partition health, systemd units, and fleet-wide update lag with Vigilmon.

Flatcar Container Linux is the direct successor to CoreOS Container Linux — an immutable, auto-updating OS designed exclusively for containerized workloads at scale. Like its predecessor, Flatcar uses an A/B partition update scheme, ships with Docker or containerd, and uses systemd for service management. The automatic update model is Flatcar's biggest operational advantage: nodes stay current without manual intervention. But that update pipeline also needs to be monitored — a stuck update-engine, a lost connection to the Nebraska update server, or a fleet drifting across OS versions can create silent security and stability debt. Vigilmon gives you the visibility layer that makes Flatcar's self-healing design actually observable.

What You'll Set Up

  • update-engine.service health monitoring (the Flatcar automatic update daemon)
  • Nebraska/Omaha update server connectivity check
  • Container runtime health monitor (Docker or containerd)
  • systemd failed-unit alerting
  • Node-level resource usage (CPU, memory, disk)
  • Fleet-wide OS version drift alerting
  • Disk partition integrity check (read-only OS partition remount detection)
  • Ignition first-boot provisioning success monitor

Prerequisites

  • Flatcar Container Linux nodes (cloud, VM, or bare metal)
  • SSH access to at least one node (or a monitoring agent deployed to each node)
  • A Kubernetes cluster or a fleet of Flatcar nodes with a monitoring path
  • A free Vigilmon account

Why Flatcar Monitoring Requires a Specific Approach

Flatcar is immutable — the OS partition is read-only and you cannot install software directly on it. Monitoring must work through one of these paths:

  1. systemd-run or a containerized monitoring agent — deploy a container (e.g., a Prometheus Node Exporter container) that exposes node metrics via HTTP
  2. Ignition provisioning — include monitoring agent configuration in your Ignition config at first boot
  3. SSH or Fleet management — run periodic checks from a management host
  4. Kubernetes DaemonSet — if running Flatcar as Kubernetes nodes, deploy a DaemonSet to each node

The examples below use a containerized monitoring approach that works on all Flatcar deployments.

Step 1: Deploy a Monitoring Container via Ignition

The right time to configure monitoring on a Flatcar node is at first boot, via Ignition. Add the monitoring container to your Ignition configuration:

# flatcar-ignition-monitoring.yaml (Butane format, compile with `butane`)
variant: flatcar
version: 1.0.0
systemd:
  units:
    - name: node-monitor.service
      enabled: true
      contents: |
        [Unit]
        Description=Node Monitor Agent
        After=docker.service
        Requires=docker.service

        [Service]
        Restart=always
        ExecStartPre=-/usr/bin/docker rm -f node-monitor
        ExecStart=/usr/bin/docker run --rm \
          --name node-monitor \
          --net host \
          --pid host \
          --privileged \
          -v /proc:/host/proc:ro \
          -v /sys:/host/sys:ro \
          -v /:/rootfs:ro \
          -p 9100:9100 \
          prom/node-exporter:latest \
          --path.procfs=/host/proc \
          --path.sysfs=/host/sys \
          --path.rootfs=/rootfs \
          --collector.filesystem.ignored-mount-points="^/(sys|proc|dev|host|etc)($|/)"

        [Install]
        WantedBy=multi-user.target

Compile and apply:

butane flatcar-ignition-monitoring.yaml -o flatcar-monitoring.ign

For existing nodes, start the container manually:

docker run -d --name node-monitor --net host --pid host --privileged \
  -v /proc:/host/proc:ro -v /sys:/host/sys:ro -v /:/rootfs:ro \
  -p 9100:9100 prom/node-exporter:latest \
  --path.procfs=/host/proc --path.sysfs=/host/sys --path.rootfs=/rootfs

Step 2: Monitor update-engine Service Health

update-engine is the Flatcar automatic update daemon. It is a critical systemd service — if it crashes, your node will never receive automatic OS updates and will silently fall behind.

Check its status from any management host or via your monitoring container:

# Via SSH
systemctl is-active update-engine.service

# Via journalctl (check for recent failures)
journalctl -u update-engine.service --since "1 hour ago" | grep -i "error\|fail\|crash"

Set up a Vigilmon heartbeat monitor for update-engine:

#!/bin/bash
# Add to a cron job or systemd timer on each Flatcar node

if systemctl is-active --quiet update-engine.service; then
  curl -s "$VIGILMON_UPDATE_ENGINE_HEARTBEAT_URL"
fi

Create the heartbeat in Vigilmon:

  1. Log in to vigilmon.online and click Add Monitor.
  2. Set Type to Heartbeat.
  3. Name it flatcar-update-engine-<hostname>.
  4. Set Expected interval to 5 minutes.
  5. Copy the heartbeat URL and add it to your monitoring cron job.

Alert rule: critical if update-engine heartbeat is missed for 15 minutes — the node is no longer receiving automatic OS updates.

Step 3: Monitor Nebraska/Flatcar Update Server Connectivity

Flatcar checks a Nebraska-compatible update server (typically the public Flatcar update server or your internal Nebraska instance) for new OS versions. If connectivity is lost, nodes stop receiving update notifications.

Test connectivity:

# Check connectivity to the Flatcar public update server
curl -s --max-time 10 \
  https://public.update.flatcar-linux.net/v1/update/ \
  -d '{}' -o /dev/null -w "%{http_code}"

Add a Vigilmon HTTP monitor:

  1. Click Add Monitor in Vigilmon.
  2. Set Type to HTTP / HTTPS.
  3. Enter the update server URL: https://public.update.flatcar-linux.net/v1/update/
  4. Set Check interval to 5 minutes.
  5. Set Expected HTTP status to 200 or 400 (the endpoint returns 400 for empty requests — that's a valid connectivity check).
  6. Click Save.

If you run an internal Nebraska server, monitor your own instance's /health endpoint instead.

Alert rule: warning if the update server is unreachable — nodes will stop receiving update availability notifications within 24 hours.

Step 4: Monitor Container Runtime Health

Flatcar ships with Docker (or containerd). If the container runtime crashes, all containers on the node stop and Kubernetes cannot schedule new pods.

Check Docker health:

# Docker health check
docker info > /dev/null 2>&1 && echo "docker_ok" || echo "docker_down"

# Or for containerd
systemctl is-active containerd.service

Add a Vigilmon heartbeat for runtime health:

#!/bin/bash
# Runs as a systemd timer every 2 minutes

DOCKER_OK=false
CONTAINERD_OK=false

docker info > /dev/null 2>&1 && DOCKER_OK=true
systemctl is-active --quiet containerd.service && CONTAINERD_OK=true

if $DOCKER_OK || $CONTAINERD_OK; then
  curl -s "$VIGILMON_RUNTIME_HEARTBEAT_URL"
fi

Set Expected interval to 5 minutes in Vigilmon to give the check two missed cycles before alerting.

Alert rule: critical immediately — a container runtime crash affects all containers on the node.

Step 5: Monitor systemd Failed Units

Flatcar uses systemd for everything. Failed systemd units can indicate a partial provisioning failure, a crashed service, or a dependency chain problem.

Check for failed units:

# List all failed units
systemctl --failed --no-legend | wc -l

# Get failed unit details
systemctl --failed --no-legend

Send a Vigilmon heartbeat only when no units are failed:

#!/bin/bash
FAILED_COUNT=$(systemctl --failed --no-legend | wc -l)

if [ "$FAILED_COUNT" -eq 0 ]; then
  curl -s "$VIGILMON_SYSTEMD_HEARTBEAT_URL"
fi

Alert rule: warning if any non-critical systemd unit is failed; critical if docker.service, containerd.service, or update-engine.service is failed.

Step 6: Monitor Node-Level Resource Usage

Use the Node Exporter container deployed in Step 1 to expose resource metrics, then hit its HTTP endpoint with Vigilmon:

  1. Click Add Monitor → HTTP / HTTPS.
  2. Enter http://<node-ip>:9100/metrics.
  3. Set Check interval to 1 minute.
  4. Set Expected HTTP status to 200.
  5. Click Save — this confirms the exporter is alive and exposing metrics.

For threshold-based alerting, add a sidecar script that reads the metrics and sends a heartbeat only when resource usage is within bounds:

#!/bin/bash
# Parse node_exporter metrics for CPU and memory
CPU_IDLE=$(curl -s http://localhost:9100/metrics \
  | grep 'node_cpu_seconds_total{.*mode="idle"' \
  | awk '{sum+=$2; n++} END {print sum/n}')

MEM_AVAIL=$(curl -s http://localhost:9100/metrics \
  | grep 'node_memory_MemAvailable_bytes' \
  | awk '{print $2}')

MEM_TOTAL=$(curl -s http://localhost:9100/metrics \
  | grep 'node_memory_MemTotal_bytes' \
  | awk '{print $2}')

MEM_PCT=$(echo "scale=0; (1 - $MEM_AVAIL / $MEM_TOTAL) * 100" | bc)

if [ "$MEM_PCT" -lt 85 ]; then
  curl -s "$VIGILMON_RESOURCE_HEARTBEAT_URL"
fi

Alert rules:

  • warning at memory > 70%
  • critical at memory > 85% or CPU sustained > 85%
  • critical at disk > 80% on the data partition

Step 7: Monitor Fleet-Wide OS Version Drift

Across a fleet of Flatcar nodes, you want visibility into how many nodes are running the current stable channel version versus the previous version.

Collect versions from each node:

# On each node, report its current version to a central collector
CURRENT_VERSION=$(cat /etc/os-release | grep VERSION_ID | cut -d= -f2 | tr -d '"')
curl -s -X POST "$FLEET_MONITOR_ENDPOINT" \
  -H "Content-Type: application/json" \
  -d "{\"host\": \"$(hostname)\", \"version\": \"$CURRENT_VERSION\"}"

Your central fleet monitor aggregates versions and sends a Vigilmon heartbeat only when >90% of nodes are on the current stable version:

# fleet_monitor.py — run as a cron job or web service
import requests, os

nodes = get_all_node_versions()  # your fleet inventory API
current = get_current_stable_version()  # fetch from update server

up_to_date = sum(1 for v in nodes.values() if v == current)
pct = up_to_date / len(nodes) * 100

if pct >= 90:
    requests.get(os.environ["VIGILMON_FLEET_HEARTBEAT_URL"])

Alert rule: warning if >10% of nodes are behind the current stable channel; critical if any node is more than one minor version behind.

Step 8: Monitor Disk Partition Integrity

The Flatcar OS partition is read-only. If it remounts as read-write (indicating a kernel or filesystem integrity issue), that is a serious security event.

Check partition mount flags:

# The OS partition should always be read-only
grep ' / ' /proc/mounts | grep -v 'ro,' && echo "INTEGRITY_VIOLATION" || echo "ok"

# Also check the update partition (the inactive A/B slot)
cat /sys/block/sda/sda3/ro  # 1 = read-only, 0 = writable

Send a heartbeat when the OS partition is correctly read-only:

#!/bin/bash
OS_RO=$(grep ' / ' /proc/mounts | grep -c ',ro,')

if [ "$OS_RO" -gt 0 ]; then
  curl -s "$VIGILMON_PARTITION_HEARTBEAT_URL"
fi

Alert rule: critical immediately if the OS partition is detected as read-write — this indicates a potential OS integrity violation.

Step 9: Monitor Ignition Provisioning Success

Flatcar's first-boot Ignition provisioning is a one-time event — but failed Ignition provisioning leaves nodes misconfigured. Check the journal for Ignition completion status:

# Check Ignition journal for success or failure
journalctl -u ignition-complete.service | tail -20

# A successfully provisioned node has this entry:
journalctl -u ignition-complete.service | grep -c "Ignition finished successfully"

For fleet provisioning audits, have each node send a heartbeat after Ignition completes successfully as part of its Ignition configuration:

# In your Butane/Ignition config
systemd:
  units:
    - name: ignition-done-notify.service
      enabled: true
      contents: |
        [Unit]
        Description=Notify monitoring after Ignition
        After=ignition-complete.target
        ConditionPathExists=/etc/ignition_done
        
        [Service]
        Type=oneshot
        ExecStart=/usr/bin/curl -s "VIGILMON_IGNITION_HEARTBEAT_URL"
        
        [Install]
        WantedBy=multi-user.target

Alert rule: warning if an expected node does not send its Ignition completion heartbeat within 30 minutes of provisioning.

Summary: Flatcar Monitoring Checklist

| Check | Monitor Type | Alert Level | Interval | |---|---|---|---| | update-engine service | Heartbeat | Critical | 5 min | | Nebraska update server connectivity | HTTP | Warning | 5 min | | Container runtime (Docker/containerd) | Heartbeat | Critical | 2 min | | systemd failed units | Heartbeat | Warning/Critical | 5 min | | Node CPU/memory/disk | HTTP + Heartbeat | Warning/Critical | 1 min | | Fleet OS version drift (>10%) | Heartbeat | Warning | 1 hour | | OS partition read-only integrity | Heartbeat | Critical | 5 min | | Ignition provisioning success | Heartbeat | Warning | once at boot |

Flatcar's self-updating design handles OS maintenance automatically — but only when the update pipeline is healthy. With Vigilmon monitoring update-engine, partition integrity, and fleet version drift, you get the observability layer that turns Flatcar's automated approach into a reliable, auditable operation rather than a silent unknown.

Start monitoring your Flatcar fleet today with a free Vigilmon account.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles