tutorial

Monitoring Bottlerocket Nodes with Vigilmon

Bottlerocket is AWS's immutable, container-optimized Linux — no package manager, no shell, no manual installs. Here's how to monitor node health, A/B partition status, containerd, and update readiness with Vigilmon.

Bottlerocket is an open-source, immutable Linux distribution from AWS built exclusively for running containers. Unlike Ubuntu or Amazon Linux, Bottlerocket has no package manager, no persistent SSH daemon, and no shell in production — the OS image is read-only and updated atomically via an A/B partition scheme. That minimal, locked-down design is a security win, but it also means your traditional host-level monitoring approaches don't apply. Vigilmon fills the observability gap: monitor the Bottlerocket API server, partition health, containerd, update operator status, and node resources — all without touching the OS directly.

What You'll Set Up

  • Bottlerocket API server health monitor (the bottlerocket-api process and Unix socket)
  • A/B OS partition update health check (active partition, last update status)
  • containerd process health monitor (container runtime availability)
  • Node-level resource usage alerts (CPU, memory, disk)
  • Kubernetes update-operator pod health (automatic OS update pipeline)
  • CIS benchmark compliance drift alert
  • SSL and certificate expiry checks for cluster API endpoints

Prerequisites

  • Bottlerocket nodes running on AWS EC2 (EKS worker nodes, ECS instances, or general EC2)
  • Access to the control container or a monitoring DaemonSet deployed to your cluster
  • kubectl access to your Kubernetes cluster (for EKS/Kubernetes deployments)
  • A free Vigilmon account

Why Monitoring Bottlerocket Is Different

Bottlerocket's immutability is its strength — but it removes the usual hooks: you can't apt install node_exporter, you can't ssh in to check logs, and you can't install arbitrary daemons. Monitoring must happen through one of these paths:

  1. The Bottlerocket API (bottlerocket-api) — an HTTP API exposed via Unix socket that accepts TOML settings and reports node state
  2. A monitoring DaemonSet — a privileged Kubernetes pod that runs on each Bottlerocket node and exposes metrics via HTTP
  3. The control container — a privileged container that provides an admin shell for emergency access and API queries
  4. Kubernetes-native probes — kube-state-metrics and cAdvisor metrics from the kubelet

Vigilmon monitors the HTTP endpoints your DaemonSet or API server exposes, and sends alerts when checks fail.

Step 1: Deploy a Monitoring DaemonSet to Expose Node Health Endpoints

Since you can't install software directly on Bottlerocket, deploy a privileged DaemonSet that reads node health via the Bottlerocket API socket and exposes it as HTTP endpoints Vigilmon can probe.

Create a DaemonSet manifest:

# bottlerocket-monitor-daemonset.yaml
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: bottlerocket-node-monitor
  namespace: monitoring
spec:
  selector:
    matchLabels:
      app: bottlerocket-node-monitor
  template:
    metadata:
      labels:
        app: bottlerocket-node-monitor
    spec:
      hostPID: true
      hostNetwork: true
      tolerations:
        - effect: NoSchedule
          operator: Exists
      containers:
        - name: monitor
          image: amazonlinux:2
          command:
            - /bin/bash
            - -c
            - |
              yum install -y socat curl python3 && python3 -m http.server 9099 &
              while true; do
                # Query Bottlerocket API via the control socket
                STATUS=$(curl -s --unix-socket /run/api.sock \
                  http://localhost/v1/metadata/os 2>/dev/null || echo '{"error":"api_unavailable"}')
                echo "$STATUS" > /tmp/br_status.json
                sleep 30
              done
          ports:
            - containerPort: 9099
              hostPort: 9099
          volumeMounts:
            - name: api-socket
              mountPath: /run/api.sock
            - name: tmp
              mountPath: /tmp
          securityContext:
            privileged: true
      volumes:
        - name: api-socket
          hostPath:
            path: /run/api.sock
            type: Socket
        - name: tmp
          emptyDir: {}

Apply it:

kubectl apply -f bottlerocket-monitor-daemonset.yaml

This approach gives Vigilmon an HTTP endpoint per node to probe. For production use, replace the shell script with a purpose-built tool such as Bottlerocket's own telemetry agent or a Prometheus exporter.

Step 2: Monitor the Bottlerocket API Server Health

The bottlerocket-api process manages all configuration changes and OS settings. If it fails, you cannot apply node settings, change container runtime config, or initiate updates.

Add a Vigilmon monitor for the API server health endpoint exposed by your DaemonSet:

  1. Log in to vigilmon.online and click Add Monitor.
  2. Set Type to HTTP / HTTPS.
  3. Enter the node endpoint: http://<node-ip>:9099/api_health (replace <node-ip> with the node's internal IP or load-balanced endpoint).
  4. Set Check interval to 1 minute.
  5. Set Expected HTTP status to 200.
  6. Under Alert thresholds, set Failure threshold to 2 consecutive failures.
  7. Click Save.

Repeat for each Bottlerocket node in your fleet, or use a load balancer health check endpoint that aggregates all nodes.

Alert rule: Trigger a critical alert if the API server is unreachable for more than 2 consecutive checks — API server failure blocks all configuration changes to the node.

Step 3: Monitor A/B Partition Update Health

Bottlerocket uses an A/B partition scheme: the OS lives on one of two partitions, and updates are applied by activating the other partition. A failed update leaves the node on the stale partition and potentially running an outdated kernel.

Query the update status via the Bottlerocket API:

# From the control container or a privileged DaemonSet pod
curl -s --unix-socket /run/api.sock http://localhost/v1/os \
  | python3 -m json.tool

Example response:

{
  "bottlerocket": {
    "version-id": "1.19.2",
    "active-partition": "A",
    "boot-count": 47
  }
}

Set up a Vigilmon cron-style heartbeat monitor to track update staleness:

  1. In your Vigilmon dashboard, click Add Monitor.
  2. Set Type to Heartbeat.
  3. Name it bottlerocket-update-check.
  4. Set Expected interval to 1 hour.
  5. Copy the unique heartbeat URL.

In your monitoring DaemonSet, add a periodic check that pings the heartbeat URL when the node is running the current OS version, and stays silent when the node is lagging. Vigilmon raises an alert if it doesn't receive a ping within the expected window — meaning the node is behind or the check itself failed.

Alert rule: Alert if a Bottlerocket node is more than 2 major versions behind the current release channel.

Step 4: Monitor containerd Health

Bottlerocket ships containerd as its only container runtime. If containerd crashes or stops responding, all containers on that node become unreachable and Kubernetes cannot schedule new pods.

Expose a containerd health check in your monitoring DaemonSet:

# Check containerd is responding via its gRPC health endpoint
ctr version > /dev/null 2>&1 && echo "ok" || echo "down"

Add a Vigilmon HTTP monitor pointing at a /containerd_health path served by your DaemonSet that returns 200 when ctr version succeeds and 503 when it fails.

Alert rule: Trigger critical immediately on any containerd health check failure — a containerd crash affects all containers on the node.

Step 5: Monitor the Bottlerocket Update Operator (EKS/Kubernetes)

For Kubernetes clusters, the Bottlerocket Update Operator (brupop) automates node updates. Monitor the operator pod and update queue:

# Check update operator pod health
kubectl get pods -n brupop-bottlerocket-aws \
  -l app.kubernetes.io/name=brupop-controller

# Check pending updates across fleet
kubectl get bottlerocketnodeupdates -A

Add a Vigilmon monitor for the update operator's metrics endpoint:

  1. Expose the operator's Prometheus metrics as an HTTP health endpoint (e.g., a sidecar that returns 200 when the operator deployment is healthy and all nodes are within 2 versions of current).
  2. Add an HTTP monitor pointing at that endpoint.
  3. Set Check interval to 5 minutes.
  4. Set Expected HTTP status to 200.

Alert rule: Alert warning if more than 10% of nodes have pending updates, critical if any node is more than 2 major versions behind.

Step 6: Monitor Node-Level Resource Usage

Even though Bottlerocket is minimal, the node itself can hit resource limits. Use Kubernetes kubelet metrics or a node metrics endpoint:

# Via kubectl top (requires metrics-server)
kubectl top node <bottlerocket-node-name>

For Vigilmon, use a cron heartbeat sent by a cluster-side monitoring job:

#!/bin/bash
# Run as a Kubernetes CronJob on each node
NODE_CPU=$(kubectl top node $(hostname) --no-headers | awk '{print $3}' | tr -d '%')
NODE_MEM=$(kubectl top node $(hostname) --no-headers | awk '{print $5}' | tr -d '%')

if [ "$NODE_CPU" -lt 85 ] && [ "$NODE_MEM" -lt 85 ]; then
  curl -s "$VIGILMON_HEARTBEAT_URL"
fi

Alert rules:

  • warning at CPU > 70% sustained for 5 minutes
  • critical at CPU > 85% or memory > 85%
  • critical at data partition usage > 80% (the OS partition is read-only; alert on the data partition /var)

Step 7: Monitor Disk Partition Health

Bottlerocket has two partitions: a read-only OS partition and a writable data partition (/var). Monitor the data partition:

# Check data partition usage from a privileged DaemonSet
df -h /var | awk 'NR==2 {print $5}' | tr -d '%'

Expose this as a /disk_health HTTP endpoint in your monitoring DaemonSet, returning 200 under 80% and 503 over 80%.

Add a Vigilmon HTTP monitor for this endpoint with a 1-minute check interval and critical alert at 503.

Step 8: Monitor Control Container Accessibility

The Bottlerocket control container is the emergency admin path — the only way to run shell commands or diagnose critical issues on a production node. Monitor its health with a heartbeat:

# The control container sends a heartbeat when it is running
# Add to the control container's startup script:
curl -s "$VIGILMON_CONTROL_CONTAINER_HEARTBEAT_URL"

# And every 5 minutes:
while true; do
  curl -s "$VIGILMON_CONTROL_CONTAINER_HEARTBEAT_URL"
  sleep 300
done
  1. In Vigilmon, create a Heartbeat monitor named bottlerocket-control-container.
  2. Set Expected interval to 10 minutes.
  3. Add the heartbeat URL to the control container's configuration.

Alert rule: critical if the control container heartbeat is missed for more than 15 minutes — loss of the control container means no admin access to the node.

Step 9: CIS Benchmark Compliance Monitoring

Bottlerocket ships with CIS benchmark hardening by default. Monitor for configuration drift using kube-bench:

# Run kube-bench against Bottlerocket nodes
kubectl apply -f https://raw.githubusercontent.com/aquasecurity/kube-bench/main/job-eks.yaml

# Check results
kubectl logs job/kube-bench | grep -E "FAIL|WARN"

Set up a Vigilmon heartbeat for compliance checks:

#!/bin/bash
# Run as a weekly Kubernetes CronJob
FAILURES=$(kubectl logs job/kube-bench | grep -c "FAIL" || echo "999")
if [ "$FAILURES" -eq 0 ]; then
  curl -s "$VIGILMON_COMPLIANCE_HEARTBEAT_URL"
fi

Alert rule: warning if kube-bench reports any new FAIL results compared to last week's baseline.

Summary: Bottlerocket Monitoring Checklist

| Check | Monitor Type | Alert Level | Interval | |---|---|---|---| | Bottlerocket API server | HTTP | Critical | 1 min | | A/B partition update status | Heartbeat | Warning | 1 hour | | containerd health | HTTP | Critical | 1 min | | Update operator pod | HTTP | Warning | 5 min | | Node CPU/memory | Heartbeat | Warning/Critical | 5 min | | Data partition usage >80% | HTTP | Critical | 5 min | | Control container accessibility | Heartbeat | Critical | 10 min | | CIS compliance drift | Heartbeat | Warning | 24 hours |

Bottlerocket's immutability means you trade manual configurability for reliability — but you still need visibility. With Vigilmon probing your DaemonSet endpoints and heartbeat monitors tracking update health and compliance, you get the observability layer Bottlerocket's locked-down design intentionally leaves out.

Get started with a free Vigilmon account and have your first Bottlerocket node monitored in under five minutes.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles