tutorial

Monitoring Falco Sidekick and Runtime Security Alert Pipelines with Vigilmon

Falco detects suspicious behavior in Kubernetes workloads—unexpected shell spawns, privilege escalations, sensitive file reads, network connections to unknow...

Falco detects suspicious behavior in Kubernetes workloads—unexpected shell spawns, privilege escalations, sensitive file reads, network connections to unknown IPs. Falco Sidekick is the fanout router that takes those events and forwards them to Slack, PagerDuty, Teams, webhooks, and dozens of other outputs. If Sidekick is degraded, your runtime security alerts disappear silently into the void while attacks continue undetected.

This tutorial shows how to monitor Falco Sidekick's alert routing health, output channel delivery rates, Falco event ingestion throughput, and overall pipeline reliability using Vigilmon—so you know the moment your runtime security pipeline stops delivering.

Why Falco Sidekick needs external monitoring

Falco and Sidekick fail in ways that look fine from the inside:

  • Sidekick is alive but outputs are broken — Slack webhook URL rotated, PagerDuty integration key expired, or a Teams connector was deleted; Sidekick receives events and drops them silently
  • Falco stops sending events — a kernel module update incompatibility, eBPF probe crash, or Falco container OOM causes event ingestion to stall; Sidekick stays healthy but receives nothing
  • Output channel backlog builds — a downstream service (SIEM, ticketing) is slow; Sidekick queues events and eventually drops them when the buffer fills
  • Rules update breaks detection — a Falco rules file change introduces a syntax error; all detection stops while Falco runs but emits zero events
  • Sidekick restarts lose in-flight events — Sidekick has no persistent queue; events received during a restart are lost

Kubernetes Running status and Sidekick's own /ping endpoint do not catch any of these conditions.

What you'll need

  • Falco installed in your cluster (DaemonSet with eBPF or kernel module driver)
  • Falco Sidekick deployed (standalone or as a Falco sidecar)
  • A free Vigilmon account

Step 1: Expose Falco Sidekick built-in health endpoints

Falco Sidekick ships with several built-in endpoints. Expose the service:

# falco-sidekick-service.yaml
apiVersion: v1
kind: Service
metadata:
  name: falco-sidekick
  namespace: falco
spec:
  selector:
    app: falco-sidekick
  ports:
    - name: http
      port: 2801
      targetPort: 2801

Check the available endpoints:

# Liveness ping
curl http://falco-sidekick:2801/ping
# pong

# Readiness
curl http://falco-sidekick:2801/healthz
# {"status":"ok"}

# Prometheus metrics
curl http://falco-sidekick:2801/metrics

Step 2: Monitor alert routing health per output channel

Falco Sidekick exposes per-output Prometheus metrics. The most important ones:

# Total events forwarded per output
falcosidekick_outputs_total{output="slack", status="ok"}
falcosidekick_outputs_total{output="pagerduty", status="ok"}
falcosidekick_outputs_total{output="webhook", status="ok"}

# Failed deliveries per output
falcosidekick_outputs_total{output="slack", status="error"}

# Falco events received by Sidekick
falcosidekick_inputs_total

Build a health bridge that queries these metrics and exposes a simple HTTP status per output channel:

# sidekick-health-bridge.py
import os, requests
from flask import Flask, jsonify

app = Flask(__name__)
SIDEKICK_URL = os.environ.get("SIDEKICK_URL", "http://falco-sidekick:2801")
OUTPUTS_TO_CHECK = os.environ.get("OUTPUTS", "slack,pagerduty,webhook").split(",")
MIN_EVENTS_PER_HOUR = int(os.environ.get("MIN_EVENTS_PER_HOUR", "0"))

def parse_prometheus(text):
    metrics = {}
    for line in text.splitlines():
        if line.startswith("#"):
            continue
        if "{" in line:
            name_labels, value = line.rsplit(" ", 1)
            metrics[name_labels] = float(value)
    return metrics

@app.get("/health")
def health():
    try:
        resp = requests.get(f"{SIDEKICK_URL}/metrics", timeout=5)
        resp.raise_for_status()
    except Exception as e:
        return jsonify({"status": "sidekick_unreachable", "error": str(e)}), 503

    metrics = parse_prometheus(resp.text)

    errors = {}
    for output in OUTPUTS_TO_CHECK:
        error_key = f'falcosidekick_outputs_total{{output="{output}",status="error"}}'
        ok_key = f'falcosidekick_outputs_total{{output="{output}",status="ok"}}'
        error_count = metrics.get(error_key, 0)
        ok_count = metrics.get(ok_key, 0)
        if error_count > ok_count * 0.1:  # >10% error rate
            errors[output] = {"errors": error_count, "ok": ok_count}

    if errors:
        return jsonify({"status": "output_errors", "outputs": errors}), 503

    return jsonify({"status": "ok", "outputs_checked": OUTPUTS_TO_CHECK}), 200

@app.get("/ingestion")
def ingestion():
    try:
        resp = requests.get(f"{SIDEKICK_URL}/metrics", timeout=5)
        resp.raise_for_status()
    except Exception as e:
        return jsonify({"status": "sidekick_unreachable", "error": str(e)}), 503

    metrics = parse_prometheus(resp.text)
    inputs_key = 'falcosidekick_inputs_total'

    # Find the key with matching prefix
    inputs = sum(v for k, v in metrics.items() if k.startswith(inputs_key))

    return jsonify({"status": "ok", "total_events_ingested": inputs}), 200

if __name__ == "__main__":
    app.run(host="0.0.0.0", port=9191)

Deploy as a sidecar or standalone:

# sidekick-health-bridge-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: sidekick-health-bridge
  namespace: falco
spec:
  replicas: 1
  selector:
    matchLabels:
      app: sidekick-health-bridge
  template:
    metadata:
      labels:
        app: sidekick-health-bridge
    spec:
      containers:
        - name: bridge
          image: python:3.12-slim
          command: ["sh", "-c", "pip install flask requests -q && python /app/sidekick-health-bridge.py"]
          env:
            - name: SIDEKICK_URL
              value: "http://falco-sidekick:2801"
            - name: OUTPUTS
              value: "slack,pagerduty,webhook"
          ports:
            - containerPort: 9191

Step 3: Monitor Falco event ingestion throughput

If Falco stops detecting events entirely (kernel module crash, rules error), Sidekick will appear healthy but receive nothing. Detect this by checking that the ingestion count is increasing over time:

# Add to sidekick-health-bridge.py
import threading, time

_last_count = {"ts": 0, "count": 0}
_lock = threading.Lock()

def _poll_ingestion():
    global _last_count
    while True:
        time.sleep(60)
        try:
            resp = requests.get(f"{SIDEKICK_URL}/metrics", timeout=5)
            metrics = parse_prometheus(resp.text)
            inputs_key = 'falcosidekick_inputs_total'
            count = sum(v for k, v in metrics.items() if k.startswith(inputs_key))
            with _lock:
                _last_count = {"ts": time.time(), "count": count}
        except Exception:
            pass

threading.Thread(target=_poll_ingestion, daemon=True).start()

@app.get("/throughput")
def throughput():
    with _lock:
        snapshot = dict(_last_count)
    age = time.time() - snapshot["ts"]
    if age > 180:
        return jsonify({"status": "stale_metrics", "age_seconds": int(age)}), 503
    return jsonify({"status": "ok", "total_events": snapshot["count"]}), 200

In a typical cluster with Falco running, there will always be some events (syscall noise, audit events). If the counter is stuck for more than an hour in a busy cluster, it is likely that Falco has stopped publishing events.

Step 4: Send a canary event to validate end-to-end delivery

The strongest test is to inject a known Falco event and verify it reaches your Slack/PagerDuty channel. Falco Sidekick accepts events via HTTP POST:

# Send a canary event to Sidekick directly
curl -X POST http://falco-sidekick:2801/ \
  -H "Content-Type: application/json" \
  -d '{
    "output": "VIGILMON_CANARY: End-to-end pipeline test",
    "priority": "Debug",
    "rule": "vigilmon_canary",
    "source": "syscalls",
    "tags": ["test", "vigilmon"],
    "time": "2026-07-01T02:00:00Z"
  }'

Run this from a CronJob every 6 hours and verify it arrives in your alert channel. Add a heartbeat to Vigilmon from the CronJob:

# canary-cronjob.yaml
apiVersion: batch/v1
kind: CronJob
metadata:
  name: sidekick-canary
  namespace: falco
spec:
  schedule: "0 */6 * * *"
  jobTemplate:
    spec:
      template:
        spec:
          restartPolicy: Never
          containers:
            - name: canary
              image: curlimages/curl:latest
              command: ["/bin/sh", "-c"]
              args:
                - |
                  curl -sf -X POST http://falco-sidekick:2801/ \
                    -H "Content-Type: application/json" \
                    -d '{"output":"VIGILMON_CANARY","priority":"Debug","rule":"vigilmon_canary","source":"syscalls","time":"'"$(date -u +%Y-%m-%dT%H:%M:%SZ)"'"}' \
                  && curl -sf "https://vigilmon.online/heartbeat/YOUR_CANARY_TOKEN"

Step 5: Configure Vigilmon monitors

Sidekick liveness

  • Type: HTTP
  • URL: http://falco-sidekick:2801/healthz (via ingress or internal lb)
  • Expected: 200
  • Interval: 1 minute

Output channel health

  • Type: HTTP
  • URL: http://sidekick-health-bridge:9191/health
  • Expected: 200
  • Interval: 2 minutes

Event ingestion throughput

  • Type: HTTP
  • URL: http://sidekick-health-bridge:9191/throughput
  • Expected: 200
  • Interval: 5 minutes

Canary delivery

  • Type: Heartbeat
  • Expected interval: 7h (canary runs every 6h)
  • Alert: Missing heartbeat = end-to-end pipeline broken

Vigilmon integration summary

| Signal | Monitor type | Alert condition | |--------|-------------|-----------------| | Sidekick alive | HTTP /healthz | Non-200 / timeout | | Output channel errors | HTTP /health | >10% error rate | | Event ingestion | HTTP /throughput | Stale or no events | | End-to-end delivery | Heartbeat | No canary in 7h |

Next steps

  • Add a Vigilmon status page component "Runtime Security Alerting" so the SOC can see pipeline health independently of cluster access
  • Wire Vigilmon → PagerDuty for canary misses with a severity of P1—a broken alert pipeline means ongoing attacks go undetected
  • Use Vigilmon's incident timeline to detect patterns in Sidekick restarts that correlate with kernel updates or node maintenance windows

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles