How to Monitor Falcosidekick with Vigilmon (HTTP + TCP + Alerts)

Falcosidekick is the output router and alert fanout engine for Falco, the cloud-native Kubernetes security monitoring tool. It receives security events from Falco and fans them out to dozens of outputs simultaneously — Slack, PagerDuty, Splunk, Elasticsearch, Datadog, webhooks, and more. If Falcosidekick goes down, all Falco security alerts stop reaching your team, your SIEM, and your incident management system simultaneously — a complete blind spot in your Kubernetes security posture. Vigilmon adds the external perspective that Falcosidekick cannot provide for itself: continuous uptime and latency checks from outside your infrastructure that keep watching even when the security alert routing layer has failed. You can add your first monitor for free, no credit card required.

Why Falcosidekick needs external monitoring

Process crash silences all Falco security alerts across every output. Falcosidekick is the single delivery mechanism for every Falco alert in your cluster. When the Falcosidekick process crashes — due to OOM, an unhandled panic in an output plugin, or a configuration error during a hot reload — every Falco alert generated anywhere in the cluster is silently dropped. Your Slack channel stops receiving security notifications, your SIEM stops ingesting events, and your PagerDuty integrations go dark. Because the failure is in the delivery layer itself, no alert fires from the outputs that are no longer receiving data.
Output plugin failures cause partial alert delivery without surfacing errors. Falcosidekick fans alerts out to multiple outputs concurrently. If one output's upstream service is unreachable — the Slack API is down, PagerDuty is rate-limiting, or an Elasticsearch cluster is overwhelmed — Falcosidekick retries those outputs but continues delivering to others. From the outside, some teams stop receiving alerts while others continue normally. Without monitoring the Falcosidekick health endpoint and output queue metrics, this partial failure goes unnoticed until someone manually audits delivery counts across outputs.
Queue backpressure causes the Falco event socket to block. Falcosidekick maintains an internal queue between its Falco input and its output goroutines. If downstream outputs become slow — network latency to a SIEM, a slow Elasticsearch indexing pipeline, or API rate limiting — the queue fills and the Falcosidekick process begins applying backpressure to the Falco gRPC output socket. Falco's internal buffer then fills and it starts dropping events at the source. This cascading failure produces no explicit error — Falco reports output writes as slow rather than failed — and an external HTTP monitor on the Falcosidekick health endpoint is the most reliable way to detect the degradation before events are dropped.
Configuration reload on a bad config file crashes the process. Falcosidekick supports configuration hot reload triggered by a SIGHUP or a webhook. If a configuration update introduces a syntax error — an invalid output URL, a malformed authentication credential, or an unknown field name — the reload attempt panics the process. All in-flight alerts are dropped, and the process does not restart until Kubernetes detects the crash and schedules a new pod. During the restart window, the security alert pipeline is completely dead. An external uptime monitor provides the only signal that the pipeline has gone offline.
Falcosidekick-UI inaccessibility hides security alert history from the team. Many deployments run Falcosidekick alongside the Falcosidekick-UI, a web dashboard that displays received Falco events for manual review. If the UI process or its Redis backend becomes unavailable, engineers investigating a security event cannot access historical alert data from the dashboard. Internal health checks that only probe the Falcosidekick API miss this failure because the routing pipeline itself remains healthy. An external HTTP monitor on the UI endpoint surfaces this class of failure independently.

What you'll need

A running Falcosidekick instance with its HTTP API accessible over HTTPS
The public hostname or IP for your Falcosidekick deployment, for example https://falcosidekick.example.com
A free Vigilmon account — monitors start running in under a minute with no credit card required

Step 1: Expose Falcosidekick's health endpoint

Verify your Falcosidekick installation is reachable and locate the health endpoint before configuring Vigilmon monitors.

# Check Falcosidekick health endpoint
curl -I https://falcosidekick.example.com/healthz

# Sample output:
# HTTP/2 200
# content-type: application/json
# x-falcosidekick-version: 2.29.0

# Verify the ping endpoint
curl -s https://falcosidekick.example.com/ping
# Expected: {"message":"pong","error":""}

# Check metrics endpoint (if enabled)
curl -s https://falcosidekick.example.com/metrics | head -10
# Expected: Prometheus-format text beginning with # HELP falcosidekick_

If Falcosidekick is running in Kubernetes, verify the Service and pod status:

# Check Falcosidekick deployment
kubectl get deployment -n falco falcosidekick

# Expected:
# NAME            READY   UP-TO-DATE   AVAILABLE
# falcosidekick   2/2     2            2

# Check pod status
kubectl get pods -n falco -l app.kubernetes.io/name=falcosidekick -o wide

# Test from inside the cluster
kubectl run tmp-curl --image=curlimages/curl --rm -it --restart=Never \
  --namespace=falco -- curl -s http://falcosidekick.falco.svc.cluster.local:2801/healthz

If Falcosidekick is running in Docker alongside Falco, confirm the container is healthy:

# Check container status
docker ps --filter name=falcosidekick --format "table {{.Names}}\t{{.Status}}\t{{.Ports}}"

# Sample output:
# NAMES            STATUS        PORTS
# falcosidekick    Up 2 days     0.0.0.0:2801->2801/tcp

# Test health inside the container
docker exec falcosidekick curl -s http://localhost:2801/healthz

Step 2: Set up HTTP monitoring in Vigilmon

Add an HTTP monitor for the Falcosidekick health endpoint:

Log in to vigilmon.online and click Add Monitor.
Set Monitor type to HTTP.
Enter the URL: https://falcosidekick.example.com/healthz
Set Check interval to 60 seconds and Expected status code to 200.
Click Save Monitor — Vigilmon will run the first check within a minute.

Add a monitor for the Falcosidekick ping endpoint

The ping endpoint provides a lightweight liveness check that does not require authentication:

Click Add Monitor again.
Set Monitor type to HTTP.
Enter the URL: https://falcosidekick.example.com/ping
Set Expected status code to 200 and Timeout to 10 seconds.
Under Response body contains, enter pong to confirm the routing process is alive and responding.
Click Save Monitor.

Why external monitoring catches what internal checks miss

Falcosidekick is the delivery layer for your Kubernetes security alerts — when it fails, no alerts reach any of your outputs, including the ones that would normally notify your team of an outage. Internal Kubernetes health probes check whether the Falcosidekick pod is running and whether it returns a liveness response from inside the cluster network — but they cannot detect whether output plugins are actually delivering alerts, whether the event queue is backing up, or whether a configuration reload has crashed the process. Vigilmon operates entirely outside your infrastructure, probing Falcosidekick from external network vantage points on a fixed interval. It is the only delivery path for alerts about failures in the alert delivery system itself.

Step 3: Monitor Falcosidekick's TCP port

A TCP monitor confirms that the Falcosidekick server is accepting connections at the network layer before HTTP processing begins. This catches scenarios where the HTTP listener goroutine has panicked but the main process is still alive, or where a cloud load balancer is failing to route traffic to the Falcosidekick pods.

In Vigilmon, click Add Monitor.
Set Monitor type to TCP.
Enter Host: falcosidekick.example.com and Port: 443.
Set Check interval to 60 seconds.
Click Save Monitor.

If your Falcosidekick instance is exposed on its default port 2801 directly without a TLS-terminating proxy, add a second TCP monitor:

# Find the ports Falcosidekick is listening on
kubectl get pod -n falco -l app.kubernetes.io/name=falcosidekick \
  -o jsonpath='{.items[0].spec.containers[0].ports}' | jq .

# Sample output:
# [
#   {"containerPort": 2801, "name": "http",      "protocol": "TCP"},
#   {"containerPort": 2802, "name": "prometheus", "protocol": "TCP"}
# ]

Add separate TCP monitors for port 2801 (HTTP API) and port 2802 (metrics) so you can isolate whether a failure is affecting the routing pipeline, the metrics endpoint, or both.

Step 4: Configure alert channels

Email alerts

In Vigilmon, go to Alert Channels and click Add Channel.
Choose Email and enter the address that should receive notifications (for example, security-ops@example.com or an on-call distribution list).
Assign this channel to all Falcosidekick monitors created in Steps 2 and 3.
Click Save — Vigilmon will send a test email to confirm delivery.

Webhook alerts

For Slack, PagerDuty, or an internal incident management system, add a webhook channel:

In Alert Channels, click Add Channel and choose Webhook.
Paste your endpoint URL (a Slack incoming webhook, PagerDuty Events API v2 endpoint, or custom URL).
Vigilmon will POST the following JSON payload when a monitor transitions to down:

{
  "monitor_id": "mon_falcosidekick_healthz",
  "monitor_name": "Falcosidekick /healthz",
  "status": "down",
  "previous_status": "up",
  "checked_at": "2026-07-12T09:14:22Z",
  "response_time_ms": null,
  "status_code": null,
  "error": "Connection refused",
  "url": "https://falcosidekick.example.com/healthz"
}

Assign all Falcosidekick monitors to this channel and click Save.

Runbook: diagnostics to run when an alert fires

#!/bin/bash
# Falcosidekick alert runbook — run these in order when Vigilmon fires

echo "=== Deployment status ==="
kubectl get deployment -n falco falcosidekick

echo "=== Pod status ==="
kubectl get pods -n falco -l app.kubernetes.io/name=falcosidekick -o wide

echo "=== Recent events ==="
kubectl get events -n falco --sort-by='.lastTimestamp' | tail -20

echo "=== Health endpoint response ==="
curl -sv https://falcosidekick.example.com/healthz 2>&1 | tail -20

echo "=== Ping response ==="
curl -s https://falcosidekick.example.com/ping

echo "=== Output statistics ==="
curl -s https://falcosidekick.example.com/metrics | grep falcosidekick_events | head -20

echo "=== Port connectivity ==="
nc -zv falcosidekick.example.com 443

echo "=== TLS certificate expiry ==="
echo | openssl s_client -connect falcosidekick.example.com:443 \
  -servername falcosidekick.example.com 2>/dev/null \
  | openssl x509 -noout -enddate

echo "=== Falcosidekick logs (last 100 lines) ==="
kubectl logs -n falco -l app.kubernetes.io/name=falcosidekick --tail=100

echo "=== Falco logs (are events being generated?) ==="
kubectl logs -n falco -l app.kubernetes.io/name=falco --tail=50 | grep -i "output"

Step 5: Create a public status page

A Vigilmon status page gives your security operations team and any teams that depend on Falco alerting a single URL to check Falcosidekick health during incidents — no Kubernetes access required.

In Vigilmon, navigate to Status Pages and click New Status Page.
Give it a name such as Falco Security Alert Pipeline and choose a subdomain like falco-status.vigilmon.page.
Add all monitors created in this tutorial: the /healthz HTTP monitor, the /ping HTTP monitor, and the TCP port monitor.
Optionally add a description explaining which security tools and teams depend on Falcosidekick for alert delivery.
Click Publish — the page is immediately accessible and updates in real time.

Share this URL in your security incident response runbook and link it from your SOC documentation so engineers know where to check when security alerts stop flowing.

Putting it all together

| Monitor | Type | What it catches | |---|---|---| | https://falcosidekick.example.com/healthz | HTTP | Process crash, OOM kill, config reload failure, TLS expiry | | https://falcosidekick.example.com/ping | HTTP | Routing pipeline liveness, queue backpressure, HTTP listener failure | | falcosidekick.example.com:443 | TCP | Network-level connectivity loss, load balancer failure, port binding failure |

# Quick-reference diagnostic one-liners when a Vigilmon alert fires

# 1. Are Falcosidekick pods running?
kubectl get pods -n falco -l app.kubernetes.io/name=falcosidekick -o wide

# 2. Are there recent errors in the router?
kubectl logs -n falco -l app.kubernetes.io/name=falcosidekick --tail=50

# 3. Can the health endpoint be reached?
curl -sf https://falcosidekick.example.com/healthz || echo "Health check FAILED"

# 4. Is the ping returning?
curl -s https://falcosidekick.example.com/ping | grep pong || echo "Ping FAILED"

# 5. Is port 443 open?
nc -zv falcosidekick.example.com 443 && echo "Port open" || echo "Port CLOSED"

# 6. How long until the TLS cert expires?
echo | openssl s_client -connect falcosidekick.example.com:443 \
  -servername falcosidekick.example.com 2>/dev/null \
  | openssl x509 -noout -enddate

# 7. How many events has Falcosidekick delivered today?
curl -s https://falcosidekick.example.com/metrics \
  | grep 'falcosidekick_events_total' | head -5

What's next

SSL certificate monitoring — Vigilmon can alert you days before the TLS certificate on falcosidekick.example.com expires, giving you time to renew before the security alert pipeline is interrupted. Add an SSL monitor pointing at the same hostname.
Heartbeat monitoring for pipeline continuity — Configure Falco to emit a synthetic test event on a regular schedule (every 5 minutes) and configure Falcosidekick to forward it to a Vigilmon heartbeat endpoint. If the synthetic event stops arriving, Vigilmon fires an alert confirming that the end-to-end Falco → Falcosidekick pipeline is broken.
Latency trend monitoring — Use Vigilmon's response time history to establish a latency baseline for the /healthz and /ping endpoints. Sudden latency increases often indicate output plugin backpressure building in the routing queue and give you time to investigate before events start being dropped.

Get started free at vigilmon.online — no credit card, monitors start running in under a minute.