Monitoring Tracee Runtime Security with Vigilmon

Tracee is Aqua Security's open-source eBPF-based runtime security and forensics tool for Linux and Kubernetes. It traces system calls and OS events to detect threats in real time—privilege escalation, container escapes, unexpected binary execution, network reconnaissance. When Tracee goes down or stops generating events, attacks go undetected. When detection rules break, you're blind to entire threat categories.

This tutorial shows how to monitor Tracee's event pipeline health, detection rule effectiveness, Prometheus metrics export, and output channel delivery using Vigilmon—so your runtime security layer is always observable.

Why Tracee needs external monitoring

Tracee fails in ways that look fine from the outside:

• eBPF probe detaches after kernel update — Tracee's tracee-ebpf process exits or fails to load probes after a kernel version change; events stop while the container stays in Running state
• Detection rule set silently errors — a Rego policy or Go signature with a syntax error disables that detection class without surfacing an error to kubectl
• Output stream backs up — Tracee writes events to stdout, a file, or a webhook; if the consumer is slow or dead, Tracee's internal buffer fills and drops events
• Falco rules integration breaks — when using Tracee with Falco rule syntax, a malformed rule file causes the ruleset to fail to load on startup without crash-looping the pod
• Metrics endpoint stops responding — Tracee's Prometheus endpoint returns stale or no data after an internal error, causing your dashboards to show "no data" instead of an alert

What you'll need

• Tracee deployed in your Kubernetes cluster (as a DaemonSet)
• Tracee's Prometheus metrics exposed
• A free Vigilmon account

Step 1: Deploy Tracee with metrics enabled

Install Tracee via Helm with Prometheus metrics enabled:

helm repo add aqua https://aquasecurity.github.io/helm-charts/
helm repo update

helm install tracee aqua/tracee \
  --namespace tracee-system \
  --create-namespace \
  --set config.output.options.parseArguments=true \
  --set serviceMonitor.enabled=true \
  --set metrics.enabled=true \
  --set metrics.port=3366

Verify the metrics endpoint:

kubectl port-forward -n tracee-system ds/tracee 3366:3366
curl http://localhost:3366/metrics | grep tracee

You should see:

# HELP tracee_events_total Total number of events traced
# TYPE tracee_events_total counter
tracee_events_total{event="execve"} 12043
tracee_events_total{event="openat"} 58201
tracee_events_total{event="connect"} 3821

# HELP tracee_signatures_total Events matched by signatures
# TYPE tracee_signatures_total counter
tracee_signatures_total{signature="TRC-1",severity="critical"} 0
tracee_signatures_total{signature="TRC-2",severity="warning"} 14

Step 2: Build a Tracee health bridge

Create a health endpoint that validates Tracee is actively generating events:

# tracee-health-bridge.py
import os, time, requests
from flask import Flask, jsonify

app = Flask(__name__)
TRACEE_METRICS_URL = os.environ.get("TRACEE_METRICS_URL", "http://tracee.tracee-system:3366/metrics")
MIN_EXECVE_RATE = float(os.environ.get("MIN_EXECVE_RATE", "1.0"))  # events/minute

_baseline = {}
_baseline_time = 0

def parse_counter(text, metric_name):
    for line in text.splitlines():
        if line.startswith(f'{metric_name}{{event="execve"}}'):
            return float(line.split()[-1])
    return None

@app.route("/health")
def health():
    global _baseline, _baseline_time
    try:
        resp = requests.get(TRACEE_METRICS_URL, timeout=5)
        resp.raise_for_status()
        text = resp.text

        current_execve = parse_counter(text, "tracee_events_total")
        if current_execve is None:
            return jsonify({"status": "error", "message": "tracee_events_total metric missing"}), 503

        now = time.time()
        if not _baseline or (now - _baseline_time) < 30:
            _baseline = {"execve": current_execve}
            _baseline_time = now
            return jsonify({"status": "ok", "message": "Establishing baseline"}), 200

        elapsed_min = (now - _baseline_time) / 60
        rate = (current_execve - _baseline["execve"]) / elapsed_min

        _baseline = {"execve": current_execve}
        _baseline_time = now

        if rate < MIN_EXECVE_RATE:
            return jsonify({
                "status": "error",
                "message": f"execve event rate {rate:.2f}/min below minimum {MIN_EXECVE_RATE}/min",
                "hint": "eBPF probes may have detached or process is idle"
            }), 503

        return jsonify({"status": "ok", "execve_rate_per_min": round(rate, 2)}), 200

    except requests.RequestException as e:
        return jsonify({"status": "error", "message": f"Cannot reach Tracee metrics: {e}"}), 503

if __name__ == "__main__":
    app.run(host="0.0.0.0", port=8080)

Deploy alongside Tracee:

# tracee-bridge.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: tracee-health-bridge
  namespace: monitoring
spec:
  replicas: 1
  selector:
    matchLabels:
      app: tracee-health-bridge
  template:
    metadata:
      labels:
        app: tracee-health-bridge
    spec:
      containers:
        - name: bridge
          image: python:3.12-slim
          command: ["python", "/app/tracee-health-bridge.py"]
          env:
            - name: TRACEE_METRICS_URL
              value: "http://tracee.tracee-system.svc:3366/metrics"
            - name: MIN_EXECVE_RATE
              value: "1.0"
          ports:
            - containerPort: 8080
---
apiVersion: v1
kind: Service
metadata:
  name: tracee-health-bridge
  namespace: monitoring
spec:
  selector:
    app: tracee-health-bridge
  ports:
    - port: 80
      targetPort: 8080

Step 3: Monitor Tracee detection signatures

Tracee ships with built-in Go-based signatures (TRC-1 through TRC-N) that detect common attack patterns. Verify they're active:

# List loaded signatures
kubectl exec -n tracee-system ds/tracee -- tracee list signatures

# Check for signature load errors in logs
kubectl logs -n tracee-system ds/tracee | grep -i "signature\|error\|failed"

Add a signature health check to your bridge:

import re

def check_signatures(text):
    """Return list of signatures that have zero detections over the run lifetime."""
    sig_counts = {}
    for line in text.splitlines():
        m = re.match(r'tracee_signatures_total\{signature="([^"]+)".*\} (\d+)', line)
        if m:
            sig_counts[m.group(1)] = int(m.group(2))
    return sig_counts

@app.route("/signatures")
def signature_health():
    try:
        resp = requests.get(TRACEE_METRICS_URL, timeout=5)
        resp.raise_for_status()
        counts = check_signatures(resp.text)
        return jsonify({"status": "ok", "signatures": counts, "total": len(counts)}), 200
    except Exception as e:
        return jsonify({"status": "error", "message": str(e)}), 503

Step 4: Configure Vigilmon monitors

Monitor 1 — Tracee metrics endpoint:

1. Log in to vigilmon.online → Add Monitor
2. Type: HTTP
3. URL: your Tracee metrics endpoint (via ingress or NodePort)
4. Keyword: tracee_events_total
5. Interval: 1 minute

Monitor 2 — Event throughput health bridge:

1. Add Monitor → HTTP
2. URL: https://your-cluster/tracee-health
3. Expected status: 200
4. Interval: 2 minutes

Monitor 3 — Tracee webhook output (if configured):

If you've configured Tracee to forward detections to a webhook receiver:

1. Add Monitor → HTTP
2. URL: https://your-tracee-webhook-receiver/health
3. Expected status: 200
4. Interval: 1 minute

Step 5: Set Prometheus alert rules for Tracee

# tracee-alerts.yaml
groups:
  - name: tracee-runtime-security
    rules:
      - alert: TraceeEventRateZero
        expr: rate(tracee_events_total{event="execve"}[10m]) == 0
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "Tracee not generating execve events"
          description: "No execve events for 5m — eBPF probes likely detached"

      - alert: TraceeErrorsIncreasing
        expr: rate(tracee_errors_total[5m]) > 0.1
        for: 2m
        labels:
          severity: warning
        annotations:
          summary: "Tracee reporting internal errors"
          description: "Error rate above threshold — check Tracee logs for probe failures"

      - alert: TraceeCriticalDetection
        expr: increase(tracee_signatures_total{severity="critical"}[5m]) > 0
        labels:
          severity: critical
        annotations:
          summary: "Tracee critical signature triggered"
          description: "A critical-severity Tracee signature fired — investigate immediately"

Step 6: Alert routing in Vigilmon

1. In Vigilmon → Alerts → Notification Channels
2. Add your on-call channel (Slack, PagerDuty, email)
3. For the event throughput monitor, configure:
   • Alert after: 1 consecutive failure
   • Recovery notification: enabled (critical to know when security resumes)
4. For the webhook output monitor, configure:
   • Alert after: 2 consecutive failures (tolerate brief network blips)

What you're now monitoring

With these monitors in place:

• eBPF probe health — zero execve events triggers an alert within 5 minutes
• Detection signature coverage — signature count drops surface configuration issues
• Output pipeline integrity — webhook delivery failures are caught immediately
• Internal errors — Tracee's own error metrics are tracked in Prometheus

Tracee gives you deep Linux kernel-level visibility into attacks in real time. Vigilmon ensures that visibility never goes dark without you knowing.