Monitoring GRR Rapid Response with Vigilmon

GRR Rapid Response is Google's open source incident response framework — it lets security teams remotely collect memory images, disk artifacts, file system data, and registry snapshots from thousands of endpoints simultaneously, without physical access. When a breach happens, GRR is the tool investigators reach for first.

That makes GRR's own availability critical. If the Admin UI goes down, investigators are blind. If the frontend server goes down, endpoints stop uploading forensic data. If workers fall behind, you're queuing evidence you can't access. Vigilmon gives you always-on monitoring for every GRR component so your forensic infrastructure is never itself a blind spot.

What You'll Set Up

• HTTP uptime monitor for the GRR Admin UI (port 8000)
• TCP monitor for the GRR frontend gRPC endpoint (port 8001)
• Heartbeat monitors for GRR worker processes and filestore health
• Alerting thresholds for hunt failure rate, client checkin lag, and MySQL connectivity

Prerequisites

• GRR Rapid Response server deployed (Admin UI on port 8000, frontend on port 8001)
• MySQL or Cloud Spanner backend running and accessible
• A free Vigilmon account

Step 1: Monitor the GRR Admin UI

The Admin UI is a Python Flask web application on port 8000 (HTTPS). It's the single interface all investigators use to launch hunts, review collected artifacts, and manage endpoints. Downtime here blocks all active incident response work.

1. Log in to vigilmon.online and click Add Monitor.
2. Set Type to HTTP / HTTPS.
3. Enter your GRR Admin UI URL: https://grr.yourdomain.com:8000/api/config.
4. Set Check interval to 1 minute.
5. Set Expected HTTP status to 200.
6. Enable Monitor SSL certificate and set Alert when expires in less than 21 days.
7. Click Save.

GRR exposes /api/config as a lightweight health probe — it requires authentication for the full response but returns a recognizable payload that confirms the Flask application is alive. If you have unauthenticated health check middleware configured, use /api/health instead.

Step 2: Monitor the GRR Frontend Server (Port 8001)

The frontend server handles all gRPC connections from GRR clients (agents) running on monitored endpoints. If it goes down, endpoints can no longer upload forensic data or receive new task assignments. You won't know hunts are failing until investigators notice empty results.

1. Click Add Monitor.
2. Set Type to TCP Port.
3. Enter your GRR server hostname and port 8001.
4. Set Check interval to 1 minute.
5. Click Save.

A TCP check confirms the frontend process is listening. GRR clients use gRPC over this port, so a successful TCP handshake means the server process is alive.

Alert on frontend unhealthy: Set the alert threshold to 1 failed check (immediate alert). A down frontend means zero forensic data collection from any endpoint.

Step 3: Monitor GRR Worker Pool Health

GRR workers run as a fleet of Python processes that consume collected forensic data, execute analysis flows, and update hunt results. A healthy worker pool keeps the analysis queue shallow. A degraded pool lets the queue grow unbounded — hunts appear to run but results never arrive.

Add a cron heartbeat to track worker health:

1. Click Add Monitor → Cron Heartbeat.
2. Set the expected ping interval to 5 minutes.
3. Copy the heartbeat URL.

Add a GRR worker health script that pings Vigilmon after confirming workers are responsive:

#!/usr/bin/env python3
import subprocess
import requests
import sys

HEARTBEAT_URL = "https://vigilmon.online/heartbeat/YOUR_ID_HERE"
MIN_WORKERS = 2  # adjust to your deployment

# Count running worker processes
result = subprocess.run(
    ["pgrep", "-c", "-f", "grr_worker"],
    capture_output=True, text=True
)
worker_count = int(result.stdout.strip() or "0")

if worker_count >= MIN_WORKERS:
    requests.get(HEARTBEAT_URL, timeout=5)
else:
    print(f"Only {worker_count} workers running (need {MIN_WORKERS})", file=sys.stderr)
    sys.exit(1)

Schedule this script with cron every 5 minutes:

*/5 * * * * /opt/grr/bin/check_workers.py

If workers fall below the minimum, the heartbeat stops and Vigilmon alerts you.

Step 4: Monitor MySQL Database Health

GRR stores all forensic data, hunt configurations, client records, and flow results in MySQL. Database connectivity loss means GRR cannot read or write any forensic data — the system appears to run but nothing persists.

1. Click Add Monitor → TCP Port.
2. Enter your MySQL server hostname and port 3306.
3. Set Check interval to 1 minute.
4. Click Save.

For deeper MySQL health visibility, add a heartbeat from a lightweight query probe:

#!/usr/bin/env python3
import pymysql
import requests

HEARTBEAT_URL = "https://vigilmon.online/heartbeat/YOUR_MYSQL_ID_HERE"
DB_CONFIG = {
    "host": "localhost",
    "port": 3306,
    "user": "grr",
    "password": "YOUR_PASSWORD",
    "database": "grr",
    "connect_timeout": 5,
}

try:
    conn = pymysql.connect(**DB_CONFIG)
    with conn.cursor() as cur:
        cur.execute("SELECT 1")
    conn.close()
    requests.get(HEARTBEAT_URL, timeout=5)
except Exception as e:
    print(f"MySQL probe failed: {e}")

Schedule it every 2 minutes. This catches cases where the MySQL process is listening but the GRR database itself is inaccessible (auth failure, table corruption, replication failure).

Step 5: Monitor GRR Filestore Health

GRR stores collected files — memory images, disk images, extracted files — in a local filestore (or cloud storage). Filestore exhaustion silently prevents artifact collection: hunts complete but files are not saved.

Add a disk utilization heartbeat for your filestore path:

#!/bin/bash
HEARTBEAT_URL="https://vigilmon.online/heartbeat/YOUR_FILESTORE_ID"
FILESTORE_PATH="/var/lib/grr/filestore"
THRESHOLD=80  # percent

USAGE=$(df "$FILESTORE_PATH" | awk 'NR==2 {print $5}' | tr -d '%')

if [ "$USAGE" -lt "$THRESHOLD" ]; then
    curl -s "$HEARTBEAT_URL" > /dev/null
else
    echo "Filestore at ${USAGE}% — threshold ${THRESHOLD}%" >&2
fi

Schedule every 10 minutes:

*/10 * * * * /opt/grr/bin/check_filestore.sh

Set the Vigilmon heartbeat interval to 15 minutes with a grace period to avoid false alerts on slow checks.

Step 6: Monitor Client Checkin Latency

GRR clients poll the frontend server on a configurable interval (poll_max, default 10 minutes). A sudden drop in active clients or a surge in clients missing their checkin window indicates agent connectivity problems — possibly a network change, a GRR client update that broke polling, or the frontend going unreachable.

Add a checkin lag probe that queries the GRR API for recently-seen clients:

#!/usr/bin/env python3
"""
Alerts if fewer than expected clients checked in within the last 30 minutes.
Requires a GRR API token with read access to clients.
"""
import requests
import sys
from datetime import datetime, timedelta, timezone

GRR_API = "https://grr.yourdomain.com:8000"
GRR_TOKEN = "YOUR_API_TOKEN"
HEARTBEAT_URL = "https://vigilmon.online/heartbeat/YOUR_CHECKIN_ID"
MIN_ACTIVE_CLIENTS = 100  # adjust to your fleet size
WINDOW_MINUTES = 30

cutoff = datetime.now(timezone.utc) - timedelta(minutes=WINDOW_MINUTES)
cutoff_ts = int(cutoff.timestamp() * 1_000_000)  # GRR uses microseconds

resp = requests.get(
    f"{GRR_API}/api/clients",
    headers={"Authorization": f"Bearer {GRR_TOKEN}"},
    params={"min_last_seen": cutoff_ts, "count": 1},
    verify=True,
    timeout=10,
)
resp.raise_for_status()
total = resp.json().get("totalCount", 0)

if total >= MIN_ACTIVE_CLIENTS:
    requests.get(HEARTBEAT_URL, timeout=5)
else:
    print(f"Only {total} clients checked in (need {MIN_ACTIVE_CLIENTS})", file=sys.stderr)
    sys.exit(1)

Schedule every 15 minutes.

Step 7: Configure Alert Routing

In Vigilmon, set up alert channels for GRR incidents:

1. Go to Settings → Alerts.
2. Add your on-call channel (email, PagerDuty, Slack webhook, or SMS).
3. Apply different escalation policies per monitor:
   • Admin UI down → immediate page to on-call security engineer
   • Frontend down → immediate page (no endpoint data collection)
   • Worker heartbeat missed → alert after 2 consecutive misses (5-minute intervals = 10 minutes)
   • MySQL TCP down → immediate page
   • Filestore >80% → alert to ops team (not on-call page)
   • Client checkin drop → alert to security team for investigation

For GRR, the Admin UI and frontend are your highest-priority monitors. Use Vigilmon's escalation chains to page a second contact if the first doesn't acknowledge within 15 minutes.

Step 8: Hunt Success Rate Alerting

GRR hunts — batch forensic collections across many endpoints — can fail silently. A hunt that errors on 50% of targets looks successful from the UI unless you check per-hunt error rates.

Add a hunt health probe that queries completed hunts and calculates the error rate:

#!/usr/bin/env python3
import requests
import sys

GRR_API = "https://grr.yourdomain.com:8000"
GRR_TOKEN = "YOUR_API_TOKEN"
HEARTBEAT_URL = "https://vigilmon.online/heartbeat/YOUR_HUNT_ID"
MAX_ERROR_RATE = 0.20  # alert if >20% of clients in recent hunts errored

resp = requests.get(
    f"{GRR_API}/api/hunts",
    headers={"Authorization": f"Bearer {GRR_TOKEN}"},
    params={"state": "COMPLETED", "count": 5},
    timeout=10,
)
resp.raise_for_status()
hunts = resp.json().get("items", [])

if not hunts:
    requests.get(HEARTBEAT_URL, timeout=5)
    sys.exit(0)

total_clients = sum(h.get("allClientsCount", 0) for h in hunts)
error_clients = sum(h.get("clientsWithErrorsCount", 0) for h in hunts)
error_rate = error_clients / total_clients if total_clients > 0 else 0

if error_rate <= MAX_ERROR_RATE:
    requests.get(HEARTBEAT_URL, timeout=5)
else:
    print(f"Hunt error rate {error_rate:.1%} exceeds threshold {MAX_ERROR_RATE:.1%}")
    sys.exit(1)

Schedule every 30 minutes. If hunt error rates spike — common when a GRR client update is pushed that breaks the Python agent — you'll know before investigators do.

Key Metrics Summary

| Monitor | Type | Interval | Alert Threshold | |---------|------|----------|----------------| | GRR Admin UI | HTTP | 1 min | 1 failure | | GRR Frontend (gRPC) | TCP | 1 min | 1 failure | | Worker pool health | Heartbeat | 5 min | 1 miss | | MySQL connectivity | TCP | 1 min | 1 failure | | MySQL query probe | Heartbeat | 2 min | 2 misses | | Filestore disk usage | Heartbeat | 10 min | 1 miss (>80%) | | Client checkin rate | Heartbeat | 15 min | 1 miss | | Hunt error rate | Heartbeat | 30 min | 1 miss (>20% error) |

Conclusion

GRR Rapid Response is your forensic infrastructure — when it's down, your incident response capability is down. With Vigilmon monitoring the Admin UI, frontend server, workers, MySQL backend, and filestore, you get alerts the moment any layer degrades rather than discovering failures mid-investigation.

The combination of HTTP/TCP checks for process-level health and cron heartbeats for logical health (enough workers, clients checking in, hunts completing) gives you full-stack visibility into a complex distributed system — without any agents to install on GRR itself.

Set up your first GRR monitor at vigilmon.online.