Monitoring TruffleHog with Vigilmon

TruffleHog is an open source secret scanning tool that searches git repositories, file systems, S3 buckets, and CI/CD pipelines for hardcoded API keys, credentials, and sensitive strings — going beyond simple regex by using entropy analysis and live verification against external APIs (AWS STS, GitHub, Stripe) to confirm found secrets are actually valid. In server mode, TruffleHog Enterprise runs as a persistent scanning service (Go binary on port 3000) that receives push webhooks from GitHub or GitLab, triggers scans automatically, and notifies teams when new secrets are found. If this server goes down silently, your entire automated secret scanning coverage disappears — and no one gets alerted when a developer accidentally commits an AWS key. Vigilmon keeps the TruffleHog scanning infrastructure itself under continuous observation.

What You'll Set Up

• HTTP uptime monitor for the TruffleHog server (port 3000)
• TCP monitor for database connectivity
• Cron heartbeat for scan job queue health
• Webhook intake health check
• SSL certificate expiry alerts for the server API
• Alert channels with appropriate thresholds

Prerequisites

• TruffleHog server or TruffleHog Enterprise running on port 3000
• Database backend (PostgreSQL or SQLite) accessible
• Webhook delivery configured from GitHub/GitLab to the TruffleHog server
• A free Vigilmon account

Step 1: Monitor the TruffleHog Server (Port 3000)

The TruffleHog server API manages scanning jobs, stores results, and provides the dashboard for teams to review secret findings. When it goes down, push webhooks from GitHub and GitLab queue up or are dropped — and no new scans run until the server recovers.

1. Log in to vigilmon.online and click Add Monitor.
2. Set Type to HTTP / HTTPS.
3. URL: https://trufflehog.yourdomain.com/health (or http://your-server-ip:3000/health).
4. Set Check interval to 1 minute.
5. Set Expected HTTP status to 200.
6. Enable Monitor SSL certificate and set Alert when certificate expires in less than 21 days.
7. Click Save.

If TruffleHog Enterprise does not expose a dedicated /health endpoint, use the API root:

1. URL: http://your-server-ip:3000/
2. Expected HTTP status: 200 or 404 — any response other than 502/503/timeout confirms the server process is running.

Step 2: Monitor Database Connectivity

TruffleHog stores scan results, repository metadata, team notification configuration, and audit history in a database. If the database goes down, the server typically fails to accept new scan requests and may stop returning results.

For PostgreSQL:

1. Click Add Monitor → TCP Port.
2. Host: localhost (or your PostgreSQL host).
3. Port: 5432
4. Check interval: 1 minute
5. Click Save.

For a PostgreSQL health endpoint (if you expose one via pgBouncer or a monitoring sidecar):

1. Click Add Monitor → HTTP / HTTPS.
2. URL: http://localhost:8008/healthz (pgBouncer stats or equivalent).
3. Expected HTTP status: 200
4. Check interval: 2 minutes
5. Click Save.

A database outage that takes down TruffleHog's persistence layer means scan results are not saved even if the scan jobs run — making database connectivity a critical signal.

Step 3: Heartbeat Monitor for Scan Job Queue Health

The most dangerous TruffleHog failure mode is not a crash — it's a queue jam: the server is up, webhooks arrive, but scan jobs are stuck or silently failing. A cron heartbeat monitors end-to-end job execution health.

1. Click Add Monitor → Cron Heartbeat.
2. Set the expected interval to 30 minutes (adjust based on your commit frequency — if your team pushes code less frequently, use 60 minutes).
3. Copy the heartbeat URL (e.g., https://vigilmon.online/heartbeat/abc123).
4. Create a lightweight canary script that submits a test scan against a known-clean repository and pings the heartbeat on success:

#!/bin/bash
# /opt/trufflehog/canary-check.sh
RESULT=$(trufflehog git \
  --no-update \
  --json \
  file:///opt/trufflehog/canary-repo \
  2>/dev/null | wc -l)

# A clean canary repo should return 0 findings
if [ "$RESULT" -eq "0" ]; then
  curl -s https://vigilmon.online/heartbeat/abc123
fi

Schedule it:

*/30 * * * * /opt/trufflehog/canary-check.sh

If the heartbeat stops arriving, it means scan jobs are failing or the TruffleHog binary is unable to execute — not just that the HTTP server is down.

Step 4: Monitor Webhook Intake from GitHub/GitLab

TruffleHog's server mode depends on receiving push event webhooks from your SCM provider. If the webhook endpoint is unreachable or returns errors, GitHub/GitLab stops delivering events and no new pushes trigger scans.

1. Click Add Monitor → HTTP / HTTPS.
2. URL: https://trufflehog.yourdomain.com/webhook (or the path your TruffleHog server registers for webhook events).
3. Method: POST
4. Expected HTTP status: 200, 400, or 405 — any of these confirms the endpoint is processing requests; a 502 or timeout means the intake path is broken.
5. Check interval: 5 minutes
6. Click Save.

Pair this monitor with the job queue heartbeat in Step 3. If the webhook endpoint is healthy but the heartbeat stops, the intake is accepting events but jobs are not executing.

Step 5: Monitor Verification Service Connectivity

TruffleHog's verification feature tests found secrets against external APIs (AWS STS, GitHub, Stripe, and others) to confirm they are live credentials rather than rotated or test values. If verification is broken, all findings are marked "unverified" — valid secrets may be buried under false positives, and your team loses the high-confidence signal that makes TruffleHog actionable.

Monitor the external APIs TruffleHog verifies against:

AWS STS (for AWS key verification):

1. Click Add Monitor → TCP Port.
2. Host: sts.amazonaws.com
3. Port: 443
4. Check interval: 5 minutes
5. Click Save.

GitHub API (for GitHub token verification):

1. Click Add Monitor → HTTP / HTTPS.
2. URL: https://api.github.com
3. Expected HTTP status: 200
4. Check interval: 5 minutes
5. Click Save.

If these external monitors go red, expect a spike in "unverified" findings from TruffleHog — not a sign of misconfiguration, but of external API unavailability.

Step 6: Monitor Notification Delivery

TruffleHog can send Slack messages or webhook notifications when new secrets are found. A failure in the notification path means engineers are never alerted about a critical credential exposure even though TruffleHog detected it.

If TruffleHog delivers notifications via a Slack webhook URL:

1. Click Add Monitor → TCP Port.
2. Host: hooks.slack.com
3. Port: 443
4. Check interval: 5 minutes
5. Click Save.

For a self-hosted notification relay (e.g., an internal webhook endpoint):

1. Click Add Monitor → HTTP / HTTPS.
2. URL: https://notifications.yourdomain.com/health
3. Expected HTTP status: 200
4. Check interval: 2 minutes
5. Click Save.

Step 7: Configure Alert Channels

1. Go to Alert Channels in Vigilmon and add Slack, email, or a webhook.
2. For the TruffleHog server monitor, set Consecutive failures before alert to 2 — brief restarts during upgrades cause single-probe failures.
3. For the database TCP monitor, set Consecutive failures to 1 — database loss immediately breaks scan persistence.
4. For the job queue heartbeat, Vigilmon alerts automatically after the expected interval passes without a ping.
5. For webhook intake, set Consecutive failures to 2 — POST probes may get a brief 503 during rolling restarts.
6. For external verification service monitors, set Consecutive failures to 3 to absorb transient internet connectivity blips without flooding alerts.

Summary

| Monitor | Target | What It Catches | |---|---|---| | TruffleHog server | :3000/health | Server crash, unreachable scanning API | | Database TCP | :5432 | Scan result persistence broken | | Scan job heartbeat | Heartbeat URL | Queue jam, job execution failure | | Webhook intake | /webhook POST | Push events not triggering scans | | AWS STS | sts.amazonaws.com:443 | AWS key verification broken | | GitHub API | api.github.com | GitHub token verification broken | | Notification relay | Slack or internal webhook | Secret alerts not delivered | | SSL certificate | Server API domain | TLS expiry on scanning API |

TruffleHog is your automated guard against credential leaks in source code — but only if the scanning infrastructure itself is running. With Vigilmon watching the server API, database, scan job pipeline, webhook intake, and notification delivery, you get alerted when your secret scanner goes dark before a committed AWS key makes it past the detection layer.