Monitoring Trivy Operator with Vigilmon: Kubernetes Security Scanning, Vulnerability Report Health, and Alert Coverage

Trivy Operator is the Kubernetes-native operator that automatically scans your cluster's running containers for OS and application vulnerabilities, exposed secrets, Kubernetes misconfiguration, and RBAC issues — creating VulnerabilityReport, ConfigAuditReport, and ExposedSecretReport custom resources that represent your cluster's security posture in real time. When the operator crashes or its scan jobs fail, vulnerability data goes stale and your security dashboard shows outdated results — giving you false confidence while new critical CVEs are introduced. When the operator can't connect to the Trivy image registry, scans fail silently. Vigilmon gives you external health monitoring for the Trivy Operator: the operator's own health endpoint, the metrics it exposes, and the supporting services it depends on.

What You'll Build

• A monitor on Trivy Operator's health endpoint for operator liveness
• A monitor on the Trivy Operator metrics endpoint for scan job health signals
• An HTTP monitor for the Trivy database update URL to ensure vulnerability definitions stay current
• A monitor for the container registry used to pull the Trivy vulnerability database
• An alerting setup that fires when scans go stale or the operator becomes unhealthy

Prerequisites

• Trivy Operator deployed in your Kubernetes cluster (typically via Helm or Operator Lifecycle Manager)
• Trivy Operator's health endpoint exposed via a Kubernetes Service or Ingress (port 9090 by default)
• A free account at vigilmon.online

Step 1: Verify Trivy Operator's Health Endpoint

Trivy Operator exposes a health endpoint on port 9090 (by default) at /healthz:

# Port-forward the operator service to check locally
kubectl port-forward svc/trivy-operator -n trivy-system 9090:9090

# In another terminal
curl http://localhost:9090/healthz

A healthy Trivy Operator returns:

{"status": "ok"}

To expose this for external monitoring, add an Ingress or LoadBalancer service (or use your existing cluster monitoring ingress):

apiVersion: v1
kind: Service
metadata:
  name: trivy-operator-health
  namespace: trivy-system
spec:
  selector:
    app.kubernetes.io/name: trivy-operator
  ports:
    - port: 9090
      targetPort: 9090
  type: ClusterIP

With an Ingress routing /healthz to this service, you can point Vigilmon at a stable external URL.

Step 2: Create a Vigilmon Monitor for the Trivy Operator Health

1. Log in to Vigilmon → Add Monitor → HTTP.
2. URL: https://trivy-monitor.example.com/healthz (your exposed health endpoint).
3. Check interval: 60 seconds.
4. Response timeout: 10 seconds.
5. Expected status: 200.
6. Keyword: ok.
7. Label: Trivy Operator health.
8. Click Save.

This monitor catches:

• Trivy Operator process crashes or OOMKilled events
• Operator pod evictions due to resource pressure
• CRD reconciliation failures that crash-loop the operator
• Failed upgrades that leave the operator in a broken state

Alert sensitivity: Set alerts after 2 consecutive failures — the operator's Kubernetes pod may be restarting after a transient issue. Two failures mean the operator is genuinely down for more than 2 minutes.

Step 3: Monitor the Trivy Operator Metrics Endpoint

Trivy Operator exposes Prometheus-format metrics that reveal scan job health — including how many scans have succeeded, failed, or are pending. These metrics are a richer signal than the simple health check:

curl http://localhost:9090/metrics | grep trivy_operator

Key metrics to watch:

• trivy_operator_vulnerabilities_count — number of vulnerability reports generated
• trivy_operator_scan_job_failed_total — count of scan jobs that failed

Rather than parsing Prometheus metrics, use Vigilmon's keyword monitor against the metrics endpoint to detect anomalies:

1. Add Monitor → HTTP.
2. URL: https://trivy-monitor.example.com/metrics.
3. Check interval: 5 minutes.
4. Response timeout: 15 seconds.
5. Expected status: 200.
6. Keyword: trivy_operator_vulnerabilities_count (confirms metrics are being emitted).
7. Label: Trivy Operator metrics.
8. Click Save.

Step 4: Monitor the Trivy Vulnerability Database Source

Trivy Operator downloads vulnerability database updates from ghcr.io (GitHub Container Registry) or a mirror you configure. If this registry is unreachable, your vulnerability definitions stop updating and new CVEs go undetected:

# Verify Trivy can pull the vulnerability database
trivy image --download-db-only

# Check the database update endpoint directly
curl -I https://ghcr.io/v2/aquasecurity/trivy-db/manifests/2

1. Add Monitor → HTTP.
2. URL: https://ghcr.io/v2/aquasecurity/trivy-db/manifests/2.
3. Check interval: 10 minutes.
4. Response timeout: 20 seconds.
5. Expected status: 401 (unauthenticated manifests return 401, confirming the registry is up and available).
6. Label: Trivy DB registry (ghcr.io).
7. Click Save.

If you mirror the Trivy vulnerability database internally, monitor your internal mirror instead of ghcr.io. This is common in air-gapped environments where the operator can't reach the internet.

Step 5: Monitor Trivy's Internal Database Update Job

When Trivy Operator runs in Standalone mode, it updates its vulnerability database by running a Kubernetes Job. You can expose a simple last-updated timestamp via a custom endpoint in your monitoring stack, or use Vigilmon's push (heartbeat) monitor to verify the update job runs on schedule:

Alternatively, track the vulnerability database freshness by querying the Trivy DB metadata endpoint:

curl http://localhost:9090/metrics | grep trivy_db_update

If you have a custom monitoring endpoint:

1. Add Monitor → HTTP.
2. URL: https://trivy-monitor.example.com/db-status.
3. Check interval: 30 minutes.
4. Expected status: 200.
5. Keyword: updated.
6. Label: Trivy vulnerability DB freshness.
7. Click Save.

Step 6: Monitor the Trivy Operator Webhook (if enabled)

Trivy Operator can be configured with a mutating or validating webhook that intercepts new pod deployments to trigger immediate scans. If this webhook is unhealthy, Kubernetes may reject pod creation entirely (depending on your failurePolicy setting):

# Check if the webhook is registered and responding
kubectl get validatingwebhookconfigurations | grep trivy

To monitor the webhook server:

# The webhook server typically runs on port 9443
curl -k https://localhost:9443/readyz

1. Add Monitor → HTTP.
2. URL: https://trivy-webhook.example.com/readyz (your exposed webhook readiness endpoint).
3. Check interval: 60 seconds.
4. Expected status: 200.
5. Label: Trivy Operator webhook server.
6. Click Save.

failurePolicy matters: If your webhook is configured with failurePolicy: Fail, a Trivy Operator crash blocks all new pod deployments. Monitor this aggressively and set failurePolicy: Ignore in production unless you're ready to respond 24/7.

Step 7: Configure Alerting

In Vigilmon under Settings → Notifications, configure your alert channels:

| Monitor | Trigger | Action | |---|---|---| | Trivy Operator health | Non-200 or ok missing | Operator down; scans stop; check pod status with kubectl get pods -n trivy-system | | Metrics endpoint | Non-200 | Metrics server crashed; check operator logs | | Trivy DB registry | Non-401/200 | Vulnerability DB updates blocked; CVE data goes stale | | DB freshness endpoint | Non-200 or updated missing | Database update job failing; check Trivy Jobs with kubectl get jobs -n trivy-system | | Webhook server | Non-200 | Webhook degraded; may block pod deployments if failurePolicy: Fail |

Alert channels: Configure a security-focused Slack channel for Trivy Operator alerts — stale vulnerability data is a compliance risk that needs a human decision about acceptable lag time.

Common Trivy Operator Failure Modes and What Vigilmon Catches

| Scenario | Vigilmon monitor | |---|---| | Operator pod OOMKilled | Health endpoint unreachable; alert within 60 s | | Vulnerability DB registry unreachable | DB registry monitor fires; CVE data stops updating | | Scan jobs timeout (large images) | Metrics show scan failures; job failure metric increases | | CRD not installed before operator | Operator crashes on startup; health monitor fires | | Trivy node agent disconnects | Operator health OK but scans missing nodes | | Webhook failurePolicy blocks deployments | Webhook monitor fires; deployments start failing | | Database becomes stale (> 24 hours) | DB freshness monitor fires | | Image registry pull-through cache fails | DB registry monitor fires; Trivy can't update | | Operator upgrade breaks CRD schema | Health monitor catches reconciliation crash | | Namespace quota prevents scan jobs | Scan job failure count rises in metrics |

Trivy Operator automates the vulnerability surveillance that keeps your Kubernetes cluster's security posture current. When the operator is healthy and its database is fresh, you have real-time visibility into every CVE affecting your running workloads. When it goes down, that visibility disappears silently — and new deployments may bring in critical vulnerabilities that go unnoticed for hours or days. Vigilmon gives you continuous external monitoring of the operator's health, metrics, vulnerability database availability, and webhook health so you always know your security scanning is working.

Start monitoring Trivy Operator in under 5 minutes — register free at vigilmon.online.