tutorial

Monitoring Trivy in CI/CD Pipelines and Kubernetes

--- title: "Monitoring Trivy in CI/CD Pipelines and Kubernetes" description: "How to monitor Trivy scan job success rates, vulnerability report freshness, a...

title: "Monitoring Trivy in CI/CD Pipelines and Kubernetes" description: "How to monitor Trivy scan job success rates, vulnerability report freshness, and operator health in CI/CD pipelines and as a Kubernetes admission controller, with Vigilmon integration." date: 2026-07-01 tags: [trivy, security, kubernetes, ci-cd, monitoring, vigilmon]

Trivy is the de facto standard for container image and filesystem vulnerability scanning. It runs in CI/CD pipelines to gate image promotion, and as a Kubernetes Operator (Trivy Operator) to continuously audit running workloads. When Trivy scans silently fail or stall, vulnerable images slip through without anyone knowing. This guide covers how to monitor Trivy in both deployment models.

Two Deployment Models

CI/CD Pipeline Mode

Trivy runs as a step in your build pipeline — GitHub Actions, GitLab CI, Jenkins — scanning images before they are pushed or deployed. Failures here manifest as:

  • Scan jobs exiting non-zero (vulnerability threshold exceeded)
  • Scan jobs timing out (network issues reaching the vulnerability DB)
  • Outdated vulnerability DB (no DB update in 24h+, missing new CVEs)
  • Image not scanned at all (the pipeline step was skipped or bypassed)

Kubernetes Operator Mode (Trivy Operator)

Trivy Operator installs as a Kubernetes controller that automatically scans images running in the cluster and writes results as VulnerabilityReport and ConfigAuditReport custom resources. Failures here manifest as:

  • Operator pod crashes or OOM kills
  • Scan jobs stuck in Pending (no resources)
  • Reports not refreshed (stale data, giving false confidence)
  • Admission controller blocking legitimate deployments due to a false positive

Monitoring Trivy in CI/CD

Track Scan Job Exit Codes

In GitHub Actions, capture and surface scan results as annotations:

- name: Scan image with Trivy
  uses: aquasecurity/trivy-action@master
  with:
    image-ref: ${{ env.IMAGE }}
    format: sarif
    output: trivy-results.sarif
    exit-code: '1'
    severity: 'CRITICAL,HIGH'

- name: Upload SARIF report
  uses: github/codeql-action/upload-sarif@v3
  with:
    sarif_file: trivy-results.sarif
  if: always()

The if: always() ensures the SARIF upload runs even when the scan fails, giving you a record of what was found.

DB Freshness Check

Trivy downloads its vulnerability DB on first run and caches it. Stale caches miss newly published CVEs. Enforce freshness:

# Fail the scan if the DB is older than 24 hours
trivy image \
  --db-repository ghcr.io/aquasecurity/trivy-db \
  --skip-db-update=false \
  --exit-code 1 \
  --severity CRITICAL,HIGH \
  $IMAGE

In GitLab CI, track the last successful DB update time as a pipeline variable and alert when it exceeds 24 hours:

trivy-scan:
  script:
    - trivy image --format json --output trivy-report.json $IMAGE
    - |
      DB_AGE=$(trivy --version 2>&1 | grep "DB Schema" | awk '{print $NF}')
      echo "Trivy DB: $DB_AGE"
  artifacts:
    reports:
      container_scanning: trivy-report.json
    expire_in: 7 days

Monitoring Trivy Operator in Kubernetes

The Trivy Operator exposes Prometheus metrics at :8080/metrics on the operator pod.

Prometheus Scrape Config

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: trivy-operator
  namespace: trivy-system
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: trivy-operator
  endpoints:
    - port: metrics
      path: /metrics

Key Metrics

# Total vulnerabilities found by severity
trivy_image_vulnerabilities{severity="CRITICAL", namespace="production"}

# Scan jobs currently running
trivy_image_vulnerabilities_sum

Check report freshness by looking at the VulnerabilityReport resource timestamps:

# Show all vulnerability reports and their age
kubectl get vulnerabilityreports -A \
  -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name,AGE:.metadata.creationTimestamp'

# Find reports older than 24 hours (stale)
kubectl get vulnerabilityreports -A -o json | \
  python3 -c "
import json, sys
from datetime import datetime, timezone, timedelta
data = json.load(sys.stdin)
cutoff = datetime.now(timezone.utc) - timedelta(hours=24)
for item in data['items']:
    ts = datetime.fromisoformat(item['metadata']['creationTimestamp'].replace('Z','+00:00'))
    if ts < cutoff:
        print(item['metadata']['namespace'], item['metadata']['name'], ts)
"

Alerting Rules

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: trivy-operator-alerts
  namespace: trivy-system
spec:
  groups:
    - name: trivy
      rules:
        - alert: TrivyCriticalVulnerabilities
          expr: trivy_image_vulnerabilities{severity="CRITICAL"} > 0
          for: 5m
          labels:
            severity: high
          annotations:
            summary: "Critical vulnerabilities in {{ $labels.image_tag }} (namespace: {{ $labels.namespace }})"

        - alert: TrivyOperatorDown
          expr: up{job="trivy-operator"} == 0
          for: 5m
          labels:
            severity: critical
          annotations:
            summary: "Trivy Operator is not reachable — cluster is not being scanned"

        - alert: TrivyScanJobStuck
          expr: |
            kube_job_status_active{namespace="trivy-system"} > 0
            and
            (time() - kube_job_created{namespace="trivy-system"}) > 1800
          for: 5m
          labels:
            severity: warning
          annotations:
            summary: "Trivy scan job running for >30 minutes — possible resource starvation"

Debugging Trivy Operator Issues

# Check operator pod health
kubectl get pods -n trivy-system
kubectl describe pod -n trivy-system -l app.kubernetes.io/name=trivy-operator

# View scan job queue
kubectl get jobs -n trivy-system

# Check for resource quota issues blocking scan pods
kubectl describe resourcequota -n trivy-system

# Operator logs
kubectl logs -n trivy-system deploy/trivy-operator --tail=100

# List all vulnerability reports with critical findings
kubectl get vulnerabilityreports -A \
  -o jsonpath='{range .items[*]}{.metadata.namespace}/{.metadata.name}: C={.report.summary.criticalCount} H={.report.summary.highCount}{"\n"}{end}'

Admission Controller Mode

When running Trivy as an admission controller (blocking deployments with critical CVEs), monitor policy enforcement:

# Check admission webhook configuration
kubectl get validatingwebhookconfigurations | grep trivy

# Recent denied admissions show up in audit logs
# Filter audit log for trivy denials:
kubectl logs -n trivy-system deploy/trivy-operator | grep "admission" | grep "denied"

If the webhook is in Fail mode and the operator crashes, all new pod deployments will fail. Monitor this with an availability probe (see Vigilmon section below).

Vigilmon Integration

Heartbeat for CI/CD Scans

Register a Vigilmon heartbeat that your pipeline calls after each successful Trivy scan:

# At the end of your scan step (after exit-code check passes)
- name: Signal scan health to Vigilmon
  run: |
    curl -fsS "https://vigilmon.online/heartbeat/YOUR-HEARTBEAT-ID"
  if: success()

Set the heartbeat period to match your pipeline frequency — if you build every 2 hours, set grace period to 3 hours. If scans stop completing successfully, Vigilmon alerts before your team notices.

Monitor Trivy Operator Endpoint

Add an HTTPS monitor for the Trivy Operator metrics endpoint to catch operator restarts and OOM kills:

  1. Go to Monitors → Add Monitor → TCP/Port
  2. Set host to your trivy-operator service cluster IP (or use a NodePort/LoadBalancer for external access)
  3. Set port to 8080
  4. Set check interval to 1 minute

Vulnerability Dashboard Status Page

Create a Vigilmon status page with components for each scanning layer:

  • CI/CD Image Scan — green when heartbeat received within period
  • Cluster Vulnerability Scan — green when Trivy Operator metrics reachable
  • Admission Webhook — green when the webhook endpoint responds

This gives security and platform teams a single pane showing whether their scan coverage is complete.

Summary

| Monitoring Layer | Signal | Tool | |---|---|---| | CI/CD scan completion | Pipeline exit code + heartbeat | CI platform + Vigilmon | | Vulnerability DB freshness | DB timestamp in scan output | CI checks | | Operator availability | Prometheus up metric | Alertmanager | | Critical CVE detection | trivy_image_vulnerabilities metric | Prometheus | | Admission webhook health | Pod status + endpoint probe | Vigilmon TCP monitor |

The riskiest gap is a Trivy Operator crash in Fail-mode — it blocks all pod creation without anyone noticing the operator is down. The TrivyOperatorDown alert and a Vigilmon TCP probe together close that gap.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles