tutorial

Monitoring Clair Vulnerability Scanning with Vigilmon: API Health Checks, Database Connectivity, TLS Certificate Alerts & Scanner Pipeline Heartbeats

How to use Vigilmon to monitor Clair container image vulnerability scanning — REST API health checks, PostgreSQL database availability, TLS certificate expiry, and heartbeat monitoring for automated scanning pipelines.

Clair is an open-source static analysis tool for detecting vulnerabilities in container images, originally developed at CoreOS and widely adopted in enterprise container registries including Quay.io. Clair ingests CVE data from upstream vulnerability databases, indexes container image layers, and exposes an HTTP API that CI/CD pipelines and registries query to block vulnerable images before deployment. When Clair's database becomes unavailable, when its vulnerability updater job silently fails to fetch new CVE data, or when the REST API becomes unresponsive, security teams lose visibility into container vulnerabilities without any alert. Vigilmon gives you external visibility into Clair's HTTP API, its PostgreSQL backend, TLS certificate health, and the automated scanning pipelines that depend on it.

What You'll Build

  • HTTP monitors on Clair's health and metrics endpoints
  • TCP monitors on the Clair API and PostgreSQL database ports
  • SSL certificate monitors for Clair's HTTPS endpoints
  • Heartbeat monitors for automated image scanning pipelines
  • Alerting runbook mapping Clair failure modes to monitoring signals

Prerequisites

  • Clair deployed (standalone binary, Docker Compose, or Kubernetes)
  • PostgreSQL database backing Clair
  • A free account at vigilmon.online

Step 1: Understand the Clair Health Surface

A production Clair deployment exposes several endpoints that reflect its operational state:

# Clair health endpoint (HTTP API mode)
curl -s http://clair.example.com:6060/healthz
# Expected: 200 OK, body: "OK"

# Clair introspection endpoint (gRPC-Gateway)
curl -s http://clair.example.com:8089/api/v1/health
# Expected: 200 OK

# Metrics endpoint (Prometheus scrape target)
curl -s http://clair.example.com:6061/metrics | head -20
# Expected: Prometheus metric lines

# Check the updater status via the matcher API
curl -s http://clair.example.com:6060/matcher/api/v1/updateoperations
# Expected: JSON list of recent update operations

# Check PostgreSQL connectivity used by Clair
psql "postgres://clair:password@db.example.com:5432/clair" \
  -c "SELECT count(*) FROM uo_envelopes;" 2>&1
# Expected: row count for update operations stored

The key failure signals to monitor:

  1. Clair HTTP API availability (the primary scanning interface)
  2. PostgreSQL database connectivity (Clair stores all vulnerability data here)
  3. TLS certificate validity on Clair's HTTPS endpoint
  4. Updater pipeline heartbeats (CVE data freshness)
  5. Scan pipeline heartbeats (automated image analysis jobs)

Step 2: Monitor the Clair HTTP API

The Clair health endpoint is the primary signal that the service is running and able to process requests:

# Standalone Clair health check (default port 6060)
curl -v http://clair.example.com:6060/healthz
# Expected: HTTP 200, body "OK"

# Clair on Kubernetes via ClusterIP service
curl -v http://clair-service.clair-system.svc.cluster.local:6060/healthz

# Test the indexer API endpoint directly
curl -s -X POST http://clair.example.com:6060/indexer/api/v1/index_report \
  -H "Content-Type: application/json" \
  -d '{"manifest": {"hash": "sha256:test", "layers": []}}' 2>&1 | head -5
# Expected: 400 (bad request) or 200 — confirms API is processing requests

Create a Vigilmon monitor for the Clair health endpoint:

  1. Log in to Vigilmon → Add Monitor → HTTP.
  2. URL: http://clair.example.com:6060/healthz (or your Clair host).
  3. Check interval: 60 seconds.
  4. Response timeout: 10 seconds.
  5. Expected status codes: 200.
  6. Expected body contains: OK.
  7. Label: Clair API health.
  8. Click Save.

If Clair exposes metrics, add a second monitor for the metrics endpoint as a secondary signal:

  1. Add Monitor → HTTP.
  2. URL: http://clair.example.com:6061/metrics.
  3. Expected status codes: 200.
  4. Label: Clair metrics endpoint.
  5. Click Save.

Step 3: Monitor the Clair API TCP Port

A TCP monitor detects network-layer failures that the HTTP monitor alone can miss (load balancer routing issues, Kubernetes service misconfiguration):

# Test TCP connectivity to Clair API
nc -zv clair.example.com 6060
# Expected: Connection succeeded

# Test from inside the Kubernetes cluster
kubectl exec -n clair-system \
  $(kubectl get pod -n clair-system -l app=clair -o name | head -1) \
  -- nc -zv clair-service 6060
  1. Add Monitor → TCP.
  2. Host: clair.example.com.
  3. Port: 6060.
  4. Check interval: 60 seconds.
  5. Label: Clair API TCP (6060).
  6. Click Save.

If Clair's gRPC introspection API runs on a separate port (default 8089):

  1. Add Monitor → TCP.
  2. Host: clair.example.com.
  3. Port: 8089.
  4. Label: Clair introspection TCP (8089).
  5. Click Save.

Step 4: Monitor the PostgreSQL Database Port

Clair stores all indexed image data, vulnerability records, and update operation history in PostgreSQL. A database connectivity failure stops all indexing and matching:

# Test PostgreSQL port reachability from the Clair host
nc -zv db.example.com 5432
# Expected: Connection succeeded

# Validate the Clair database exists and is accessible
psql "postgres://clair:password@db.example.com:5432/clair" \
  -c "\dt" 2>&1 | grep "update_operation\|indexer_manifest" | head -5
# Expected: table names from Clair's schema

# Check active Clair connections to the database
psql "postgres://postgres:adminpass@db.example.com:5432/postgres" \
  -c "SELECT count(*) FROM pg_stat_activity WHERE datname='clair';"
# Expected: count > 0 while Clair is running
  1. Add Monitor → TCP.
  2. Host: db.example.com (your PostgreSQL host).
  3. Port: 5432.
  4. Check interval: 60 seconds.
  5. Label: Clair PostgreSQL DB TCP (5432).
  6. Click Save.

A TCP failure on port 5432 while the Clair API HTTP monitor returns a non-200 response confirms the failure is at the database layer rather than the application layer.

Step 5: Monitor TLS Certificates

If Clair is exposed behind a TLS-terminating proxy (Nginx, Traefik, or a Kubernetes Ingress), an expired certificate breaks all client connections — including CI/CD pipelines that scan images before deployment:

# Check TLS certificate on Clair's HTTPS endpoint
echo | openssl s_client -connect clair.example.com:443 2>/dev/null | \
  openssl x509 -noout -dates
# notAfter=Sep 30 00:00:00 2026 GMT

# Check with curl verbose TLS output
curl -v https://clair.example.com/healthz 2>&1 | grep -E "expire|SSL|TLS|certificate"

# List all Clair TLS certificates in Kubernetes
kubectl get certificate -n clair-system
# Expected: certificate resources with READY=True
  1. Add Monitor → SSL Certificate.
  2. Domain: clair.example.com.
  3. Alert when expiry is within: 30 days.
  4. Alert again: 14 days, 7 days, 3 days, 1 day.
  5. Click Save.

Add SSL monitors for any other domains used to reach Clair (e.g., quay.example.com if running alongside Quay).

Step 6: Heartbeat Monitoring for Clair Pipelines

Clair's value depends on two automated processes running continuously: the updater that fetches new CVE data from upstream sources, and the scanning pipeline that indexes new container images. Neither has an externally queryable health signal — Vigilmon heartbeats fill this gap.

CVE Database Updater Pipeline

Clair's updater fetches vulnerability data from NVD, Red Hat, Debian, Ubuntu, and Alpine security databases on a regular schedule. If it stops running silently, scans continue against stale CVE data with no alert:

#!/bin/bash
# check-clair-updater.sh — run via cron every 6 hours

CLAIR_API="http://clair.example.com:6060"
VIGILMON_HB="https://vigilmon.online/heartbeat/abc123"

# Query the latest update operation timestamp
LATEST_UPDATE=$(curl -sf "${CLAIR_API}/matcher/api/v1/updateoperations" | \
  python3 -c "
import sys, json
ops = json.load(sys.stdin)
if ops and len(ops) > 0:
    # Find the most recent update operation date
    dates = [op.get('date', '') for op in ops.values() if isinstance(op, dict)]
    print(max(dates) if dates else '')
" 2>/dev/null)

if [ -z "${LATEST_UPDATE}" ]; then
  echo "ERROR: Could not retrieve update operations from Clair"
  exit 1
fi

echo "Latest Clair CVE update: ${LATEST_UPDATE}"
# Signal that the updater check ran successfully
curl -s "${VIGILMON_HB}"

Set up the heartbeat monitor:

  1. Add Monitor → Cron Heartbeat.
  2. Expected interval: 420 minutes (Clair default updater interval is 6 hours, with a 60-minute grace period).
  3. Copy the heartbeat URL: https://vigilmon.online/heartbeat/abc123.
  4. Schedule the check script: 0 */6 * * * /opt/scripts/check-clair-updater.sh.
  5. Label: Clair CVE updater pipeline.
  6. Click Save.

Image Scanning Pipeline (CI/CD Integration)

A GitHub Actions workflow that scans container images with Clair before pushing to the production registry:

# .github/workflows/image-scan.yml
name: Scan image with Clair

on:
  push:
    branches: [main]

jobs:
  scan:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Build image
        run: docker build -t myapp:${{ github.sha }} .

      - name: Submit image to Clair for indexing
        run: |
          # Push to staging registry so Clair can index it
          docker tag myapp:${{ github.sha }} registry.example.com/staging/myapp:${{ github.sha }}
          docker push registry.example.com/staging/myapp:${{ github.sha }}

          # Trigger Clair index via the API
          MANIFEST=$(docker inspect --format='{{index .RepoDigests 0}}' \
            registry.example.com/staging/myapp:${{ github.sha }})
          curl -sf -X POST http://clair.example.com:6060/indexer/api/v1/index_report \
            -H "Content-Type: application/json" \
            -d "{\"manifest\": {\"hash\": \"${MANIFEST}\", \"layers\": []}}"

      - name: Retrieve and check vulnerability report
        run: |
          DIGEST=$(docker inspect --format='{{index .RepoDigests 0}}' \
            registry.example.com/staging/myapp:${{ github.sha }})
          REPORT=$(curl -sf \
            "http://clair.example.com:6060/matcher/api/v1/vulnerability_report/${DIGEST}")
          CRITICAL=$(echo "${REPORT}" | python3 -c \
            "import sys, json; d=json.load(sys.stdin); print(sum(1 for v in d.get('vulnerabilities', {}).values() if v.get('normalized_severity') == 'Critical'))")
          echo "Critical vulnerabilities: ${CRITICAL}"
          if [ "${CRITICAL}" -gt 0 ]; then
            echo "FAIL: ${CRITICAL} critical CVEs found"
            exit 1
          fi

      - name: Ping Vigilmon heartbeat
        if: success()
        run: curl -s https://vigilmon.online/heartbeat/def456
  1. Add Monitor → Cron Heartbeat.
  2. Expected interval: 1500 minutes (expected daily scans with grace period).
  3. Label: Clair CI/CD image scan pipeline.
  4. Click Save.

Scheduled Retrospective Scan Job

A cron job that re-scans all images in the registry when new CVE data arrives:

#!/bin/bash
# retrospective-scan.sh — triggered after Clair updater completes

REGISTRY="registry.example.com"
CLAIR_API="http://clair.example.com:6060"
VIGILMON_HB="https://vigilmon.online/heartbeat/ghi789"
FAILED=0

# Enumerate all images in the registry (using registry API)
IMAGES=$(curl -sf "https://${REGISTRY}/v2/_catalog" | \
  python3 -c "import sys, json; [print(r) for r in json.load(sys.stdin)['repositories']]")

for IMAGE in ${IMAGES}; do
  DIGEST=$(crane digest "${REGISTRY}/${IMAGE}:latest" 2>/dev/null)
  if [ -z "${DIGEST}" ]; then
    echo "SKIP: Cannot get digest for ${REGISTRY}/${IMAGE}:latest"
    continue
  fi

  REPORT=$(curl -sf \
    "${CLAIR_API}/matcher/api/v1/vulnerability_report/${DIGEST}" 2>/dev/null)
  if [ $? -ne 0 ]; then
    echo "FAIL: Could not retrieve report for ${IMAGE}"
    FAILED=1
  else
    CRITICAL=$(echo "${REPORT}" | python3 -c \
      "import sys, json; d=json.load(sys.stdin); print(sum(1 for v in d.get('vulnerabilities', {}).values() if v.get('normalized_severity') == 'Critical'))" 2>/dev/null)
    echo "OK: ${IMAGE} — ${CRITICAL} critical CVEs"
  fi
done

if [ "${FAILED}" -eq 0 ]; then
  curl -s "${VIGILMON_HB}"
fi
  1. Add Monitor → Cron Heartbeat.
  2. Expected interval: 720 minutes (expected after every 6-hour updater cycle, with grace period).
  3. Label: Clair retrospective scan job.
  4. Click Save.

Step 7: Configure Alerting

In Vigilmon under Settings → Notifications, configure alert channels and response runbooks:

| Monitor | Trigger | Immediate action | |---|---|---| | Clair API health | Non-200 or timeout | Check Clair process; run curl http://clair:6060/healthz from inside the cluster; check Clair logs for DB errors | | Clair metrics endpoint | Non-200 | Secondary signal; confirm with API health monitor; check Prometheus scrape config | | Clair API TCP | Connection refused | Check Kubernetes service and pod status; kubectl get pods -n clair-system | | PostgreSQL TCP | Connection refused | Database connectivity failure; all Clair indexing and matching will fail; check DB host | | TLS certificate | < 30 days | Rotate certificate on reverse proxy; verify with openssl s_client after renewal | | CVE updater heartbeat | Missed ping | Check Clair updater logs; verify network access to NVD, Red Hat, and Debian advisory feeds | | CI/CD scan heartbeat | Missed ping | Check GitHub Actions logs; verify Clair API is reachable from the CI runner | | Retrospective scan heartbeat | Missed ping | Check cron job logs; verify Clair is accepting index requests and returning reports |

Common Clair Failure Modes and What Vigilmon Catches

| Scenario | Vigilmon signal | |---|---| | Clair API process crashed | HTTP health monitor fires; all image scans blocked | | PostgreSQL database down | TCP monitor on port 5432 fires; Clair logs DB connection errors; API returns 503 | | Clair network namespace issue in Kubernetes | TCP monitor fires; HTTP monitor fires; kubectl describe pod shows network policy block | | TLS certificate expired on Clair reverse proxy | SSL monitor fires at 30-day threshold; CI pipelines get TLS errors on scan API calls | | CVE updater stopped fetching new data | Updater heartbeat goes silent; scans continue against stale CVE database | | NVD or upstream feed outage blocks updater | Updater heartbeat fires; check Clair logs for upstream HTTP errors | | CI scan pipeline misconfigured | CI scan heartbeat goes silent; images ship without vulnerability gating | | Clair indexer queue backlogged | API returns 200 but reports are delayed; heartbeat latency increases | | Database disk full stops writes | PostgreSQL available on TCP but Clair logs write errors; health endpoint may return 500 | | Clair update to new major version breaks API | HTTP monitor fires after upgrade; revert or fix the deployment |

Clair makes container vulnerability scanning possible — but only when its API, database, and updater pipeline are all operating correctly. Vigilmon gives you the external visibility layer that alerts you when Clair's own health endpoint is unreachable, when the PostgreSQL database drops off the network, or when the updater pipeline silently stops refreshing CVE data. When a CVE database updater fails silently and images continue scanning against stale data, Vigilmon's heartbeat monitor is the only thing standing between your team and a missed critical vulnerability.

Start monitoring your Clair deployment in under 5 minutes — register free at vigilmon.online.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles