tutorial

Monitoring kube-score Static Analysis Pipelines with Vigilmon: CI/CD Gate Heartbeats, Score Regression Detection, TLS Certificate Alerts & Kubernetes Manifest Health Checks

How to use Vigilmon to monitor kube-score Kubernetes static analysis pipelines — CI/CD gate heartbeats, score regression detection scripts, and TLS certificate monitoring for kube-score-integrated release workflows.

kube-score is a static code analysis tool for Kubernetes objects that produces security and reliability recommendations by evaluating your YAML manifests against a set of built-in checks. Unlike policy engines that block deployments at admission time, kube-score is designed to run in CI/CD pipelines as a pre-commit or pre-merge gate — catching missing resource limits, absent network policies, containers running as root, and missing probes before YAML reaches the cluster. When the kube-score step in a CI pipeline breaks or is silently disabled, insecure or unreliable Kubernetes manifests ship to production unchecked. Vigilmon gives you heartbeat monitoring for the kube-score CI/CD gates that protect your clusters, and HTTP/SSL monitors for any web services that kube-score integrates with.

What You'll Build

  • Heartbeat monitors for kube-score CI/CD pipeline gates
  • Heartbeat monitors for scheduled score regression checks
  • HTTP monitors for any API or webhook endpoints in the scoring pipeline
  • SSL certificate monitors for those endpoints
  • Alerting runbook mapping kube-score pipeline failures to Vigilmon signals

Prerequisites

  • kube-score installed (brew install kube-score or download from GitHub releases)
  • Kubernetes YAML manifests or Helm charts to score
  • CI/CD pipeline (GitHub Actions, GitLab CI, or similar)
  • A free account at vigilmon.online

Step 1: Understand the kube-score Pipeline Health Surface

kube-score runs as a CLI tool in CI/CD pipelines and has no persistent HTTP service. Its health surface is entirely pipeline-based:

# Score a single Kubernetes manifest
kube-score score deployment.yaml
# Expected: List of checks with OK/WARNING/CRITICAL

# Score all manifests in a directory
kube-score score k8s/
# Expected: Combined scoring output, exit code 1 on CRITICAL findings

# Score Helm chart output
helm template my-release ./charts/myapp | kube-score score -
# Expected: Scored against templated manifests

# Score with specific checks enabled
kube-score score --enable-optional-test pod-networkpolicy deployment.yaml

# Output as JSON for programmatic processing
kube-score score --output-format json deployment.yaml | \
  python3 -c "
import sys, json
results = json.load(sys.stdin)
for r in results:
    for c in r.get('checks', []):
        if c.get('grade') == 1:  # CRITICAL
            print(f\"CRITICAL: {r['object']['name']} - {c['check']['name']}\")
"

# Check kube-score version
kube-score version
# Expected: kube-score 1.x.x ...

# Return codes
# 0 — all checks passed
# 1 — one or more checks failed at CRITICAL level
# 2 — kube-score encountered a parsing error

The key health signals to monitor:

  1. kube-score CI gate running and completing successfully on every pull request
  2. Scheduled score regression checks that catch manifest drift
  3. Any webhook or API endpoints in the scoring workflow (Slack notifications, ticketing integrations)
  4. TLS certificate health for those webhook endpoints

Step 2: Heartbeat Monitoring for kube-score CI Gates

The primary risk is the kube-score gate being disabled, broken, or skipped silently. Vigilmon heartbeats detect when the gate stops running.

Pull Request CI Gate

A GitHub Actions workflow that gates merges on a clean kube-score result:

# .github/workflows/kube-score.yml
name: kube-score manifest analysis

on:
  pull_request:
    paths:
      - 'k8s/**'
      - 'charts/**'

jobs:
  kube-score:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Install kube-score
        run: |
          curl -sL https://github.com/zegl/kube-score/releases/latest/download/kube-score_linux_amd64.tar.gz | \
            tar xzf - -C /usr/local/bin kube-score

      - name: Score Kubernetes manifests
        run: |
          # Find all YAML files in k8s/ directory
          YAMLS=$(find k8s/ -name '*.yaml' -o -name '*.yml' | tr '\n' ' ')
          if [ -z "${YAMLS}" ]; then
            echo "No Kubernetes YAML files found"
            exit 0
          fi
          # kube-score exits 1 on CRITICAL findings
          kube-score score ${YAMLS}

      - name: Score Helm chart
        run: |
          helm template app charts/myapp | kube-score score -

      - name: Ping Vigilmon heartbeat
        if: success()
        run: curl -s https://vigilmon.online/heartbeat/abc123

Set up the heartbeat monitor:

  1. Log in to Vigilmon → Add Monitor → Cron Heartbeat.
  2. Expected interval: 1500 minutes (expected on every PR with active development; 1-day grace period).
  3. Copy the heartbeat URL: https://vigilmon.online/heartbeat/abc123.
  4. Paste it as the last step in the workflow.
  5. Label: kube-score PR gate — k8s manifests.
  6. Click Save.

Pre-Merge Gate (Branch Protection)

When kube-score is a required status check for branch protection, also add a nightly confirmation heartbeat to catch the case where the gate passes vacuously (no YAML changed, no jobs triggered):

# .github/workflows/kube-score-nightly.yml
name: Nightly kube-score full scan

on:
  schedule:
    - cron: '0 3 * * *'

jobs:
  full-scan:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Install kube-score
        run: |
          curl -sL https://github.com/zegl/kube-score/releases/latest/download/kube-score_linux_amd64.tar.gz | \
            tar xzf - -C /usr/local/bin kube-score

      - name: Score all manifests
        run: |
          find k8s/ charts/ -name '*.yaml' -o -name '*.yml' | \
            xargs kube-score score --output-format json > /tmp/kube-score-results.json
          
          # Count critical findings
          CRITICAL=$(python3 -c "
import json
with open('/tmp/kube-score-results.json') as f:
    results = json.load(f)
count = sum(1 for r in results for c in r.get('checks', []) if c.get('grade') == 1)
print(count)
")
          echo "Critical findings: ${CRITICAL}"
          if [ "${CRITICAL}" -gt 0 ]; then
            echo "FAIL: ${CRITICAL} critical kube-score findings"
            exit 1
          fi

      - name: Ping Vigilmon heartbeat
        if: success()
        run: curl -s https://vigilmon.online/heartbeat/def456
  1. Add Monitor → Cron Heartbeat.
  2. Expected interval: 1500 minutes (nightly at 03:00 UTC with 60-minute grace period).
  3. Label: kube-score nightly full manifest scan.
  4. Click Save.

Step 3: Score Regression Detection Pipeline

A score regression occurs when a manifest that previously passed kube-score starts failing — for example, after a resource limit is removed or a security context is changed. A scheduled regression detector catches these silently:

#!/bin/bash
# detect-kube-score-regression.sh — run via cron or CI on a schedule

MANIFEST_DIR="${1:-k8s/}"
BASELINE_FILE="/var/lib/kube-score/baseline-results.json"
VIGILMON_HB="https://vigilmon.online/heartbeat/ghi789"
FAILED=0

# Run kube-score on all manifests
CURRENT_RESULTS=$(find "${MANIFEST_DIR}" -name '*.yaml' -o -name '*.yml' | \
  xargs kube-score score --output-format json 2>/dev/null)

if [ -z "${CURRENT_RESULTS}" ]; then
  echo "ERROR: No results returned from kube-score"
  exit 1
fi

# Count current critical findings
CURRENT_CRITICAL=$(echo "${CURRENT_RESULTS}" | python3 -c "
import sys, json
results = json.load(sys.stdin)
count = sum(1 for r in results for c in r.get('checks', []) if c.get('grade') == 1)
print(count)
")

# Compare with baseline if it exists
if [ -f "${BASELINE_FILE}" ]; then
  BASELINE_CRITICAL=$(python3 -c "
import json
with open('${BASELINE_FILE}') as f:
    results = json.load(f)
count = sum(1 for r in results for c in r.get('checks', []) if c.get('grade') == 1)
print(count)
")
  
  if [ "${CURRENT_CRITICAL}" -gt "${BASELINE_CRITICAL}" ]; then
    echo "REGRESSION: Critical findings increased from ${BASELINE_CRITICAL} to ${CURRENT_CRITICAL}"
    FAILED=1
  else
    echo "OK: ${CURRENT_CRITICAL} critical findings (baseline: ${BASELINE_CRITICAL})"
    # Update baseline when we're better or equal
    echo "${CURRENT_RESULTS}" > "${BASELINE_FILE}"
  fi
else
  echo "No baseline found — saving current results as baseline (${CURRENT_CRITICAL} critical)"
  mkdir -p "$(dirname "${BASELINE_FILE}")"
  echo "${CURRENT_RESULTS}" > "${BASELINE_FILE}"
fi

if [ "${FAILED}" -eq 0 ]; then
  curl -s "${VIGILMON_HB}"
fi

Schedule via cron:

# /etc/cron.d/kube-score-regression
0 */4 * * * ci-user /opt/scripts/detect-kube-score-regression.sh /srv/k8s-manifests/ >> /var/log/kube-score-regression.log 2>&1
  1. Add Monitor → Cron Heartbeat.
  2. Expected interval: 270 minutes (runs every 4 hours with a 30-minute grace period).
  3. Label: kube-score regression detector.
  4. Click Save.

Step 4: Helm Chart Score Pipeline

When kube-score is applied to Helm chart output, the scoring step depends on both Helm templating and kube-score being available:

# .github/workflows/helm-kube-score.yml
name: kube-score Helm chart analysis

on:
  pull_request:
    paths:
      - 'charts/**'

jobs:
  helm-score:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Install Helm
        run: |
          curl -fsSL https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash

      - name: Install kube-score
        run: |
          curl -sL https://github.com/zegl/kube-score/releases/latest/download/kube-score_linux_amd64.tar.gz | \
            tar xzf - -C /usr/local/bin kube-score

      - name: Score all Helm charts
        run: |
          FAILED=0
          for CHART in charts/*/; do
            CHART_NAME=$(basename "${CHART}")
            echo "Scoring Helm chart: ${CHART_NAME}"
            
            # Template with default values and score
            if ! helm template "test-${CHART_NAME}" "${CHART}" | kube-score score -; then
              echo "FAIL: kube-score found critical issues in ${CHART_NAME}"
              FAILED=1
            fi
          done
          exit "${FAILED}"

      - name: Ping Vigilmon heartbeat
        if: success()
        run: curl -s https://vigilmon.online/heartbeat/jkl012
  1. Add Monitor → Cron Heartbeat.
  2. Expected interval: 1500 minutes (runs on every Helm chart change, expected at least weekly).
  3. Label: kube-score Helm chart gate.
  4. Click Save.

Step 5: Monitor Notification Webhook Endpoints

If your kube-score pipeline posts results to a Slack webhook, a ticketing system API, or a custom scoring dashboard, those endpoints need uptime monitoring:

# Verify Slack webhook is accepting POST requests
curl -s -X POST https://hooks.slack.com/services/YOUR/WEBHOOK/ID \
  -H "Content-Type: application/json" \
  -d '{"text": "kube-score connectivity test"}' 2>&1
# Expected: "ok"

# Verify custom scoring dashboard API
curl -I https://kube-score-dashboard.example.com/api/results
# Expected: 200 or 401

# Verify ticketing system API (e.g., Jira)
curl -I https://jira.example.com/rest/api/2/issue/createmeta
# Expected: 200 or 401

For each notification endpoint:

  1. Add Monitor → HTTP.
  2. URL: https://kube-score-dashboard.example.com/api/results.
  3. Expected status codes: 200, 401.
  4. Label: kube-score results dashboard API.
  5. Click Save.

Add SSL certificate monitors for the same endpoints:

  1. Add Monitor → SSL Certificate.
  2. Domain: kube-score-dashboard.example.com.
  3. Alert when expiry is within: 30 days.
  4. Label: kube-score dashboard TLS certificate.
  5. Click Save.

Step 6: Configure Alerting

In Vigilmon under Settings → Notifications, configure alert channels and response runbooks:

| Monitor | Trigger | Immediate action | |---|---|---| | PR gate heartbeat | Missed ping | Check GitHub Actions for workflow disabled/broken; no active PRs touching YAML is OK — distinguish by checking PR activity | | Nightly scan heartbeat | Missed ping | Check scheduled workflow run; look for kube-score install failure or YAML parse errors | | Regression detector heartbeat | Missed ping | Check cron job logs; verify kube-score binary is on the PATH for the CI user | | Helm chart gate heartbeat | Missed ping | Check for Helm template errors before kube-score runs; look for missing chart dependencies | | Dashboard API | Non-200/401 | Results are not being stored; teams cannot review score trends | | Dashboard TLS certificate | < 30 days | Rotate certificate; check dashboard remains accessible from CI runners after renewal |

Common kube-score Pipeline Failure Modes and What Vigilmon Catches

| Scenario | Vigilmon signal | |---|---| | kube-score CI step disabled (workflow file edited to skip) | PR gate heartbeat goes silent; insecure manifests can merge | | kube-score binary not installed in CI image | CI pipeline fails at install step; PR gate heartbeat fires (step fails before ping) | | Scheduled nightly scan workflow disabled | Nightly scan heartbeat goes silent; accumulating manifest drift undetected | | Score regression from removed resource limit | Regression detector heartbeat fires; alert teams to restore the resource limit | | Helm templating breaks before kube-score runs | Helm chart gate heartbeat fires; chart changes cannot be validated | | Notification webhook URL changed or rotated | Webhook API monitor fires; team no longer receives kube-score Slack alerts | | Dashboard TLS certificate expired | SSL monitor fires; CI pipelines cannot POST results to dashboard | | kube-score version mismatch between local and CI | Inconsistent results; regression detector catches new failures introduced by version change | | YAML files moved out of scanned paths | PR gate passes vacuously; nightly scan heartbeat confirms actual scoring still runs |

kube-score makes Kubernetes manifest quality measurable and enforceable in CI/CD — but only when the pipeline gate is actually running on every change. Vigilmon's heartbeat monitors give you the external signal that confirms your kube-score gates are active, your regression detectors are running on schedule, and the notification endpoints that surface findings to your team are reachable. When a CI step is accidentally commented out or a scheduled scan workflow stops triggering, Vigilmon alerts you before insecure or unreliable Kubernetes manifests reach production unchecked.

Start monitoring your kube-score pipelines in under 5 minutes — register free at vigilmon.online.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles