Monitoring in-toto Supply Chain Integrity with Vigilmon

in-toto is a CNCF graduated framework for software supply chain security. It works by defining a signed layout that describes which steps (build, test, package, sign) must occur in your pipeline, and then collecting cryptographically signed link metadata (attestations) at each step to prove the supply chain executed as defined. When your signing keys rotate incorrectly, link metadata is missing, or the verification service itself goes down, your entire supply chain integrity guarantee silently fails — deployments may succeed but without the cryptographic proof that the artifacts are what you claim they are. Vigilmon gives you external monitoring for in-toto verification endpoint health, layout policy compliance, attestation signing status, and link metadata freshness so you catch supply chain integrity gaps before they become security incidents.

What You'll Set Up

• in-toto verification service health monitoring
• Layout policy compliance checks via cron heartbeats
• Attestation signing service availability
• Link metadata freshness monitoring
• Verification step health tracking

Prerequisites

• in-toto tools installed (pip install in-toto or binary distribution)
• A verification service or verification scripts accessible from monitored systems
• Access to your layout file and signing keys (for verification tests)
• A free Vigilmon account

Step 1: Monitor the Verification Endpoint

If you run in-toto verification as a service (common in Kubernetes-based admission controllers or CI/CD policy gates), that service endpoint is the primary health target. An unavailable verification service means deployments either bypass supply chain checks entirely or fail to deploy — both outcomes are bad. Set up an HTTP monitor first:

1. In Vigilmon, click Add Monitor → HTTP(s).
2. Set the URL to your in-toto verification service health endpoint (e.g., https://intoto-verifier.internal/health).
3. Set Check interval to 1 minute.
4. Set Expected HTTP status to 200.

If your in-toto integration runs as a Kubernetes admission webhook rather than a standalone service, monitor the webhook endpoint directly:

URL: https://your-webhook-service.namespace.svc.cluster.local/healthz
Check interval: 1 minute
Expected status: 200

Also add an SSL certificate expiry check for the webhook service — an expired TLS certificate on an admission webhook will block all deployments cluster-wide:

1. Add a second monitor → HTTP(s) with SSL certificate monitoring enabled.
2. Set SSL alert threshold to 30 days — admission webhook cert rotation is easy to miss.

Step 2: Monitor Layout Policy Compliance via Cron Heartbeat

The in-toto layout file describes the expected supply chain: which functionaries must perform which steps, which products are expected, and what rules apply. Verify that your current build artifacts satisfy the active layout on a schedule — not just at deployment time. This catches layout drift (e.g., a step that used to run is no longer producing link metadata) before it silently bypasses integrity checks:

1. In Vigilmon, click Add Monitor → Cron Heartbeat.
2. Set Expected ping interval to 30 minutes.
3. Copy the heartbeat URL (e.g., https://vigilmon.online/heartbeat/abc123).

Create the layout verification script:

#!/bin/bash
# /usr/local/bin/intoto-layout-check.sh

HEARTBEAT_URL="https://vigilmon.online/heartbeat/abc123"
LAYOUT_FILE="/etc/in-toto/root.layout"
LAYOUT_KEY="/etc/in-toto/root.pub"
LINK_DIR="/var/lib/in-toto/links"
TIMEOUT=30

# Verify layout exists and is parseable
if [ ! -f "$LAYOUT_FILE" ]; then
  echo "Layout file missing: $LAYOUT_FILE"
  exit 1
fi

# Run in-toto verification against the latest link directory
# (dry-run / last known good artifacts)
if timeout "$TIMEOUT" in-toto-verify \
  --layout "$LAYOUT_FILE" \
  --layout-keys "$LAYOUT_KEY" \
  --link-dir "$LINK_DIR" \
  --verbose 2>&1 | grep -q "Supply chain inspection passed"; then
  echo "Layout verification passed"
  curl -s "$HEARTBEAT_URL"
else
  VERIFY_OUTPUT=$(timeout "$TIMEOUT" in-toto-verify \
    --layout "$LAYOUT_FILE" \
    --layout-keys "$LAYOUT_KEY" \
    --link-dir "$LINK_DIR" 2>&1)
  echo "Layout verification failed: $VERIFY_OUTPUT"
  exit 1
fi

Install the cron job:

chmod +x /usr/local/bin/intoto-layout-check.sh
(crontab -l 2>/dev/null; echo "*/30 * * * * /usr/local/bin/intoto-layout-check.sh >> /var/log/intoto-layout.log 2>&1") | crontab -

If verification fails, Vigilmon alerts immediately — giving you time to investigate missing or invalid link metadata before the next deployment attempt.

Step 3: Monitor Attestation Signing Service Availability

Link metadata in in-toto is only as trustworthy as the signing process that produces it. If your signing service (a signing sidecar, a Cosign instance, a KMS-backed signer, or a Sigstore Fulcio integration) becomes unavailable, build steps will either fail to produce signed attestations or will silently skip signing — depending on pipeline configuration. Monitor the signing service as a separate health target:

#!/bin/bash
# /usr/local/bin/intoto-signer-check.sh

HEARTBEAT_URL="https://vigilmon.online/heartbeat/def456"
# Replace with your actual signing service endpoint
SIGNER_URL="https://signer.internal/health"
TIMEOUT=10

HTTP_STATUS=$(curl -s -o /dev/null -w "%{http_code}" \
  --max-time "$TIMEOUT" \
  "$SIGNER_URL")

if [ "$HTTP_STATUS" = "200" ]; then
  # Optionally: perform a test sign operation
  # in-toto-record will create a signed link for a null operation
  TEST_OUTPUT=$(echo "test" | \
    in-toto-sign --signer-uri "$SIGNER_URL" \
    --output /dev/null 2>&1 || true)

  echo "Signing service OK: HTTP $HTTP_STATUS"
  curl -s "$HEARTBEAT_URL"
else
  echo "Signing service unavailable: HTTP $HTTP_STATUS"
  exit 1
fi

Create a Vigilmon heartbeat with a 5 minute expected interval. Set urgency high — a signing service outage means your supply chain cannot produce new attestations, and CI pipelines will start producing unattested artifacts that will fail verification at deployment time.

For KMS-backed signers (AWS KMS, Google Cloud KMS, HashiCorp Vault):

# Check KMS key accessibility
aws kms describe-key --key-id "alias/intoto-signing-key" \
  --query 'KeyMetadata.KeyState' --output text | \
  grep -q "Enabled" && curl -s "$HEARTBEAT_URL"

Step 4: Monitor Link Metadata Freshness

in-toto link files represent attestations that specific pipeline steps ran at a specific point in time. If a pipeline step stops producing link metadata — because the step was removed, the CI job is failing silently, or the link upload is broken — old link files may be present from a previous run while the current build is unattested. Monitor link file freshness to catch stale attestations:

#!/bin/bash
# /usr/local/bin/intoto-link-freshness.sh

HEARTBEAT_URL="https://vigilmon.online/heartbeat/ghi789"
LINK_DIR="/var/lib/in-toto/links"
MAX_AGE_HOURS=2   # Alert if any required link is older than 2 hours
REQUIRED_STEPS=("build" "test" "package" "sign")  # Your pipeline steps

FAILED=0
CURRENT_TIME=$(date +%s)

for STEP in "${REQUIRED_STEPS[@]}"; do
  # Find the most recent link file for this step
  LATEST_LINK=$(find "$LINK_DIR" -name "${STEP}.*.link" \
    -newer "/tmp" 2>/dev/null | sort -t. -k2 | tail -1)

  if [ -z "$LATEST_LINK" ]; then
    echo "No link file found for step: $STEP"
    FAILED=1
    continue
  fi

  # Check modification time
  LINK_MTIME=$(stat -c%Y "$LATEST_LINK" 2>/dev/null || echo 0)
  AGE_SECONDS=$(( CURRENT_TIME - LINK_MTIME ))
  AGE_HOURS=$(( AGE_SECONDS / 3600 ))

  if [ "$AGE_HOURS" -ge "$MAX_AGE_HOURS" ]; then
    echo "Stale link for step '$STEP': ${AGE_HOURS}h old (max: ${MAX_AGE_HOURS}h)"
    FAILED=1
  else
    echo "Link OK for step '$STEP': ${AGE_HOURS}h old"
  fi
done

if [ "$FAILED" -eq 0 ]; then
  curl -s "$HEARTBEAT_URL"
else
  exit 1
fi

Run this check every hour with a Vigilmon heartbeat at 2 hour expected interval. Stale links are often caused by CI pipeline step failures that don't propagate errors correctly — the build "succeeds" but one step quietly stopped running.

Step 5: Monitor Verification Step Health

in-toto supports inspection steps — post-run checks that verify properties of the artifacts (file sizes, checksums, presence of specific files). These inspection steps run during in-toto-verify and can fail even when all link metadata is present. Monitor that inspections are passing separately from layout compliance:

#!/bin/bash
# /usr/local/bin/intoto-inspection-check.sh

HEARTBEAT_URL="https://vigilmon.online/heartbeat/jkl012"
LAYOUT_FILE="/etc/in-toto/root.layout"
LAYOUT_KEY="/etc/in-toto/root.pub"
LINK_DIR="/var/lib/in-toto/links"

# Run verification and capture inspection results
VERIFY_OUTPUT=$(in-toto-verify \
  --layout "$LAYOUT_FILE" \
  --layout-keys "$LAYOUT_KEY" \
  --link-dir "$LINK_DIR" \
  --verbose 2>&1)

EXIT_CODE=$?

if [ "$EXIT_CODE" -eq 0 ]; then
  # Count passing inspections
  PASS_COUNT=$(echo "$VERIFY_OUTPUT" | grep -c "Inspection.*passed" || echo 0)
  FAIL_COUNT=$(echo "$VERIFY_OUTPUT" | grep -c "Inspection.*failed" || echo 0)

  echo "Inspections: ${PASS_COUNT} passed, ${FAIL_COUNT} failed"

  if [ "$FAIL_COUNT" -eq 0 ]; then
    curl -s "$HEARTBEAT_URL"
  else
    echo "Inspection failures detected in: $VERIFY_OUTPUT"
    exit 1
  fi
else
  echo "Verification failed (exit $EXIT_CODE): $VERIFY_OUTPUT"
  exit 1
fi

Create a Vigilmon heartbeat with a 1 hour expected interval. Inspection failures often indicate artifact tampering or pipeline configuration drift — they should be treated with the same urgency as a security incident until the root cause is confirmed.

Step 6: Set Up Alert Channels

Configure Vigilmon to route in-toto alerts with security-appropriate urgency:

1. Go to Alert Channels in Vigilmon.
2. Add a Slack webhook to #security-alerts — supply chain integrity failures are security events.
3. Add PagerDuty for the verification service and signing service monitors — outages of these components block all deployments.
4. Add email notification to your security team for layout compliance and inspection failures.
5. Set Consecutive failures before alert to 1 for inspection and compliance monitors — a single failure may indicate an active supply chain attack and should not be dismissed as noise.

# Example: send a formatted security alert on verification failure
curl -X POST https://hooks.slack.com/services/YOUR/WEBHOOK/URL \
  -H 'Content-type: application/json' \
  --data '{
    "text": ":red_circle: *in-toto verification failed* — supply chain integrity check did not pass. Investigate immediately before next deployment.",
    "channel": "#security-alerts"
  }'

Summary

| Monitor | Target | What It Catches | |---|---|---| | HTTP check (verification service) | /health endpoint | Verification service unavailability | | HTTP check (SSL cert) | TLS certificate expiry | Webhook cert expiry blocking deployments | | Cron heartbeat (layout compliance) | in-toto-verify against latest links | Missing steps, invalid signatures, policy violations | | Cron heartbeat (signing service) | Signer health + test sign | Signing outage, KMS key unavailability | | Cron heartbeat (link freshness) | Link file modification times | Stale attestations, silent pipeline step failures | | Cron heartbeat (inspections) | in-toto-verify inspection results | Artifact tampering, inspection policy drift |

in-toto's supply chain guarantees are only as strong as the verification and attestation infrastructure supporting them. A broken signing service or stale link files undermine the entire framework — but they do so silently, because in-toto doesn't alert you when attestation production fails; it only alerts you at verification time, which may be hours later during a deployment attempt. Vigilmon closes this proactive monitoring gap so your security team knows about supply chain integrity issues in minutes, not at the next failed deployment.

Start monitoring for free at vigilmon.online.