tutorial

Monitoring Fulcio with Vigilmon: Keeping Your Sigstore Certificate Authority Healthy

How to monitor Fulcio — the Sigstore keyless signing CA — with Vigilmon, tracking certificate issuance endpoints, OIDC token validation, CT log connectivity, and transparency log health from outside your infrastructure.

Fulcio is the certificate authority at the heart of Sigstore's keyless code signing ecosystem. When a developer runs cosign sign or a CI pipeline signs a container image, Fulcio validates the OIDC identity token, issues a short-lived X.509 certificate, and records the issuance in a Certificate Transparency log. If Fulcio goes down — API crash, OIDC provider connectivity loss, CT log submission failure — all new code signing stops. Pipelines that enforce signature verification start rejecting unsigned artifacts, blocking deployments. Vigilmon gives you external monitoring that watches Fulcio from the client's perspective and alerts the moment certificate issuance is at risk.

What You'll Build

  • A Vigilmon HTTP monitor for Fulcio's health and certificate issuance endpoints
  • A heartbeat monitor to verify OIDC-to-certificate issuance end-to-end
  • A CT log connectivity probe
  • Alert channels routed to your security and platform teams

Prerequisites

  • A running Fulcio instance (self-hosted) or access to the Sigstore public instance
  • Network access to expose Fulcio's gRPC/HTTP endpoints with TLS
  • A valid OIDC provider (e.g. Google, GitHub Actions, Dex) configured in Fulcio
  • A free account at vigilmon.online

Why Monitoring Fulcio Matters

Fulcio sits on the critical path for every keyless signing operation. Its failure modes are particularly impactful for software supply chain security:

API unavailability blocks all new code signing. In environments that enforce signature verification at deploy time (via policy controllers, Kyverno, or OPA Gatekeeper), an unavailable Fulcio means no new deployments succeed. The failure cascades from a signing service outage into a deployment outage.

OIDC provider connectivity loss prevents Fulcio from validating identity tokens. Even if the Fulcio API is reachable, certificate issuance fails if it cannot call the OIDC provider's JWKS endpoint to verify the token signature. This is a silent dependency — Fulcio's /healthz may return 200 while all certificate issuance fails.

Certificate Transparency log submission failures cause Fulcio to fail issuance in strict mode. Fulcio submits every issued certificate to a CT log (typically Rekor or an RFC 6962-compliant log). If the CT log is unreachable or rejects submissions, certificate issuance stops.

Certificate clock drift causes OIDC token validation to fail. Fulcio validates the iat and exp claims in OIDC tokens; if the Fulcio server's clock drifts more than the OIDC provider's allowed skew, all certificate issuance fails with token validation errors.

External monitoring with Vigilmon detects all of these from outside the system, from the same network path a signing client uses.

Step 1: Monitor the Fulcio Health Endpoint

Fulcio exposes an HTTP health endpoint at /healthz:

# For self-hosted Fulcio
curl https://fulcio.internal.yourdomain.com/healthz

# For the Sigstore public instance
curl https://fulcio.sigstore.dev/healthz

In the Vigilmon dashboard:

  1. Go to Add Monitor → HTTP
  2. Configure:

| Field | Value | |---|---| | Name | fulcio /healthz | | URL | https://fulcio.internal.yourdomain.com/healthz | | Method | GET | | Expected status | 200 | | Check interval | 1 minute | | Timeout | 10 seconds | | Alert after | 2 consecutive failures |

Step 2: Monitor the Certificate Issuance Endpoint

The health endpoint doesn't verify that Fulcio can actually issue certificates. Add a monitor for the certificate issuance API route that checks reachability and TLS validity:

# Verify the signing certificate chain endpoint is reachable
curl -I https://fulcio.internal.yourdomain.com/api/v2/signingCert
# Expect 400 (missing request body) — proves the route is alive and TLS is valid

Add a second Vigilmon HTTP monitor:

| Field | Value | |---|---| | Name | fulcio /api/v2/signingCert reachable | | URL | https://fulcio.internal.yourdomain.com/api/v2/signingCert | | Method | POST | | Expected status | 400 | | Check interval | 2 minutes | | Timeout | 15 seconds | | Alert after | 2 consecutive failures |

A 400 Bad Request (missing body) is the expected response when Fulcio is healthy — it proves the route is routed and the TLS certificate is valid. A 502 or connection timeout means the API process is down.

Step 3: Monitor the Root Certificate Endpoint

Verify Fulcio's signing certificate chain is accessible — clients need this to verify issued certificates:

In Vigilmon, add a third HTTP monitor:

| Field | Value | |---|---| | Name | fulcio root certificate | | URL | https://fulcio.internal.yourdomain.com/api/v2/trustBundle | | Method | GET | | Expected status | 200 | | Response body contains | chains | | Check interval | 5 minutes | | Timeout | 10 seconds | | Alert after | 2 consecutive failures |

Step 4: Heartbeat Monitor for End-to-End OIDC Signing

The most critical probe validates the full signing path: OIDC token acquisition → Fulcio certificate issuance. This catches OIDC connectivity loss that pod-level health checks miss.

Create a CronJob that uses cosign to perform a full keyless signing operation against a test OCI artifact:

# fulcio-signing-probe.yaml
apiVersion: batch/v1
kind: CronJob
metadata:
  name: fulcio-signing-probe
  namespace: security-monitoring
spec:
  schedule: "*/15 * * * *"
  jobTemplate:
    spec:
      template:
        spec:
          restartPolicy: OnFailure
          serviceAccountName: fulcio-probe
          containers:
            - name: signing-probe
              image: gcr.io/projectsigstore/cosign:v2.2.0
              env:
                - name: FULCIO_URL
                  value: "https://fulcio.internal.yourdomain.com"
                - name: REKOR_URL
                  value: "https://rekor.internal.yourdomain.com"
                - name: OIDC_TOKEN_URL
                  value: "https://your-oidc-provider/token"
                - name: HEARTBEAT_URL
                  valueFrom:
                    secretKeyRef:
                      name: vigilmon-secrets
                      key: fulcio-signing-heartbeat-url
              command:
                - /bin/sh
                - -c
                - |
                  set -e

                  # Acquire OIDC token using Kubernetes service account token
                  TOKEN=$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)

                  # Attempt to get a certificate from Fulcio
                  # Uses cosign's built-in OIDC exchange
                  cosign sign \
                    --fulcio-url="$FULCIO_URL" \
                    --rekor-url="$REKOR_URL" \
                    --identity-token="$TOKEN" \
                    --yes \
                    ttl.sh/vigilmon-canary:5m 2>&1 | tee /tmp/cosign_output.txt

                  # Verify signing succeeded
                  if grep -q "error\|Error\|FAIL" /tmp/cosign_output.txt; then
                    echo "FAIL: Signing probe encountered errors"
                    cat /tmp/cosign_output.txt
                    exit 1
                  fi

                  wget -qO- "$HEARTBEAT_URL"
                  echo "Fulcio signing heartbeat sent"
          volumes:
            - name: service-account-token
              projected:
                sources:
                  - serviceAccountToken:
                      audience: sigstore
                      expirationSeconds: 3600
                      path: token

Create the heartbeat monitor in Vigilmon:

  1. Add Monitor → Heartbeat
  2. Name: fulcio end-to-end signing
  3. Expected interval: 15 minutes
  4. Grace period: 10 minutes

Store the secret:

kubectl create secret generic vigilmon-secrets \
  --from-literal=fulcio-signing-heartbeat-url='https://vigilmon.online/heartbeat/YOUR_TOKEN' \
  -n security-monitoring

Step 5: Monitor CT Log Connectivity

Fulcio requires a Certificate Transparency log for certificate submission. Monitor it separately so you can distinguish Fulcio API failures from CT log dependencies:

# fulcio-ctlog-probe.yaml
apiVersion: batch/v1
kind: CronJob
metadata:
  name: fulcio-ctlog-probe
  namespace: security-monitoring
spec:
  schedule: "*/10 * * * *"
  jobTemplate:
    spec:
      template:
        spec:
          restartPolicy: OnFailure
          containers:
            - name: ctlog-probe
              image: curlimages/curl:latest
              env:
                - name: CTLOG_URL
                  value: "https://ctfe.internal.yourdomain.com"
                - name: HEARTBEAT_URL
                  valueFrom:
                    secretKeyRef:
                      name: vigilmon-secrets
                      key: fulcio-ctlog-heartbeat-url
              command:
                - /bin/sh
                - -c
                - |
                  set -e
                  # Check CT log STH (Signed Tree Head) endpoint
                  STH=$(curl -sf "$CTLOG_URL/ct/v1/get-sth")
                  if [ -z "$STH" ]; then
                    echo "FAIL: CT log STH endpoint returned empty response"
                    exit 1
                  fi

                  # Verify tree_size is present (indicates active log)
                  echo "$STH" | grep -q "tree_size" || {
                    echo "FAIL: CT log STH missing tree_size"
                    exit 1
                  }

                  wget -qO- "$HEARTBEAT_URL"
                  echo "CT log heartbeat sent"

Step 6: Configure Alert Channels

Route Fulcio alerts to both security and platform teams:

  1. In Vigilmon: Alert Channels → Add Channel → Slack Webhook
  2. Add your #security-alerts and #platform-alerts webhooks
  3. Assign both channels to all Fulcio monitors

For the end-to-end signing probe (complete signing pipeline failure):

  1. Alert Channels → Add Channel → PagerDuty or email
  2. Assign it only to the fulcio end-to-end signing heartbeat
  3. This is an on-call escalation — signing failure blocks deployments

What You're Now Monitoring

| Component | Monitor Type | What It Detects | |---|---|---| | Fulcio API process | HTTP /healthz | Service crash or restart | | Issuance route | HTTP POST /api/v2/signingCert | Route routing or API process issues | | Trust bundle | HTTP GET /api/v2/trustBundle | Certificate chain unavailable | | End-to-end signing | Heartbeat (15 min) | OIDC connectivity, full signing pipeline | | CT log | Heartbeat (10 min) | CT log unreachable (blocks issuance) |

Conclusion

Fulcio is a foundational component of keyless supply chain security, but its failure is invisible until developers try to sign and fail, or verifying systems start rejecting unsigned artifacts and blocking deployments. External monitoring with Vigilmon gives your security team early warning from the same network path signing clients use — not just pod-level health signals.

Get started free at vigilmon.online. The trust bundle monitor and signing heartbeat together give you the two most important signals in under ten minutes: is Fulcio reachable, and can it actually issue certificates?

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles