Monitoring ChartMuseum with Vigilmon: Health Endpoint, Helm Index, Chart API & Push Pipeline Heartbeat

ChartMuseum is the standard self-hosted Helm chart repository server used in Kubernetes continuous delivery pipelines. It hosts custom Helm charts as a standard index.yaml + .tgz repository that the Helm CLI understands natively, with storage backends ranging from local disk to S3, GCS, and Azure Blob Storage. When ChartMuseum goes down, helm install, helm upgrade, and GitOps controllers like Flux and ArgoCD can't fetch charts, stalling every deployment to every environment. Vigilmon gives you external visibility into ChartMuseum's health endpoint, the Helm index file, the REST API, SSL certificate expiry, and the chart push pipeline — catching failures before deployment pipelines break.

What You'll Build

• An HTTP monitor on ChartMuseum's health endpoint (/health)
• A Helm repository index availability check (/index.yaml)
• A chart REST API availability check (/api/charts)
• SSL certificate monitoring for ingress-terminated TLS in Kubernetes
• A chart push pipeline heartbeat to detect CI/CD push failures

Prerequisites

• ChartMuseum running (typically on port 8080, or behind a Kubernetes ingress)
• A domain or IP:port accessible externally
• A free account at vigilmon.online

Step 1: Check the ChartMuseum Health Endpoint

ChartMuseum exposes /health — an unauthenticated endpoint that returns JSON confirming both the Go HTTP server and the configured storage backend are operational:

curl http://charts.yourdomain.com/health
# {"healthy": true}

A 200 response with "healthy": true confirms ChartMuseum is running and can reach its storage backend (local disk, S3, GCS, or Azure Blob). A healthy: false or connection refused means either the Go process has crashed or the storage backend is unreachable.

Step 2: Create the Health Monitor in Vigilmon

1. Log in to Vigilmon → Add Monitor → HTTP.
2. URL: https://charts.yourdomain.com/health
3. Check interval: 60 seconds.
4. Response timeout: 10 seconds.
5. Expected status: 200.
6. Keyword: "healthy":true (confirms both server and storage are operational).
7. Label: ChartMuseum Health
8. Click Save.

This monitor catches:

• ChartMuseum Go process crashes
• Storage backend connectivity failures (S3 credentials expired, GCS bucket permissions changed, disk full)
• Kubernetes ingress routing failures to the ChartMuseum pod
• Container OOM kills or pod evictions

Step 3: Monitor the Helm Repository Index

The /index.yaml file is what the Helm CLI fetches when you run helm repo update. It lists every chart and version hosted in the repository with their download URLs and checksums. If this endpoint is broken, Helm reports Error: looks like ... is not a valid chart repository and every helm install or helm upgrade referencing this repo fails:

curl https://charts.yourdomain.com/index.yaml
# apiVersion: v1
# entries:
#   my-chart:
#   - apiVersion: v2
#     name: my-chart
#     version: 1.2.3
#     ...
# generated: "2024-01-15T10:00:00Z"

1. Add Monitor → HTTP.
2. URL: https://charts.yourdomain.com/index.yaml
3. Expected status: 200.
4. Keyword: apiVersion (confirms the Helm index YAML structure is intact).
5. Check interval: 120 seconds.
6. Label: ChartMuseum Helm Index

A healthy /health endpoint combined with a broken /index.yaml typically indicates a storage backend read failure — ChartMuseum is running but cannot read or regenerate the index from the chart store.

Step 4: Monitor the Chart REST API

ChartMuseum's /api/charts endpoint returns JSON with all charts organized by name, each with their available versions. This confirms that the REST API layer and the chart metadata store are both operational:

curl https://charts.yourdomain.com/api/charts
# {"my-chart":[{"name":"my-chart","version":"1.2.3","description":"...","urls":["charts/my-chart-1.2.3.tgz"],...}]}

1. Add Monitor → HTTP.
2. URL: https://charts.yourdomain.com/api/charts
3. Expected status: 200.
4. Keyword: name (present in any non-empty chart listing).
5. Check interval: 300 seconds.
6. Label: ChartMuseum Chart API

If the health check passes but the chart API returns an error or empty object when charts are expected, the issue is in the chart metadata indexing layer — charts may have been accidentally deleted or the index is corrupted.

Step 5: Monitor Your SSL Certificate

ChartMuseum in Kubernetes is typically exposed through an ingress controller (nginx ingress, Traefik, Istio) with TLS termination handled at the ingress layer using cert-manager and Let's Encrypt. When the ingress certificate expires, the Helm CLI fails with tls: certificate has expired for every chart repo operation:

1. Add Monitor → SSL Certificate.
2. Domain: charts.yourdomain.com
3. Alert when expiry is within: 30 days.
4. Alert again: 14 days, 7 days, 3 days.

In Kubernetes environments, cert-manager renewals can silently fail when a ClusterIssuer loses ACME account state after a cluster migration. The 30-day alert window gives you time to diagnose and fix cert-manager before Helm operations break.

Step 6: Chart Push Pipeline Heartbeat

In an active GitOps pipeline, new chart versions are pushed to ChartMuseum by CI/CD jobs after every successful build. If the push step in CI silently fails — due to authentication errors, a broken ChartMuseum plugin, network issues between CI and the chart server, or a misconfigured storage backend — charts stop being updated without any immediate observable failure. Your /health endpoint will still return {"healthy": true} because ChartMuseum itself is fine; only the pipeline is broken.

Set up a heartbeat that your CI/CD pipeline pings after every successful chart push:

# In your CI/CD pipeline (GitHub Actions, GitLab CI, Jenkins, etc.)
# Add this step after the helm cm-push step:

- name: Push Helm chart to ChartMuseum
  run: |
    helm cm-push ./charts/my-chart/ chartmuseum

- name: Ping Vigilmon heartbeat after successful push
  run: |
    curl -fsS -X POST https://vigilmon.online/api/heartbeat/YOUR-HEARTBEAT-ID

In Vigilmon:

1. Add Monitor → Heartbeat.
2. Expected interval: Set to your typical chart release cadence. For teams releasing daily: 25 hours. For multiple daily releases: 6 hours.
3. Grace period: 2 hours.
4. Label: ChartMuseum Push Pipeline

When the CI/CD push step stops succeeding — whether because of a ChartMuseum authentication failure, a Helm plugin version mismatch, or a network policy change blocking CI to the chart server — the heartbeat stops pinging and Vigilmon alerts you. You'll detect the broken pipeline before a deployment tries to helm upgrade to a version that was never pushed.

For environments with ChartMuseum metrics enabled, you can complement the heartbeat with a metrics-based check:

curl https://charts.yourdomain.com/metrics
# chart_museum_charts_served_total{...} 42
# chart_museum_requests_total{status="200",...} 158

The chart_museum_charts_served_total metric gives you an absolute chart count you can track for anomalous drops.

Common ChartMuseum Failure Modes and What Vigilmon Catches

| Scenario | Vigilmon monitor | |---|---| | ChartMuseum pod OOM-killed or crashed | Health endpoint returns connection refused; alert within 60 s | | S3/GCS backend credentials expired | Health returns healthy: false; index.yaml fails | | Kubernetes ingress misconfigured after upgrade | Health and index monitors fail while pod is running | | cert-manager renewal fails silently | SSL monitor alerts at 30 days | | /index.yaml not regenerating after chart pushes | Index keyword check catches stale or empty YAML | | CI chart push step broken (auth error, plugin mismatch) | Push pipeline heartbeat stops; alert fires | | Chart store accidentally wiped (empty repo) | Chart API returns {} while health passes | | Ingress TLS certificate expires | SSL monitor fires before Helm CLI gets TLS errors | | DNS change breaks chart server routing | All monitors fire simultaneously | | Multi-tenant namespace routing broken | Health passes; specific namespace index returns 404 |

ChartMuseum sits at the center of your Helm-based deployment workflow — every release and rollback depends on it. A Vigilmon setup covering the health endpoint, Helm index, chart API, SSL, and the CI/CD push pipeline gives you external visibility into every failure mode before a deployment stalls waiting for a chart that was never pushed.

Start monitoring ChartMuseum in under 5 minutes — register free at vigilmon.online.