Monitoring Grafana Pyroscope with Vigilmon

Grafana Pyroscope is an open-source continuous profiling platform that collects, stores, and queries profiling data for application performance analysis. Built in Go and running on port 4040, it ingests profiling data from push agents and pull-mode scrapers, stores it in object storage or locally, and serves flamegraph queries to Grafana dashboards. When Pyroscope goes down, your performance data has gaps — and when a performance incident happens, you'll be flying blind. Vigilmon keeps every Pyroscope component under observation so your profiling pipeline never silently fails.

What You'll Set Up

• HTTP availability monitor for the Pyroscope web server
• Profiling data ingestion endpoint health check
• Storage backend accessibility monitor
• Query engine service health monitor
• Scrape job execution monitor (pull-mode profiling)
• Push endpoint response time monitor
• Flamegraph rendering service health check
• Retention and compaction job monitor
• Grafana datasource connectivity monitor

Prerequisites

• Grafana Pyroscope deployed (binary or Docker) accessible on port 4040
• Object storage configured (S3, GCS, or local disk) for profiling data
• Grafana with the Pyroscope datasource plugin installed (optional but recommended)
• A free Vigilmon account

Step 1: Monitor the Pyroscope Web Server

Pyroscope's primary HTTP interface serves the API, UI, and ingestion endpoint on port 4040. Start with a basic availability check:

1. Log in to vigilmon.online and click Add Monitor.
2. Set Type to HTTP / HTTPS.
3. Enter your Pyroscope URL: https://pyroscope.yourdomain.com/ready.
4. Set Check interval to 1 minute.
5. Set Expected HTTP status to 200.
6. Click Save.

The /ready endpoint indicates Pyroscope has finished initializing and is ready to accept profiling data:

curl https://pyroscope.yourdomain.com/ready
# ready

A non-200 response means Pyroscope is starting up, overloaded, or crashed — and profiling agents are failing to push data.

Step 2: Monitor the Profiling Data Ingestion Endpoint

Pyroscope's ingest endpoint (/ingest) receives profiling data from push agents (pyroscope-go, pyroscope-java, etc.). Verify it is accepting connections:

1. Add a new HTTP / HTTPS monitor.
2. URL: https://pyroscope.yourdomain.com/ingest?name=vigilmon.probe&sampleRate=100&format=jfr.
3. Set Expected HTTP status to 422 (Unprocessable Entity — a GET or empty POST is rejected with a validation error, confirming the endpoint is alive and routing).
4. Interval: 1 minute.

Alternatively, send a minimal valid profiling payload in a cron script and monitor the heartbeat:

#!/bin/bash
# Send a minimal profiling ping to Pyroscope
curl -s -o /dev/null -w "%{http_code}" \
  -X POST "https://pyroscope.yourdomain.com/ingest?name=vigilmon.probe&sampleRate=100" \
  -H "Content-Type: text/plain" \
  --data-binary "main;foo 1" \
  | grep -q "200" && \
  curl -s "https://vigilmon.online/heartbeat/YOUR_HEARTBEAT_ID"

1. Add a Cron Heartbeat monitor in Vigilmon with a 5 minute interval.
2. Run the script on a schedule (cron job or systemd timer).

If the ingest endpoint goes down, your profiling pipeline is silently losing data.

Step 3: Monitor Storage Backend Accessibility

Pyroscope depends on a storage backend — S3, GCS, Azure Blob, or local disk — to persist profiling data. A storage failure means data is lost even if the ingest endpoint stays up.

For S3/GCS/object storage

Monitor Pyroscope's component status endpoint, which reports storage health:

1. Add an HTTP / HTTPS monitor.
2. URL: https://pyroscope.yourdomain.com/ready.
3. Pyroscope returns non-ready status when storage is inaccessible.

For deeper visibility, add a dedicated object storage monitor. If Pyroscope is configured with S3, monitor the S3 bucket endpoint:

# Check S3 bucket accessibility
aws s3 ls s3://your-pyroscope-bucket --region us-east-1 > /dev/null 2>&1 && \
  curl -s "https://vigilmon.online/heartbeat/STORAGE_HEARTBEAT_ID"

For local disk storage

Monitor disk usage to catch full-disk conditions before Pyroscope fails to write:

USAGE=$(df /var/lib/pyroscope | awk 'NR==2{print $5}' | tr -d '%')
if [ "$USAGE" -lt 85 ]; then
  curl -s "https://vigilmon.online/heartbeat/DISK_HEARTBEAT_ID"
fi

Step 4: Monitor the Query Engine

Pyroscope's query engine serves flamegraph data to Grafana and the Pyroscope UI. Probe the query API directly:

1. Add an HTTP / HTTPS monitor.
2. URL: https://pyroscope.yourdomain.com/pyroscope/render?query=process_cpu:cpu:nanoseconds:cpu:nanoseconds%7B%7D&from=now-1h&until=now.
3. Expected status: 200.
4. Interval: 2 minutes.

This query issues a real render request. A 200 response confirms the query engine can read from storage and process profile data. A 500 or timeout means the query path is broken even if ingestion is working.

Step 5: Monitor Pull-Mode Scrape Jobs

In pull mode, Pyroscope scrapes profiling endpoints from target services on a schedule. Add a heartbeat to verify scrape cycles are executing:

# Script run after each Pyroscope scrape cycle completes
# Check scrape target status via Pyroscope API
SCRAPE_STATUS=$(curl -s "https://pyroscope.yourdomain.com/api/v1/targets" | \
  python3 -c "import sys,json; t=json.load(sys.stdin); \
  healthy=sum(1 for x in t.get('activeTargets',[]) if x.get('health')=='up'); \
  print('ok' if healthy > 0 else 'error')")

if [ "$SCRAPE_STATUS" = "ok" ]; then
  curl -s "https://vigilmon.online/heartbeat/SCRAPE_HEARTBEAT_ID"
fi

1. Add a Cron Heartbeat with interval matching your scrape cycle (e.g. 1 minute).
2. Schedule the script with cron or systemd.

Step 6: Monitor Push Endpoint Response Times

Pyroscope SDKs push profiling data continuously. Slow response times cause SDK timeout errors and dropped samples. Monitor response time:

1. Open the ingest endpoint monitor created in Step 2.
2. Enable Response time alerting.
3. Set an alert threshold: 500ms — Pyroscope ingest should respond in under 200ms under normal load.

If response times spike, check Pyroscope's resource utilization (CPU, memory) and object storage write latency.

Step 7: Monitor the Flamegraph Rendering Service

The flamegraph renderer is the user-facing query path. Add an HTTP monitor targeting the render API:

1. Add an HTTP / HTTPS monitor.
2. URL: https://pyroscope.yourdomain.com/pyroscope/v1/series?q=process_cpu:cpu:nanoseconds:cpu:nanoseconds%7B%7D.
3. Expected status: 200.
4. Interval: 5 minutes.

This confirms the series endpoint — which backs the Grafana datasource — is responsive.

Step 8: Monitor Retention and Compaction Jobs

Pyroscope runs background jobs to compact profiling data and enforce retention policies. If these jobs stall, storage grows unboundedly and query performance degrades.

Add a heartbeat check in a script that monitors Pyroscope logs or metrics for compaction activity:

# Check Pyroscope metrics for compaction activity (requires Prometheus metrics endpoint)
COMPACTION_COUNT=$(curl -s "https://pyroscope.yourdomain.com/metrics" | \
  grep 'pyroscope_compaction_runs_total' | awk '{print $2}')

# Compare against last known value stored in a temp file
LAST_COUNT=$(cat /tmp/pyroscope_compaction_last 2>/dev/null || echo "0")
if [ "$COMPACTION_COUNT" != "$LAST_COUNT" ]; then
  echo "$COMPACTION_COUNT" > /tmp/pyroscope_compaction_last
  curl -s "https://vigilmon.online/heartbeat/COMPACTION_HEARTBEAT_ID"
fi

Set the heartbeat interval to 60 minutes — compaction runs are not per-minute events.

Step 9: Monitor the Grafana Datasource Connection

If Grafana uses Pyroscope as a datasource, a Pyroscope API change or auth failure will break all Grafana profiling dashboards without any visible error in Pyroscope itself.

1. Add an HTTP / HTTPS monitor targeting the Pyroscope API health from Grafana's network path.
2. URL: https://pyroscope.yourdomain.com/pyroscope/v1/series?q=.
3. Expected status: 200.

Also monitor the Grafana datasource health check endpoint if exposed:

https://grafana.yourdomain.com/api/datasources/proxy/UID_HERE/ready

Step 10: Configure Alert Channels and Summary

In Vigilmon, go to Alert Channels and add Slack, email, or a webhook. Configure thresholds:

• Consecutive failures before alert: 2 for the web UI and query engine.
• Consecutive failures before alert: 1 for the ingest endpoint — any failure means profiling data is being lost.

| Monitor | Target | What It Catches | |---|---|---| | Web server | /ready | Pyroscope process down or unready | | Ingest endpoint | /ingest heartbeat | Profiling data ingestion broken | | Storage backend | S3/disk heartbeat | Storage write failure, data loss | | Query engine | /pyroscope/render | Flamegraph queries broken | | Scrape jobs | Targets API heartbeat | Pull-mode profiling stalled | | Response time | Ingest endpoint | SDK timeouts, dropped samples | | Flamegraph render | /pyroscope/v1/series | Datasource queries failing | | Compaction jobs | Metrics heartbeat | Storage bloat, query degradation | | Grafana datasource | Datasource health | Dashboard integration broken |

Pyroscope's value is only realized when its data pipeline is running continuously — gaps in profiling data mean gaps in your performance analysis, which you'll feel most acutely during the next performance incident. With Vigilmon monitoring every layer from ingest to query, you'll catch Pyroscope failures before they create blind spots in your observability stack.