tutorial

How to Monitor Parca Continuous Profiling Health and Query API with Vigilmon

Parca continuous profiling silently degrades when agents drop profiles or the storage backend slows down. Learn how to monitor Parca agent health, profile ingestion rate, storage backend status, query API latency, and label discovery accuracy with Vigilmon external endpoint checks.

Parca is a CNCF continuous profiling system that collects CPU, memory, and goroutine profiles from your services and stores them for long-term analysis. When Parca works, it surfaces the exact function calls burning CPU in production. When it degrades silently — agents fail to scrape, storage fills up, or the query API slows to a crawl — developers stop trusting it, profiles stop flowing, and you lose your most precise debugging tool right when you need it most.

Vigilmon adds external health monitoring for Parca through HTTP endpoint checks that validate agent connectivity, storage health, and query API availability. This tutorial covers agent health, profile ingestion rate, storage backend monitoring, query API latency tracking, and label discovery accuracy.

Why Parca Needs External Monitoring

Parca's architecture has several failure modes that won't surface as obvious errors:

  • Parca Agent scrape failure: Agents silently skip targets when processes restart or ports change. Profile ingestion appears to continue while coverage quietly shrinks.
  • Storage backend pressure: The embedded storage in Parca single-binary mode degrades under high cardinality or long retention windows without throwing explicit errors.
  • Query API latency drift: Slow queries don't fail — they timeout on the client side, making the UI appear broken while the server reports healthy.
  • Label discovery staleness: If Parca's target discovery misses new services, their profiles never appear in the UI and the gap is invisible.

External monitoring with Vigilmon catches these conditions before they erode developer trust in continuous profiling.

Step 1: Validate Parca Agent Health and Scrape Status

Parca Agent exposes Prometheus metrics including scrape success counts. Verify the agent's debug endpoint is reachable:

# Check Parca Agent health
curl -s http://parca-agent:7071/healthz
# Expected: 200 OK

# Check agent metrics for active scrapes
curl -s http://parca-agent:7071/metrics | grep parca_agent

Key metrics to watch:

# Successful profile scrapes per target
parca_agent_scrape_duration_seconds_count

# Failed scrape attempts
parca_agent_scrape_samples_scraped

# Active targets being profiled
parca_agent_targets_active_total

Build a health aggregation endpoint that evaluates scrape success rate:

# parca_health.py
from flask import Flask, jsonify
import requests
import os

app = Flask(__name__)

PARCA_URL = os.environ.get('PARCA_URL', 'http://parca:7070')
PARCA_AGENT_URL = os.environ.get('PARCA_AGENT_URL', 'http://parca-agent:7071')
MIN_ACTIVE_TARGETS = int(os.environ.get('MIN_ACTIVE_TARGETS', '1'))


@app.route('/health/parca/agent')
def parca_agent_health():
    try:
        resp = requests.get(f'{PARCA_AGENT_URL}/healthz', timeout=5)
        if resp.status_code != 200:
            return jsonify(status='down', code=resp.status_code), 503
    except Exception as e:
        return jsonify(status='down', error=str(e)), 503

    # Parse active target count from metrics
    try:
        metrics_resp = requests.get(f'{PARCA_AGENT_URL}/metrics', timeout=5)
        active_targets = 0
        for line in metrics_resp.text.splitlines():
            if line.startswith('parca_agent_targets_active_total'):
                try:
                    active_targets = int(float(line.split()[-1]))
                except ValueError:
                    pass

        if active_targets < MIN_ACTIVE_TARGETS:
            return jsonify(
                status='degraded',
                reason='no_active_targets',
                active_targets=active_targets,
            ), 503

        return jsonify(status='ok', active_targets=active_targets), 200
    except Exception:
        # Metrics unavailable but healthz passed — partial health
        return jsonify(status='ok', active_targets='unknown'), 200


if __name__ == '__main__':
    app.run(host='0.0.0.0', port=int(os.environ.get('PORT', '5011')))

Step 2: Monitor Profile Ingestion Rate

A healthy Parca deployment sees a steady stream of profiles arriving at the server. Use Parca's gRPC API or Prometheus metrics to validate ingestion is active:

@app.route('/health/parca/ingestion')
def parca_ingestion_health():
    try:
        # Parca server exposes Prometheus metrics at /metrics
        resp = requests.get(f'{PARCA_URL}/metrics', timeout=10)
        resp.raise_for_status()

        ingested_total = None
        for line in resp.text.splitlines():
            if 'parca_profilestore_profiles_ingested_total' in line and not line.startswith('#'):
                try:
                    ingested_total = float(line.split()[-1])
                except ValueError:
                    pass

        if ingested_total is None:
            return jsonify(
                status='degraded',
                reason='ingestion_metric_missing',
                note='parca may not be receiving profiles',
            ), 503

        if ingested_total == 0:
            return jsonify(
                status='degraded',
                reason='zero_profiles_ingested',
            ), 503

        return jsonify(
            status='ok',
            profiles_ingested_total=ingested_total,
        ), 200

    except Exception as e:
        return jsonify(status='down', error=str(e)), 503

Step 3: Monitor Storage Backend Health

Parca's embedded storage (FrostDB) can degrade under high-cardinality label sets or when the compaction cycle falls behind. Add a storage health check:

@app.route('/health/parca/storage')
def parca_storage_health():
    try:
        resp = requests.get(f'{PARCA_URL}/metrics', timeout=10)
        resp.raise_for_status()

        compaction_errors = 0
        db_size_bytes = None

        for line in resp.text.splitlines():
            if line.startswith('#'):
                continue
            if 'frostdb_compaction_errors_total' in line:
                try:
                    compaction_errors = int(float(line.split()[-1]))
                except ValueError:
                    pass
            if 'frostdb_db_size_bytes' in line:
                try:
                    db_size_bytes = int(float(line.split()[-1]))
                except ValueError:
                    pass

        storage_limit_bytes = int(os.environ.get('STORAGE_LIMIT_BYTES', str(10 * 1024**3)))  # 10 GB default
        fill_ratio = (db_size_bytes / storage_limit_bytes) if db_size_bytes else None

        if compaction_errors > 0:
            return jsonify(
                status='degraded',
                reason='compaction_errors_detected',
                compaction_errors=compaction_errors,
            ), 503

        if fill_ratio and fill_ratio >= 0.85:
            return jsonify(
                status='degraded',
                reason='storage_near_capacity',
                fill_ratio=round(fill_ratio, 3),
                db_size_bytes=db_size_bytes,
            ), 503

        return jsonify(
            status='ok',
            compaction_errors=compaction_errors,
            db_size_bytes=db_size_bytes,
            fill_ratio=round(fill_ratio, 3) if fill_ratio else None,
        ), 200

    except Exception as e:
        return jsonify(status='down', error=str(e)), 503

Step 4: Track Query API Latency

Parca's query API serves flame graph and profile comparison requests. Slow queries silently timeout in the UI. Add a latency probe that fires a real query and measures response time:

import time

@app.route('/health/parca/query-api')
def parca_query_api_health():
    # Use Parca's ProfileTypes endpoint as a lightweight API probe
    start = time.monotonic()
    try:
        resp = requests.get(
            f'{PARCA_URL}/api/v1/profiletypes',
            timeout=15,
        )
        elapsed_ms = (time.monotonic() - start) * 1000

        if resp.status_code != 200:
            return jsonify(
                status='degraded',
                reason='query_api_error',
                code=resp.status_code,
                elapsed_ms=round(elapsed_ms, 1),
            ), 503

        latency_warn_ms = int(os.environ.get('QUERY_LATENCY_WARN_MS', '3000'))
        if elapsed_ms > latency_warn_ms:
            return jsonify(
                status='degraded',
                reason='query_api_slow',
                elapsed_ms=round(elapsed_ms, 1),
                threshold_ms=latency_warn_ms,
            ), 503

        profile_types = resp.json().get('types', [])
        return jsonify(
            status='ok',
            elapsed_ms=round(elapsed_ms, 1),
            profile_type_count=len(profile_types),
        ), 200

    except requests.Timeout:
        elapsed_ms = (time.monotonic() - start) * 1000
        return jsonify(
            status='down',
            reason='query_api_timeout',
            elapsed_ms=round(elapsed_ms, 1),
        ), 503
    except Exception as e:
        return jsonify(status='down', error=str(e)), 503

Step 5: Configure Vigilmon HTTP Monitors for Parca

Monitor 1: Parca Server Availability

  1. Log in to vigilmon.online and go to Monitors → New Monitor
  2. Choose HTTP / HTTPS
  3. URL: https://parca.your-domain.com/api/v1/profiletypes
  4. Interval: 1 minute
  5. Expected: Status code 200
  6. Response timeout: 10000ms (query API may be slow under load)
  7. Alert: Slack + PagerDuty
  8. Save

Monitor 2: Parca Agent Health

  • URL: https://parca-health.your-domain.com/health/parca/agent
  • Expected: 200, body contains "status":"ok"
  • Interval: 2 minutes
  • Alert: Slack + PagerDuty (agent down means profiling coverage gap)

Monitor 3: Profile Ingestion Rate

  • URL: https://parca-health.your-domain.com/health/parca/ingestion
  • Expected: 200, body contains "status":"ok"
  • Interval: 5 minutes
  • Alert: Slack (zero ingestion indicates pipeline failure)

Monitor 4: Storage Backend Health

  • URL: https://parca-health.your-domain.com/health/parca/storage
  • Expected: 200, body contains "status":"ok"
  • Interval: 5 minutes
  • Alert: Slack (compaction errors or near-full storage are P2)

Monitor 5: Query API Latency

  • URL: https://parca-health.your-domain.com/health/parca/query-api
  • Expected: 200, body contains "status":"ok"
  • Interval: 5 minutes
  • Alert: Slack (slow queries degrade developer experience before outright failure)

Step 6: Heartbeat Monitor for Label Discovery

Validate that Parca is discovering new profiling targets continuously by pinging Vigilmon whenever a label query returns results:

#!/bin/bash
# parca_label_heartbeat.sh
LABEL_COUNT=$(curl -sf "http://parca:7070/api/v1/labels" 2>/dev/null | python3 -c "import sys,json; d=json.load(sys.stdin); print(len(d.get('labelNames', [])))" 2>/dev/null)
if [ "${LABEL_COUNT:-0}" -gt 0 ]; then
  curl -sf "$VIGILMON_HEARTBEAT_URL" > /dev/null
fi

*/5 * * * * /opt/parca/parca_label_heartbeat.sh

In Vigilmon:

  1. Monitors → New Monitor → Heartbeat
  2. Name: parca-label-discovery
  3. Expected interval: 10 minutes, Grace: 15 minutes
  4. Save and configure VIGILMON_HEARTBEAT_URL

Alert Routing Summary

| Monitor | Signal | Priority | |---|---|---| | Parca server API | Process liveness and query response | P1 | | Parca Agent health | Agent scrape and target coverage | P1 | | Profile ingestion rate | Active profile data flowing to server | P2 | | Storage backend health | Compaction errors and capacity fill | P2 | | Query API latency | Slow queries before timeout | P2 | | Label discovery heartbeat | Active target discovery running | P3 |

Summary

Parca's value is entirely dependent on continuous, complete profile collection — but its failure modes are gradual and silent. External monitoring with Vigilmon provides the early warning layer that keeps your continuous profiling investment working:

| What Vigilmon Monitors | Why It Matters | |---|---| | Server API availability | Catches full Parca outages immediately | | Agent health and targets | Detects scrape gaps before coverage degrades | | Profile ingestion metrics | Confirms profiles are actually reaching the server | | Storage health | Prevents silent data loss from compaction failures | | Query API latency | Flags slow queries before developers lose trust | | Label discovery | Validates new services are being auto-discovered |

Get started free at vigilmon.online — your first Parca monitor is running in under two minutes.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles