tutorial

How to Monitor Grafana Agent: Scrape Health, Remote Write, and Silent Failure Detection with Vigilmon

A Grafana Agent that silently stops scraping or fails remote_write leaves your observability stack flying blind. Learn how to monitor scrape success rates, remote_write target health, WAL replay lag, and use Vigilmon to detect silent agent failures before your dashboards go stale.

Grafana Agent is the collection backbone for your metrics, logs, and traces — but it fails quietly. A misconfigured scrape job returns no data, a remote_write target starts rejecting samples, the WAL fills up, or the agent process dies without alerting anyone. Your Grafana dashboards look fine until someone notices the graphs stopped updating twelve hours ago.

Vigilmon adds an external watchdog layer: HTTP monitors for the Agent's built-in health endpoints, heartbeat monitors wired to Agent's own scrape cycles, and response time alerting to catch WAL replay lag before you lose metric data.

How Grafana Agent Fails

Grafana Agent failure modes are subtle because they often don't crash the process:

  • Scrape target down — the agent runs but silently records no samples for a target; Prometheus up{job="x"} metric drops to 0 but no alert fires unless you've set one up
  • remote_write rejection — the remote backend rejects samples (rate limit, auth failure, schema mismatch); the agent logs errors but continues running with an empty WAL drain
  • WAL replay lag — after a restart, the agent replays the Write-Ahead Log; during heavy replay, live scrapes are delayed and remote_write lags behind real-time
  • Component panic — a flow-mode component crashes and the agent falls back to degraded operation without restarting the process
  • Memory limit OOM — the agent is killed by the OOM killer and doesn't restart if no watchdog is configured

Step 1: Use Grafana Agent's Built-In Health Endpoints

Grafana Agent (both static mode and flow mode) exposes health endpoints you can probe directly:

# Agent liveness
GET http://localhost:12345/-/healthy
# Returns 200 "Agent is Healthy." when the process is up

# Agent readiness
GET http://localhost:12345/-/ready
# Returns 200 when all components have initialized; 503 during startup

# Prometheus-format metrics (for self-monitoring)
GET http://localhost:12345/metrics

For flow mode, the component graph is available:

GET http://localhost:12345/api/v0/web/components
# Returns JSON list of components and their health status

You can probe these endpoints directly from Vigilmon — no sidecar needed for basic liveness.

Step 2: Build a Richer Health Endpoint

For deep health checks (scrape success rates, remote_write lag), parse the Agent's own /metrics endpoint:

// grafana-agent-health.js
const express = require('express');
const axios = require('axios');

const app = express();
const AGENT_URL = process.env.GRAFANA_AGENT_URL || 'http://localhost:12345';
const MIN_SCRAPE_SUCCESS_RATE = parseFloat(process.env.MIN_SCRAPE_SUCCESS_RATE || '0.8');
const MAX_WAL_LAG_SECONDS = parseInt(process.env.MAX_WAL_LAG_SECONDS || '120');

async function parseAgentMetrics() {
  const resp = await axios.get(`${AGENT_URL}/metrics`, { timeout: 5000 });
  const lines = resp.data.split('\n').filter(l => !l.startsWith('#') && l.trim());

  const metrics = {};
  for (const line of lines) {
    const match = line.match(/^(\S+)\s+([\d.e+\-]+)/);
    if (match) {
      metrics[match[1]] = parseFloat(match[2]);
    }
  }
  return metrics;
}

app.get('/health/grafana-agent', async (req, res) => {
  try {
    // Check basic liveness first
    await axios.get(`${AGENT_URL}/-/healthy`, { timeout: 3000 });

    const metrics = await parseAgentMetrics();

    const issues = [];

    // Check scrape success rate
    const scrapeSuccess = metrics['prometheus_target_scrapes_sample_out_of_order_total'] !== undefined
      ? 1 - (metrics['prometheus_target_scrapes_sample_out_of_order_total'] /
             Math.max(metrics['prometheus_sd_discovered_targets'] || 1, 1))
      : null;

    // Simpler: check if any targets are up
    const targetsTotal = metrics['prometheus_sd_discovered_targets'] || 0;
    const scrapeFailures = metrics['prometheus_target_scrape_pool_exceeded_target_limit_total'] || 0;

    // remote_write queue lag
    const walSegmentSize = metrics['prometheus_remote_storage_wal_storage_total_appended_samples'] || 0;
    const walShipped = metrics['prometheus_remote_storage_samples_total'] || 0;
    const walLag = walSegmentSize - walShipped;

    if (walLag > MAX_WAL_LAG_SECONDS * 1000) {
      issues.push(`WAL lag: ${walLag} samples behind`);
    }

    // Check remote_write failures
    const rwFailures = metrics['prometheus_remote_storage_failed_samples_total'] || 0;
    const rwTotal = metrics['prometheus_remote_storage_samples_total'] || 1;
    const rwFailureRate = rwFailures / rwTotal;

    if (rwFailureRate > 0.05) {
      issues.push(`remote_write failure rate ${(rwFailureRate * 100).toFixed(1)}%`);
    }

    if (issues.length > 0) {
      return res.status(503).json({ status: 'degraded', issues, targets: targetsTotal });
    }

    return res.status(200).json({
      status: 'ok',
      targets: targetsTotal,
      wal_lag: walLag,
      rw_failure_rate: rwFailureRate.toFixed(4),
    });
  } catch (err) {
    return res.status(503).json({ status: 'down', error: err.message });
  }
});

app.get('/health/grafana-agent/components', async (req, res) => {
  // Flow mode: check component graph health
  try {
    const resp = await axios.get(`${AGENT_URL}/api/v0/web/components`, { timeout: 5000 });
    const unhealthy = resp.data.filter(c => c.health?.state === 'unhealthy');

    if (unhealthy.length > 0) {
      return res.status(503).json({
        status: 'degraded',
        unhealthy_components: unhealthy.map(c => c.name),
      });
    }

    return res.status(200).json({ status: 'ok', components: resp.data.length });
  } catch (err) {
    return res.status(503).json({ status: 'down', error: err.message });
  }
});

app.listen(3012, () => console.log('Grafana Agent health sidecar on :3012'));

Step 3: Configure Vigilmon HTTP Monitors

Monitor 1 — Agent Liveness (Direct Probe)

You can probe the Agent's built-in endpoint directly without a sidecar:

  1. Log in to vigilmon.online → Monitors → New Monitor → HTTP / HTTPS
  2. URL: http://your-agent-host:12345/-/healthy
  3. Interval: 1 minute
  4. Expected status: 200
  5. Body contains: Healthy
  6. Response time threshold: 3000ms
  7. Alert channel: PagerDuty (P1 — agent down means metrics go dark)

Monitor 2 — Deep Health Check

  1. New monitor → HTTP / HTTPS
  2. URL: https://your-agent-host.example.com/health/grafana-agent
  3. Interval: 2 minutes
  4. Expected: 200, body contains "status":"ok"
  5. Response time threshold: 6000ms
  6. Alert channel: Slack (P2 — degraded agent, not fully down)

Monitor 3 — Flow Mode Components (if using flow mode)

  • URL: https://your-agent-host.example.com/health/grafana-agent/components
  • Interval: 2 minutes
  • Alert channel: Slack

Step 4: Heartbeat Monitoring — Detect Silent Scrape Failures

The most dangerous Grafana Agent failure is a silent one: the process is up, the health endpoint returns 200, but scrape jobs are returning empty results because all targets are down or unreachable.

Use a Vigilmon heartbeat monitor driven by the Agent's own metric collection:

# In grafana-agent.yaml (static mode) — add a synthetic remote_write to Vigilmon
# Use a metrics generator component that hits your Vigilmon heartbeat URL

metrics:
  global:
    scrape_interval: 60s
  configs:
    - name: default
      scrape_configs:
        - job_name: vigilmon_heartbeat
          static_configs:
            - targets: ['localhost:12345']
          metrics_path: /-/healthy
          relabel_configs:
            - source_labels: [__address__]
              action: replace
              target_label: __address__
              replacement: 'localhost:12345'

# After scraping, use a remote_write that hits a webhook-compatible endpoint
# (or use a dedicated heartbeat sender component in flow mode)

For flow mode, use an HTTP component to ping Vigilmon:

// agent.flow — send a heartbeat to Vigilmon after each successful scrape cycle
prometheus.scrape "default" {
  targets = prometheus.operator.podmonitors.targets
  forward_to = [prometheus.remote_write.grafana_cloud.receiver, loki.write.default.receiver]
}

// Heartbeat sender — runs on an interval
prometheus.exporter.unix "self" {}

// Use a custom component to ping Vigilmon every 5 minutes
// (wrap in a prometheus.scrape targeting a heartbeat-sender sidecar)

Set up the heartbeat monitor in Vigilmon:

  1. Monitors → New Monitor → Heartbeat
  2. Name: grafana-agent-scrape-cycle
  3. Expected interval: 5 minutes
  4. Grace period: 10 minutes
  5. Alert channel: PagerDuty (a stuck scrape cycle means your dashboards are stale)

Step 5: Alert Routing

| Monitor | Trigger | Channel | Priority | |---|---|---|---| | HTTP: /-/healthy | Agent process down | PagerDuty | P1 | | HTTP: deep health | WAL lag or remote_write failures | Slack | P2 | | HTTP: flow components | Unhealthy component | Slack | P2 | | Heartbeat: scrape cycle | No scrape completion in 10 min | PagerDuty | P1 |

Summary

Grafana Agent failures are invisible until your dashboards stop updating. Layer Vigilmon's external probes on top of the Agent's built-in endpoints to catch every failure mode:

| Layer | What It Catches | |---|---| | HTTP: /-/healthy | Agent process crash or OOM | | HTTP: deep metrics check | WAL lag, remote_write rejection, high failure rate | | HTTP: flow components | Unhealthy flow-mode component | | Heartbeat: scrape cycle | Silent scrape failure, targets all down |

Monitor your observability agent at vigilmon.online — because your monitoring stack needs to be monitored too.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles