Monitoring Grafana Alloy with Vigilmon: Component Health, Pipeline Throughput, Memory Usage & Receiver/Exporter Metrics

Grafana Alloy is the next-generation OpenTelemetry distribution and successor to Grafana Agent, built around a composable component model where pipelines are defined as graphs of interconnected components — receivers, processors, exporters, and connectors. When an Alloy component fails to start, the entire downstream pipeline segment stops collecting data silently. When memory pressure causes Alloy to drop spans, your distributed trace is incomplete without any external indication. When a remote write exporter can't reach its backend, metrics queue up until the buffer fills and data is permanently lost. Vigilmon gives you external visibility into Alloy's HTTP endpoints and cluster API, so you catch pipeline failures, memory saturation, and exporter outages before they create observability blind spots.

What You'll Build

• An HTTP monitor on Alloy's readiness endpoint to detect pipeline startup failures
• A liveness check that confirms Alloy is still processing data actively
• Component health monitoring via Alloy's UI API endpoint
• Memory and queue saturation alerting via Prometheus metric scraping
• SSL certificate monitoring for Alloy's OTLP/HTTP receiver endpoint
• An alerting runbook mapping Alloy failure modes to Vigilmon monitor states

Prerequisites

• Grafana Alloy 1.x installed (via Helm, systemd, or Docker)
• Alloy configured with at least one receiver and one exporter component
• Prometheus scraping Alloy's /metrics endpoint (port 12345 by default)
• Alloy's HTTP server enabled (set in alloy run --server.http.listen-addr=0.0.0.0:12345)
• A free account at vigilmon.online

Step 1: Understand Alloy's Health Surface

Grafana Alloy exposes an HTTP server on port 12345 by default. This server hosts:

• /ready — readiness probe; returns 200 Ready when all components have started successfully
• /metrics — Prometheus-format metrics including component health, queue sizes, memory usage
• /api/v0/web/components — live component graph with per-component health status

Check the health surface from the host where Alloy runs:

# Readiness check
curl http://localhost:12345/ready
# Returns: Ready

# List running components
curl http://localhost:12345/api/v0/web/components | jq '.[].health.state'
# Returns: "healthy" per component when running

# Check Alloy version and build info
curl http://localhost:12345/metrics | grep alloy_build_info

Key metrics that reflect pipeline health:

# Total spans received by OTLP receiver
otelcol_receiver_accepted_spans_total{receiver="otlp"}

# Spans dropped due to queue saturation
otelcol_processor_dropped_spans_total

# Queue size approaching capacity (watch this threshold carefully)
otelcol_exporter_queue_size{exporter="otlphttp/grafana"}

# Memory currently in use by Alloy process
process_resident_memory_bytes

Step 2: Monitor the Readiness Endpoint

The /ready endpoint confirms that all Alloy components have initialized. If a component fails to start — for example, because a remote backend is unreachable or a configuration syntax error slipped through — the readiness check returns a non-200 status.

1. Log in to Vigilmon → Add Monitor → HTTP.
2. URL: https://alloy.example.com/ready (your externally accessible Alloy HTTP server).
3. Check interval: 60 seconds.
4. Response timeout: 10 seconds.
5. Expected status: 200.
6. Keyword: Ready.
7. Label: Grafana Alloy readiness.
8. Click Save.

This monitor catches:

• Component initialization failures caused by unreachable backends
• Alloy process crashes and restart loops
• Kubernetes liveness probe failures leaving the pod in a broken state
• Configuration reload errors after a SIGHUP or alloy reload

Alert after: 1 consecutive failure. Alloy not ready means no telemetry data is flowing.

Step 3: Monitor the Metrics Endpoint for Component Health

Alloy's /metrics endpoint is where Prometheus scrapes pipeline statistics. Monitoring this endpoint also confirms the HTTP server itself is responsive:

1. Add Monitor → HTTP.
2. URL: https://alloy.example.com/metrics.
3. Check interval: 2 minutes.
4. Response timeout: 15 seconds.
5. Expected status: 200.
6. Keyword: alloy_component_controller_running_components.
7. Label: Alloy metrics endpoint.
8. Click Save.

The keyword alloy_component_controller_running_components confirms both that the metrics endpoint is serving and that Alloy's component controller is alive. If the keyword is absent, the component graph has stopped updating.

Step 4: Monitor Pipeline Throughput via Prometheus Queries

Alloy's Prometheus metrics expose per-component throughput. Set up Prometheus alerting rules to fire when throughput drops to zero or when the exporter queue fills:

# prometheus-alloy-alerts.yaml
groups:
  - name: alloy_pipeline
    rules:
      - alert: AlloyReceiverDroppedSpans
        expr: rate(otelcol_receiver_refused_spans_total[5m]) > 0
        for: 2m
        labels:
          severity: warning
        annotations:
          summary: "Alloy receiver is refusing spans"
          description: "OTLP receiver {{ $labels.receiver }} is refusing spans — check backend connectivity."

      - alert: AlloyExporterQueueFull
        expr: otelcol_exporter_queue_size / otelcol_exporter_queue_capacity > 0.9
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "Alloy exporter queue near capacity"
          description: "Exporter {{ $labels.exporter }} queue is {{ $value | humanizePercentage }} full — data loss imminent."

      - alert: AlloyHighMemoryUsage
        expr: process_resident_memory_bytes{job="alloy"} > 2e9
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "Alloy process memory high"
          description: "Alloy is using {{ $value | humanize1024 }}B of memory — consider adjusting batch processor limits."

Apply these rules:

kubectl apply -f prometheus-alloy-alerts.yaml
# Verify they're loaded
curl http://prometheus:9090/api/v1/rules | jq '.data.groups[].rules[].name'

For a Vigilmon-native proxy approach (if Prometheus is not externally accessible), monitor the Alloy metrics endpoint for the absence of specific metric names. A keyword monitor on otelcol_exporter_queue_size will fire if Alloy stops reporting that metric entirely — indicating the exporter component has crashed.

Step 5: Monitor the OTLP HTTP Receiver Endpoint

If Alloy is configured as an OTLP/HTTP receiver (common when it acts as a local telemetry gateway), the receiver endpoint itself should be reachable from application hosts. Monitor it from Vigilmon to catch network-level failures:

// Alloy config fragment — otelcol.receiver.otlp
otelcol.receiver.otlp "default" {
  http {
    endpoint = "0.0.0.0:4318"
  }
  output {
    traces  = [otelcol.processor.batch.default.input]
    metrics = [otelcol.processor.batch.default.input]
    logs    = [otelcol.processor.batch.default.input]
  }
}

1. Add Monitor → HTTP.
2. URL: https://alloy.example.com:4318/ (Alloy's OTLP/HTTP receiver).
3. Check interval: 2 minutes.
4. Response timeout: 10 seconds.
5. Expected status: 405 (Method Not Allowed — GET on the OTLP endpoint returns 405, confirming the port is open and TLS is working).
6. Label: Alloy OTLP/HTTP receiver.
7. Click Save.

Monitoring for HTTP 405 is the correct approach here: the OTLP/HTTP endpoint only accepts POST requests to /v1/traces, /v1/metrics, and /v1/logs. A 405 from a GET request confirms the server is live and TLS is valid without sending any telemetry data.

Step 6: SSL Certificate Monitoring

Alloy's OTLP/HTTP and gRPC endpoints typically terminate TLS. Certificate expiry on these endpoints breaks all instrumented applications trying to push telemetry through Alloy:

1. Add Monitor → SSL Certificate.
2. Domain: alloy.example.com.
3. Alert when expiry is within: 30 days.
4. Alert again: 14 days, 7 days, 3 days.
5. Click Save.

Add a second SSL monitor if the gRPC receiver (port 4317) uses a separate certificate.

Step 7: Configure Alerting for Pipeline Failures

In Vigilmon under Settings → Notifications, configure alert channels and map them to the correct response runbook:

| Monitor | Trigger | Immediate action | |---|---|---| | Alloy readiness /ready | Non-200 or keyword missing | Check component status with curl /api/v0/web/components; restart Alloy if all components are unhealthy | | Alloy metrics endpoint | Non-200 or keyword missing | HTTP server has crashed; telemetry data is not being reported to Prometheus | | OTLP/HTTP receiver | Non-405 or connection refused | Applications cannot push telemetry; check TLS cert and firewall rules on port 4318 | | SSL certificate | < 30 days to expiry | Renew certificate; all OTLP clients will begin failing TLS handshakes on expiry |

Monitor grouping: Group all Alloy monitors under an alloy-pipeline group so a single node failure generates one aggregated alert.

Common Alloy Failure Modes and What Vigilmon Catches

| Scenario | Vigilmon signal | |---|---| | Component fails to initialize at startup | Readiness monitor fires; /ready returns non-200 | | Alloy process OOM killed | Readiness monitor fires; process restart detected by keyword absence | | OTLP receiver TLS certificate expired | SSL monitor fires 30 days before; OTLP monitor fires on connection failure | | Remote write backend unreachable | Queue fills; Prometheus alert fires on otelcol_exporter_queue_size | | Alloy config reload breaks a component | Readiness monitor fires after SIGHUP if reload fails | | Metrics endpoint stops responding | Metrics endpoint monitor fires; Prometheus scrapes go stale | | Alloy cluster member leaves (cluster mode) | /cluster API degraded; check alloy_cluster_peers metric | | Batch processor memory limit exceeded | High memory monitor fires; spans may be dropped | | Exporter authentication token expires | Exporter drops data; check otelcol_exporter_send_failed_spans_total rising | | Kubernetes resource limits throttle CPU | Pipeline latency increases; check otelcol_processor_batch_timeout_trigger_send_total |

Grafana Alloy is the connective tissue between your applications and every observability backend — when it stops processing data, you lose traces, metrics, and logs across your entire infrastructure without any direct application error. Vigilmon gives you external endpoint checks on Alloy's readiness probe, its OTLP receiver, and its metrics API, so pipeline failures surface as monitor alerts rather than as mysterious gaps in your Grafana dashboards. When Alloy can't tell you something is wrong, Vigilmon can.

Start monitoring your Grafana Alloy pipelines in under 5 minutes — register free at vigilmon.online.