How to Monitor Vector Aggregator Health with Vigilmon

Vector is Datadog's high-performance observability data pipeline — a single binary that collects, transforms, and routes logs, metrics, and traces at massive scale. When Vector goes down or falls behind, your observability stack goes blind: metrics stop flowing to your TSDB, logs vanish before reaching storage, and you have no idea your pipeline has stalled until users report missing data minutes or hours later.

Vigilmon adds an external layer of health monitoring around Vector using its built-in Prometheus metrics endpoint and an optional HTTP health sidecar. This tutorial covers source/sink health, component throughput, buffer utilization, error rates per transform, and how to tie everything into Vigilmon alerting.

Why Vector Needs External Monitoring

Vector exposes rich internal telemetry via its /metrics Prometheus endpoint — but only if something is scraping that endpoint and alerting on anomalies. External monitoring with Vigilmon gives you:

• Process-level liveness checks that fire if Vector crashes or freezes
• Buffer overflow alerts before data starts being dropped at sinks
• Source ingestion rate monitoring to detect upstream stalls (e.g., a syslog source stops receiving events)
• Sink delivery error rate alerting when a downstream target (Elasticsearch, S3, Datadog) starts rejecting events
• Multi-region probe consensus to eliminate false positives from transient network blips

These checks complement — but don't replace — your Prometheus/Grafana dashboards. Vigilmon adds the external "is it alive at all?" layer.

Step 1: Enable Vector's API and Metrics Endpoint

Vector ships with a built-in API server and Prometheus-compatible /metrics endpoint. Add both to your vector.yaml:

# vector.yaml

api:
  enabled: true
  address: "0.0.0.0:8686"   # Vector GraphQL API + health endpoint

# Prometheus exporter for scraping
[sources.internal_metrics]
  type = "internal_metrics"

[sinks.prometheus_exporter]
  type = "prometheus_exporter"
  inputs = ["internal_metrics"]
  address = "0.0.0.0:9598"   # Prometheus scrape endpoint

With these enabled, Vector exposes:

• http://localhost:8686/health — JSON liveness endpoint (returns {"ok":true} when healthy)
• http://localhost:9598/metrics — full Prometheus metrics for all components

Verify both are reachable before proceeding:

curl -s http://localhost:8686/health
# {"ok":true}

curl -s http://localhost:9598/metrics | head -20

Step 2: Build a Vector Health Aggregation Endpoint

Vigilmon probes HTTP endpoints — it can't natively interpret Prometheus metric text. Build a lightweight health sidecar that reads Vector's metrics, applies thresholds, and returns a simple 200/503:

// vector-health.js
const express = require('express');
const axios = require('axios');

const app = express();
const VECTOR_METRICS_URL = process.env.VECTOR_METRICS_URL || 'http://localhost:9598/metrics';
const VECTOR_API_URL = process.env.VECTOR_API_URL || 'http://localhost:8686/health';

// Thresholds (tune to your pipeline)
const BUFFER_FILL_THRESHOLD = 0.85;     // 85% full triggers degraded
const ERROR_RATE_THRESHOLD = 0.01;      // 1% error rate triggers alert
const COMPONENT_ERRORS_MAX = 10;        // absolute error count per check

function parseMetric(text, name) {
  const lines = text.split('\n');
  const values = [];
  for (const line of lines) {
    if (line.startsWith(name) && !line.startsWith('#')) {
      const parts = line.split(' ');
      const val = parseFloat(parts[parts.length - 1]);
      if (!isNaN(val)) values.push(val);
    }
  }
  return values;
}

app.get('/health/vector', async (req, res) => {
  try {
    // Check Vector process liveness
    const liveness = await axios.get(VECTOR_API_URL, { timeout: 5000 });
    if (!liveness.data.ok) {
      return res.status(503).json({ status: 'down', reason: 'vector_not_healthy' });
    }

    // Fetch Prometheus metrics
    const metricsResp = await axios.get(VECTOR_METRICS_URL, { timeout: 5000 });
    const metrics = metricsResp.data;

    // Check buffer utilization (vector_buffer_byte_size / vector_buffer_max_byte_size)
    const bufferUsed = parseMetric(metrics, 'vector_buffer_byte_size');
    const bufferMax = parseMetric(metrics, 'vector_buffer_max_byte_size');
    let maxFillRatio = 0;
    for (let i = 0; i < Math.min(bufferUsed.length, bufferMax.length); i++) {
      if (bufferMax[i] > 0) {
        maxFillRatio = Math.max(maxFillRatio, bufferUsed[i] / bufferMax[i]);
      }
    }

    // Check component error counts
    const componentErrors = parseMetric(metrics, 'vector_component_errors_total');
    const totalErrors = componentErrors.reduce((a, b) => a + b, 0);

    // Determine status
    if (maxFillRatio >= BUFFER_FILL_THRESHOLD) {
      return res.status(503).json({
        status: 'degraded',
        reason: 'buffer_near_full',
        fill_ratio: maxFillRatio.toFixed(3),
      });
    }

    if (totalErrors > COMPONENT_ERRORS_MAX) {
      return res.status(503).json({
        status: 'degraded',
        reason: 'component_errors_elevated',
        total_errors: totalErrors,
      });
    }

    return res.status(200).json({
      status: 'ok',
      buffer_fill_ratio: maxFillRatio.toFixed(3),
      component_errors: totalErrors,
    });
  } catch (err) {
    return res.status(503).json({ status: 'down', error: err.message });
  }
});

app.listen(process.env.PORT || 3007, () => {
  console.log('Vector health sidecar listening on :3007');
});

Run it alongside Vector:

npm install express axios
node vector-health.js &

Step 3: Configure Vigilmon HTTP Monitors for Vector

Monitor 1: Vector Process Liveness

1. Log in to vigilmon.online and go to Monitors → New Monitor
2. Choose HTTP / HTTPS
3. Set the URL to https://your-host.example.com/health/vector
4. Set the check interval to 1 minute
5. Under Expected response: Status code 200, body contains "status":"ok"
6. Response time threshold: 5000ms
7. Assign your on-call alert channel
8. Save

Monitor 2: Vector API Health (Direct)

Add a second, direct monitor on Vector's own API endpoint as a redundant check:

• URL: http://your-host:8686/health (or proxied through your reverse proxy)
• Expected: 200, body contains "ok":true
• Interval: 1 minute

This fires even if your health sidecar crashes, giving you defense in depth.

Monitor 3: Per-Sink Delivery Health

For critical sinks (Elasticsearch, S3, Datadog), add a separate check that confirms the downstream target is reachable from the same host as Vector:

# Add a dedicated endpoint to your sidecar for sink connectivity
app.get('/health/vector/sink/elasticsearch', async (req, res) => {
  try {
    await axios.get(process.env.ES_HEALTH_URL || 'http://elasticsearch:9200/_cluster/health', { timeout: 3000 });
    return res.status(200).json({ status: 'ok' });
  } catch (err) {
    return res.status(503).json({ status: 'down', error: err.message });
  }
});

Monitor this at /health/vector/sink/elasticsearch with a 2-minute interval. If Vector can't reach Elasticsearch, events will buffer up — you want to know before the buffer overflows.

Step 4: Heartbeat Monitoring for Vector Pipeline Throughput

Process health checks confirm Vector is running — but they don't prove data is flowing. A zero-throughput pipeline (no events passing any transform) looks healthy from the outside.

Wire a throughput heartbeat into your pipeline using Vector's http sink:

# In vector.yaml — add a heartbeat sink after your main transforms

[transforms.rate_check]
  type = "remap"
  inputs = ["your_main_transform"]
  source = '''
  .heartbeat_eligible = true
  '''

[sinks.vigilmon_heartbeat]
  type = "http"
  inputs = ["rate_check"]
  uri = "${VIGILMON_HEARTBEAT_URL}"
  method = "get"
  # Only send heartbeat once per minute, not once per event
  batch.timeout_secs = 60
  encoding.codec = "json"
  # Suppress the actual event body — we just need the ping
  request.headers.content-length = "0"

Set the heartbeat monitor in Vigilmon with a 5-minute expected interval and a 10-minute grace period. If events stop flowing through your Vector pipeline entirely, the heartbeat stops and Vigilmon alerts within 10 minutes.

Step 5: Alert Routing for Vector

| Monitor | Alert Channel | Priority | |---|---|---| | Vector liveness /health/vector | Slack + PagerDuty | P1 | | Vector API /health direct | PagerDuty | P1 | | Sink connectivity /health/vector/sink/* | Slack | P2 | | Pipeline throughput heartbeat | Slack + email | P2 |

Set response time thresholds:

• Alert at 2000ms for the liveness endpoint (slow health checks signal process pressure)
• Alert at 1000ms for the Vector API direct endpoint (should be near-instant)

For multi-region Vector deployments, create separate monitors per region/cluster and group them in Vigilmon using monitor tags (region:us-east, region:eu-west). This makes it easy to distinguish regional outages from global Vector incidents.

Summary

Vector is fast and reliable by design, but when it fails, it fails silently — events buffer up, transforms stall, and sinks stop receiving data. External monitoring with Vigilmon covers what internal dashboards miss:

| Monitor Type | What It Covers | |---|---| | HTTP on /health/vector | Process liveness, buffer fill ratio, component errors | | HTTP on Vector API /health | Direct process health, defense in depth | | HTTP on sink checks | Downstream connectivity before buffer overflow | | Heartbeat | Active data flow through the pipeline |

Get started free at vigilmon.online — your first Vector monitor is running in under two minutes.