How to Monitor Vector: Source/Sink Health, Event Throughput, and Observability Pipeline Uptime with Vigilmon

Vector, built by Datadog, is a high-performance observability data pipeline. It ingests logs, metrics, and traces from sources (files, Kafka, Syslog, HTTP) and routes them to sinks (Elasticsearch, S3, Datadog, Loki, Kafka). When a Vector pipeline stalls — a backpressure buildup, a dead sink, or a source that stops delivering events — your observability infrastructure goes dark. Dashboards show no data, alerts stop firing, and the only indicator is the absence of information.

Vigilmon gives you external visibility into your Vector pipelines through Vector's built-in API, a lightweight health sidecar, and heartbeat monitoring for your pipeline's own throughput. This tutorial covers all three approaches.

Why Vector Needs External Monitoring

Vector's internal tooling (vector top, Prometheus metrics, the Vector API) gives rich internal telemetry — but only if someone is watching it. External monitoring with Vigilmon catches:

• Complete pipeline failure — Vector process crashes or OOMs
• Sink delivery failures — events accumulate in buffers while the sink is unreachable
• Buffer saturation — disk or memory buffers full, causing backpressure and eventual event drops
• Source stalls — input sources stop delivering events (Kafka consumer falls behind, file source stops tailing)
• Component error rate spikes — individual transforms or sinks throwing errors at elevated rates

The critical insight: when Vector's pipeline stalls, your other monitoring tools may also go dark (if they depend on Vector for their log/metric delivery). You need an out-of-band monitor that doesn't rely on the pipeline itself.

Step 1: Enable Vector's Built-In API

Vector ships with a built-in GraphQL API that exposes component health, throughput, and error rates. Enable it in your vector.toml:

[api]
enabled = true
address = "0.0.0.0:8686"
# Only expose internally — do not expose to the public internet

With the API enabled, Vector's playground and health endpoints are available:

# Health check — returns 200 when Vector is running
curl -i http://localhost:8686/health

# Prometheus metrics endpoint
curl http://localhost:8686/metrics

# GraphQL API for component stats
curl -X POST http://localhost:8686/graphql \
  -H "Content-Type: application/json" \
  -d '{"query":"{ components { edges { node { componentId componentType } } } }"}'

Point a Vigilmon HTTP monitor at http://your-vector-host:8686/health as your first monitor.

Step 2: Build a Comprehensive Health Sidecar

The /health endpoint only tells you Vector is running. For meaningful alerting on throughput, buffer fullness, and error rates, build a sidecar that queries Vector's API:

// health/vector.js
const express = require('express');
const fetch = require('node-fetch');

const app = express();
const VECTOR_API = process.env.VECTOR_API_URL || 'http://localhost:8686';

// GraphQL query for component statistics
const COMPONENT_STATS_QUERY = `
  query {
    componentAllocatedBytes {
      componentId
      metric {
        value
      }
    }
    componentReceivedEventsThroughput {
      componentId
      throughput
    }
    componentSentEventsThroughput {
      componentId
      throughput
    }
    componentErrors {
      componentId
      metric {
        value
      }
    }
  }
`;

async function queryVectorAPI(query) {
  const res = await fetch(`${VECTOR_API}/graphql`, {
    method: 'POST',
    headers: { 'Content-Type': 'application/json' },
    body: JSON.stringify({ query }),
    signal: AbortSignal.timeout(10_000),
  });
  if (!res.ok) throw new Error(`Vector API returned HTTP ${res.status}`);
  return res.json();
}

app.get('/health/vector', async (req, res) => {
  try {
    // First, check the health endpoint
    const healthRes = await fetch(`${VECTOR_API}/health`, {
      signal: AbortSignal.timeout(5_000),
    });
    if (!healthRes.ok) {
      return res.status(503).json({ status: 'down', reason: 'vector_health_failed' });
    }

    // Then check component stats
    const data = await queryVectorAPI(COMPONENT_STATS_QUERY);
    const { componentErrors, componentSentEventsThroughput, componentReceivedEventsThroughput } = data.data;

    const ERROR_THRESHOLD = parseFloat(process.env.VECTOR_ERROR_THRESHOLD || '10');
    const issues = [];

    // Check for components with elevated error rates
    for (const comp of (componentErrors || [])) {
      if (comp.metric.value > ERROR_THRESHOLD) {
        issues.push({
          component: comp.componentId,
          errors: comp.metric.value,
          threshold: ERROR_THRESHOLD,
        });
      }
    }

    // Check for source components with zero throughput (potential stall)
    const STALL_CHECK_COMPONENTS = (process.env.VECTOR_MONITOR_SOURCES || '').split(',').filter(Boolean);
    for (const comp of (componentReceivedEventsThroughput || [])) {
      if (STALL_CHECK_COMPONENTS.includes(comp.componentId) && comp.throughput === 0) {
        issues.push({ component: comp.componentId, issue: 'zero_input_throughput' });
      }
    }

    if (issues.length > 0) {
      return res.status(503).json({ status: 'degraded', issues });
    }

    return res.status(200).json({
      status: 'ok',
      components: componentSentEventsThroughput?.length || 0,
    });
  } catch (err) {
    return res.status(503).json({ status: 'down', error: err.message });
  }
});

// Separate buffer health endpoint using Prometheus metrics
app.get('/health/vector/buffers', async (req, res) => {
  try {
    const metricsRes = await fetch(`${VECTOR_API}/metrics`, {
      signal: AbortSignal.timeout(5_000),
    });
    if (!metricsRes.ok) return res.status(503).json({ status: 'down' });

    const metricsText = await metricsRes.text();
    const issues = [];

    // Parse buffer byte size and max size to calculate fill percentage
    const bufferSizeMatches = [...metricsText.matchAll(/vector_buffer_byte_size\{component_id="([^"]+)"[^}]*\} ([\d.]+)/g)];
    const bufferMaxMatches = [...metricsText.matchAll(/vector_buffer_max_byte_size\{component_id="([^"]+)"[^}]*\} ([\d.]+)/g)];

    for (const match of bufferSizeMatches) {
      const componentId = match[1];
      const currentSize = parseFloat(match[2]);
      const maxEntry = bufferMaxMatches.find(m => m[1] === componentId);
      if (maxEntry) {
        const maxSize = parseFloat(maxEntry[2]);
        const fillPct = (currentSize / maxSize) * 100;
        const BUFFER_FILL_THRESHOLD = parseFloat(process.env.VECTOR_BUFFER_FILL_THRESHOLD || '80');
        if (fillPct > BUFFER_FILL_THRESHOLD) {
          issues.push({ component: componentId, fill_pct: Math.round(fillPct), threshold: BUFFER_FILL_THRESHOLD });
        }
      }
    }

    if (issues.length > 0) {
      return res.status(503).json({ status: 'degraded', reason: 'buffer_near_full', issues });
    }

    return res.status(200).json({ status: 'ok', buffers_healthy: bufferSizeMatches.length });
  } catch (err) {
    return res.status(503).json({ status: 'down', error: err.message });
  }
});

app.listen(3014, () => console.log('Vector health sidecar running on :3014'));

Step 3: Configure Vigilmon Monitors for Vector

1. Log in to vigilmon.online → Monitors → New Monitor → HTTP / HTTPS

Set up the following monitors:

| Monitor | URL | Interval | Priority | |---|---|---|---| | Vector process health | http://vector-host:8686/health | 1 minute | P1 | | Component + error rates | /health/vector | 2 minutes | P1 | | Buffer fill levels | /health/vector/buffers | 2 minutes | P2 |

For each monitor:

• Expected status: 200
• Response body contains: "status":"ok" (for sidecar endpoints) or ok (for the native /health endpoint)
• Response time threshold: 10 000ms
• Alert channel: Slack #observability-oncall for P1

Step 4: Heartbeat Monitoring via Vector's HTTP Sink

The most reliable way to monitor Vector is to make Vector itself prove it's working. Configure an HTTP sink that sends a periodic heartbeat to Vigilmon — if events stop flowing through the pipeline, the heartbeat stops and Vigilmon fires an alert.

Set Up the Heartbeat Monitor in Vigilmon

1. Vigilmon → Monitors → New Monitor → Heartbeat
2. Name: vector-pipeline-throughput
3. Expected interval: 5 minutes
4. Grace period: 10 minutes
5. Copy the heartbeat URL: https://vigilmon.online/heartbeat/abc123xyz

Configure Vector to Send Heartbeats

Add a source that generates synthetic events, a transform to rate-limit them, and an HTTP sink that pings Vigilmon:

# vector.toml

# Internal metrics as source events every 60 seconds
[sources.internal_metrics]
type = "internal_metrics"
scrape_interval_secs = 60

# Rate-limit: only forward one event per 5 minutes to avoid hammering Vigilmon
[transforms.heartbeat_ratelimit]
type = "throttle"
inputs = ["internal_metrics"]
threshold = 1
window_secs = 300

# Transform the metric into a minimal HTTP-pingable payload
[transforms.heartbeat_payload]
type = "remap"
inputs = ["heartbeat_ratelimit"]
source = '''
  . = {"status": "ok", "ts": now()}
'''

# HTTP sink to Vigilmon heartbeat URL
[sinks.vigilmon_heartbeat]
type = "http"
inputs = ["heartbeat_payload"]
uri = "${VIGILMON_HEARTBEAT_URL}"
method = "get"
encoding.codec = "json"

[sinks.vigilmon_heartbeat.request]
timeout_secs = 10
retry_attempts = 3

This configuration proves end-to-end pipeline liveness — if any critical component between the source and the HTTP sink fails, the heartbeat stops.

Monitoring a Specific Pipeline Path

For critical pipelines (e.g., security logs → SIEM), configure a dedicated heartbeat path through the exact components you want to verify:

[sources.security_logs]
type = "file"
include = ["/var/log/auth.log"]

[transforms.parse_security]
type = "remap"
inputs = ["security_logs"]
source = '''
  . = parse_syslog!(string!(.message))
'''

[sinks.siem]
type = "elasticsearch"
inputs = ["parse_security"]
endpoints = ["${ELASTICSEARCH_URL}"]

# Parallel heartbeat branch from the same source
[transforms.security_heartbeat_payload]
type = "remap"
inputs = ["security_logs"]
source = '''
  . = {"status": "ok", "pipeline": "security_logs"}
'''

[sinks.vigilmon_security_heartbeat]
type = "http"
inputs = ["security_heartbeat_payload"]
uri = "${VIGILMON_SECURITY_HEARTBEAT_URL}"
method = "get"
encoding.codec = "json"

[sinks.vigilmon_security_heartbeat.request]
timeout_secs = 10

Step 5: Alert Routing for Observability Pipeline Failures

Observability pipeline failures require careful alert routing — an alert about your monitoring being down needs to reach you through a channel that doesn't depend on the pipeline being monitored.

Configure alert routing in Vigilmon:

| Monitor | Alert Channel | Rationale | |---|---|---| | Vector process health | PagerDuty + email | Pipeline completely down — use OOB channels | | Component error rate | Slack #observability-oncall | Degraded, but still processing | | Buffer fill > 80% | Slack #observability-oncall | Imminent backpressure | | Pipeline heartbeat | PagerDuty + email | End-to-end delivery stalled |

Use email and PagerDuty (not Slack) for Vector health alerts if your Slack notifications are routed through a Vector pipeline — a downed pipeline would suppress exactly the alerts you need to receive.

Summary

Vector pipeline failures go dark silently. External monitoring with Vigilmon catches process crashes, sink failures, buffer saturation, and component error spikes before your observability gap becomes an incident gap:

| Monitor Type | What It Covers | |---|---| | HTTP: Vector /health | Process liveness | | HTTP: component errors | Per-component error rate spikes | | HTTP: buffer levels | Backpressure and near-full buffers | | Heartbeat via HTTP sink | End-to-end pipeline event delivery |

Get started free at vigilmon.online — your observability pipeline monitors are live in under two minutes.