tutorial

How to Monitor Vector.dev Pipelines, Sources, Sinks, and Buffer Health with Vigilmon

Vector is a high-performance observability data pipeline that routes logs, metrics, and traces between sources (Kubernetes, files, syslog) and sinks (Elastic...

Vector is a high-performance observability data pipeline that routes logs, metrics, and traces between sources (Kubernetes, files, syslog) and sinks (Elasticsearch, Datadog, S3, Kafka). When Vector goes silent — due to a crashed process, a blocked sink, or buffer saturation — your entire observability stack loses data without any indication that collection has stopped.

In this tutorial you'll set up uptime monitoring for your Vector deployments using Vigilmon — free tier, no credit card required.

Why Vector needs external monitoring

Vector processes your observability data, which means Vector failures are invisible from inside the systems you're trying to observe. Key failure modes:

  • Vector process crashes — the systemd service or Kubernetes pod restarts, losing buffered events during the restart window; silently drops data
  • Sink backpressure — an Elasticsearch or Kafka sink falls behind; Vector's in-memory buffers fill up; upstream sources begin dropping new events
  • Disk buffer saturation — Vector's disk-based buffer fills the partition; new events are dropped; the partition may fill to 100% and crash the host
  • Source disconnection — a file source stops following rotated logs; the docker_logs source loses connection to the Docker daemon; data silently stops flowing
  • Transform latency spike — a VRL (Vector Remap Language) transform doing heavy regex or JSON parsing causes event queue buildup upstream
  • TLS certificate expiry on sinks — HTTPS sinks begin rejecting connections; Vector retries indefinitely but data never lands at the destination
  • API endpoint unavailable — Vector's built-in HTTP API (used for health checks and topology inspection) becomes unreachable, hiding all diagnostic information

External monitoring gives you an independent heartbeat for both Vector's process health and the end-to-end data pipeline.

What you'll need

  • A running Vector deployment (systemd service, Docker container, or Kubernetes DaemonSet)
  • Vector's HTTP API enabled (see Step 1)
  • A free Vigilmon account (sign up takes 30 seconds)

Step 1: Enable Vector's built-in HTTP API

Vector ships with an HTTP API that exposes health, topology, and component metrics. Enable it in your Vector config:

# vector.toml

[api]
  enabled = true
  address = "0.0.0.0:8686"
  # Set to false in production to disable the GraphQL playground
  playground = false

For Kubernetes deployments using the Vector Helm chart:

# values.yaml
customConfig:
  api:
    enabled: true
    address: 0.0.0.0:8686
    playground: false

service:
  ports:
    - name: api
      port: 8686
      protocol: TCP

Verify the API is running:

curl http://localhost:8686/health
# {"ok":true}

# List all configured components
curl http://localhost:8686/graphql \
  -H 'Content-Type: application/json' \
  -d '{"query":"{ components { nodes { componentId componentType } } }"}'

Step 2: Monitor Vector process health

Vector's /health endpoint is the fastest liveness check. It returns {"ok":true} only when the Vector process is fully initialized and not in a crash loop.

Log in to Vigilmon and navigate to Monitors → Add Monitor:

For systemd/Docker deployments:

| Field | Value | |-------|-------| | Monitor type | HTTP | | Name | Vector Pipeline (primary) | | URL | http://<vector-host>:8686/health | | HTTP method | GET | | Expected status | 200 | | Response must contain | "ok":true | | Check interval | 1 minute | | Regions | Select 3+ globally distributed regions |

For Kubernetes DaemonSet (per-node monitoring):

Expose the API via a NodePort or headless service, then add one monitor per critical node or use a load-balanced service endpoint that returns healthy only when all pods are healthy.

# vector-api-service.yaml
apiVersion: v1
kind: Service
metadata:
  name: vector-api
  namespace: vector
spec:
  type: LoadBalancer
  selector:
    app.kubernetes.io/name: vector
  ports:
    - name: api
      port: 8686
      targetPort: 8686

Step 3: Monitor event throughput with a custom health endpoint

A healthy Vector process that isn't processing events is still a failure. Build a throughput probe that checks recent event counts:

# vector-throughput-probe/app.py
import requests
from flask import Flask, jsonify
import time

app = Flask(__name__)

VECTOR_API = "http://localhost:8686"
MIN_EVENTS_PER_MINUTE = 100  # Set to expected minimum throughput

# GraphQL query to get component-level event counts
THROUGHPUT_QUERY = """
{
  sources {
    nodes {
      componentId
      metrics {
        receivedEventsTotal {
          receivedEventsTotal
          timestamp
        }
      }
    }
  }
}
"""

last_counts = {}
last_check = 0

@app.route('/health/throughput')
def throughput_health():
    global last_counts, last_check

    try:
        resp = requests.post(
            f'{VECTOR_API}/graphql',
            json={'query': THROUGHPUT_QUERY},
            timeout=5
        )
        resp.raise_for_status()
        data = resp.json()
    except Exception as e:
        return jsonify({'status': 'error', 'message': str(e)}), 503

    sources = data.get('data', {}).get('sources', {}).get('nodes', [])
    now = time.time()
    issues = []

    for source in sources:
        component_id = source['componentId']
        total = source.get('metrics', {}).get('receivedEventsTotal', {})
        current_count = total.get('receivedEventsTotal', 0)

        if component_id in last_counts and last_check > 0:
            elapsed_minutes = (now - last_check) / 60
            rate = (current_count - last_counts[component_id]) / elapsed_minutes
            if rate < MIN_EVENTS_PER_MINUTE:
                issues.append({
                    'source': component_id,
                    'events_per_minute': round(rate, 1),
                    'minimum': MIN_EVENTS_PER_MINUTE,
                })

        last_counts[component_id] = current_count

    last_check = now

    if issues:
        return jsonify({'status': 'low_throughput', 'sources': issues}), 503

    return jsonify({
        'status': 'ok',
        'source_count': len(sources),
        'checked_at': now,
    })

if __name__ == '__main__':
    app.run(host='0.0.0.0', port=9094)

Add a Vigilmon monitor:

| Field | Value | |-------|-------| | Monitor type | HTTP | | Name | Vector Event Throughput | | URL | http://<vector-host>:9094/health/throughput | | Expected status | 200 | | Response must contain | "status":"ok" | | Check interval | 2 minutes |

Step 4: Monitor buffer utilization

Vector's disk buffers can fill when sinks are slow or offline. A full buffer means data loss. Monitor buffer utilization:

# Vector exposes buffer metrics via its GraphQL API
curl -s http://localhost:8686/graphql \
  -H 'Content-Type: application/json' \
  -d '{"query":"{ sinks { nodes { componentId metrics { bufferEventsTotal { bufferEventsTotal } bufferByteSize { bufferByteSize } } } } }"}' \
  | python3 -m json.tool

Build a buffer health endpoint:

// buffer-health-probe/index.js
const axios = require('axios');
const express = require('express');
const app = express();

const VECTOR_API = process.env.VECTOR_API || 'http://localhost:8686';
const MAX_BUFFER_BYTES = parseInt(process.env.MAX_BUFFER_BYTES || '1073741824'); // 1GB default

const BUFFER_QUERY = `{
  sinks {
    nodes {
      componentId
      metrics {
        bufferByteSize { bufferByteSize }
        bufferEventsTotal { bufferEventsTotal }
      }
    }
  }
}`;

app.get('/health/buffers', async (req, res) => {
  try {
    const { data } = await axios.post(
      `${VECTOR_API}/graphql`,
      { query: BUFFER_QUERY },
      { timeout: 5000 }
    );

    const sinks = data?.data?.sinks?.nodes || [];
    const saturated = [];

    for (const sink of sinks) {
      const bufferBytes = sink.metrics?.bufferByteSize?.bufferByteSize || 0;
      const utilization = bufferBytes / MAX_BUFFER_BYTES;

      if (utilization > 0.8) {
        saturated.push({
          sink: sink.componentId,
          buffer_bytes: bufferBytes,
          utilization_pct: Math.round(utilization * 100),
        });
      }
    }

    if (saturated.length > 0) {
      return res.status(503).json({
        status: 'saturated',
        sinks: saturated,
      });
    }

    res.json({ status: 'ok', sinks_checked: sinks.length });
  } catch (err) {
    res.status(503).json({ status: 'error', message: err.message });
  }
});

app.listen(9095);

Add a Vigilmon monitor:

| Field | Value | |-------|-------| | Monitor type | HTTP | | Name | Vector Buffer Utilization | | URL | http://<vector-host>:9095/health/buffers | | Expected status | 200 | | Response must contain | "status":"ok" | | Check interval | 2 minutes |

Step 5: Monitor sink connectivity with TCP checks

Even before Vector starts rejecting events, you can detect sink connectivity issues with TCP monitors. Add one per critical sink:

Elasticsearch sink:

| Field | Value | |-------|-------| | Monitor type | TCP | | Name | Vector → Elasticsearch (sink) | | Host | <elasticsearch-host> | | Port | 9200 | | Check interval | 1 minute |

Kafka sink:

| Field | Value | |-------|-------| | Monitor type | TCP | | Name | Vector → Kafka Broker (sink) | | Host | <kafka-broker-host> | | Port | 9092 | | Check interval | 1 minute |

These TCP monitors alert you when the sink is unreachable, even before Vector's own retry logic gives up and starts dropping events.

Step 6: Monitor Vector's component error rate

Vector tracks per-component error rates. An elevated error rate on a source or sink indicates data is being dropped:

# Query error metrics via GraphQL
curl -s http://localhost:8686/graphql \
  -H 'Content-Type: application/json' \
  -d '{"query":"{ sources { nodes { componentId metrics { receivedBytesTotal { receivedBytesTotal } } } } }"}'

Add the error rate check to your throughput probe or create a dedicated endpoint that returns non-200 when the error rate on any component exceeds a threshold:

# In your existing probe, add:
ERROR_QUERY = """
{
  components {
    nodes {
      componentId
      componentType
      metrics {
        errorsTotal {
          errorsTotal
        }
      }
    }
  }
}
"""

@app.route('/health/errors')
def error_health():
    try:
        resp = requests.post(
            f'{VECTOR_API}/graphql',
            json={'query': ERROR_QUERY},
            timeout=5
        )
        data = resp.json()
    except Exception as e:
        return jsonify({'status': 'error', 'message': str(e)}), 503

    components = data.get('data', {}).get('components', {}).get('nodes', [])
    elevated = [
        {'component': c['componentId'], 'errors': c['metrics']['errorsTotal']['errorsTotal']}
        for c in components
        if c.get('metrics', {}).get('errorsTotal', {}).get('errorsTotal', 0) > 0
    ]

    if elevated:
        return jsonify({'status': 'errors_detected', 'components': elevated}), 503

    return jsonify({'status': 'ok'})

Step 7: Configure alerts

In Vigilmon, go to Alerts → Alert Channels and set up your notification channels.

Recommended alert policy for Vector monitors:

| Monitor | Alert after | Rationale | |---------|-------------|-----------| | Vector Health HTTP | 1 failure | Process crash; all data collection stops | | Event Throughput HTTP | 2 failures | Allow one check for pipeline startup ramp | | Buffer Utilization HTTP | 2 failures | Give time for sink recovery before alerting | | Sink TCP monitors | 2 failures | Brief network hiccups vs. real sink failure |

Enable Recovery alerts on all monitors — knowing when Vector resumes processing is as important as knowing it stopped.

Step 8: Create a pipeline observability status page

Vector is often the single pipeline delivering logs and metrics to your observability tools. Create a status page your team can check before investigating "why is Grafana empty?":

  1. In Vigilmon, go to Status Pages → Create Status Page
  2. Name it "Observability Pipeline Health"
  3. Group monitors as:
    • Collection: Vector Health, Event Throughput
    • Pipeline: Buffer Utilization, Error Rate
    • Destinations: Elasticsearch TCP, Kafka TCP, (any other sinks)
  4. Share the URL in your #ops or #observability Slack channel

What you're monitoring now

| Monitor | What it detects | |---------|-----------------| | Vector Health HTTP | Process crash, restart loop, initialization failure | | Event Throughput HTTP | Sources going silent, pipeline stalls | | Buffer Utilization HTTP | Sink backpressure causing data loss risk | | Sink TCP monitors | Connectivity failures before Vector starts dropping data | | Error Rate HTTP | Per-component drops and processing failures |

Conclusion

Vector is the foundation of your observability pipeline, which makes it especially easy to miss failures — if Vector stops collecting data, your dashboards go quiet but don't alert. Vigilmon gives you an independent external heartbeat that catches process crashes, buffer saturation, throughput drops, and sink connectivity failures before they turn into gaps in your observability data.

Sign up for a free Vigilmon account and make your observability pipeline observable.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles