Monitoring Telegraf with Vigilmon

Telegraf is the workhorse of the TICK stack — a plugin-driven metrics agent that collects from 300+ input sources and ships to 50+ destinations. It runs quietly in the background gathering CPU, memory, Docker containers, Kubernetes pods, databases, and more. Precisely because it runs quietly, a crashed or error-looping Telegraf daemon silently breaks every downstream dashboard and alert. Vigilmon monitors Telegraf itself, so you know when your metrics pipeline is healthy before gaps appear in InfluxDB or Prometheus.

What You'll Set Up

• HTTP uptime monitor for Telegraf's Prometheus metrics endpoint (port 9273)
• Keyword check confirming internal Telegraf metrics are being served
• Error-rate monitor detecting failing input plugin gather cycles
• SSL certificate alerts for TLS-enabled Telegraf endpoints
• Cron heartbeat confirming Telegraf's gather cycles are completing end-to-end

Prerequisites

• Telegraf 1.20+ installed and running as a daemon
• prometheus_client output plugin enabled in telegraf.conf (exposes metrics on port 9273)
• internal input plugin enabled in telegraf.conf (provides self-monitoring metrics)
• A free Vigilmon account

Step 1: Enable Telegraf's Internal Metrics and Prometheus Endpoint

Telegraf exposes self-monitoring metrics through two plugins that must be active for Vigilmon to probe. Verify or add both in /etc/telegraf/telegraf.conf:

# Collect internal Telegraf metrics (gather counts, errors, write latency)
[[inputs.internal]]
  collect_memstats = true

# Expose metrics in Prometheus format on port 9273
[[outputs.prometheus_client]]
  listen = ":9273"
  metric_version = 2

Restart Telegraf after any config change:

sudo systemctl restart telegraf
sudo systemctl status telegraf

Verify the endpoint is live:

curl -s http://localhost:9273/metrics | grep telegraf_internal

You should see lines like:

telegraf_internal_gather_metrics_gathered{input="cpu",...} 4
telegraf_internal_gather_errors{input="cpu",...} 0

If you see output, Telegraf's self-monitoring is working and ready for Vigilmon.

Step 2: Monitor the Telegraf Prometheus Metrics Endpoint

Add a Vigilmon HTTP monitor for the /metrics endpoint:

1. Log in to vigilmon.online and click Add Monitor.
2. Set Type to HTTP / HTTPS.
3. Enter the URL: http://YOUR_SERVER_IP:9273/metrics
4. Set Check interval to 1 minute.
5. Set Expected HTTP status to 200.
6. Under Keyword monitoring, enable it and enter the keyword: telegraf_internal
7. Click Save.

The keyword check telegraf_internal confirms that:

• The prometheus_client output plugin is running
• The internal input plugin is active and producing metrics
• Telegraf is not just alive at the TCP level but actually gathering and reporting metrics

A 200 response without the keyword means Telegraf is running but the internal metrics plugin has stopped, which is a degraded state worth alerting on.

Step 3: Monitor Telegraf Gather Error Rate

Rising gather errors mean input plugins are silently failing — metrics gaps appear in your dashboards without any visible crash. Add a second Vigilmon monitor targeting the same endpoint with a keyword that detects a clean error state:

1. Click Add Monitor → HTTP / HTTPS.
2. URL: http://YOUR_SERVER_IP:9273/metrics
3. Check interval: 2 minutes
4. Under Keyword monitoring, enter: gather_errors 0
5. Click Save.

This keyword matches the telegraf_internal_gather_errors metric when its value is 0. If any input plugin begins producing errors, the metric value changes and the keyword no longer matches, triggering a Vigilmon alert.

For a more targeted check, you can also query a specific plugin's error count:

# Check for any non-zero gather errors across all plugins
curl -s http://localhost:9273/metrics | grep gather_errors | grep -v ' 0$'

If this returns any lines, those input plugins are failing to collect and Telegraf is producing silent metric gaps.

Step 4: Check the Telegraf Health Endpoint (If Available)

Some Telegraf versions (1.28+) expose a dedicated /health endpoint when the health output plugin is configured:

[[outputs.health]]
  service_address = "http://:8080"

  [[outputs.health.compares]]
    field = "gather_errors"
    lt = 1.0

  [[outputs.health.contains]]
    field = "running"

If your Telegraf version supports this, add an additional Vigilmon monitor:

1. Add Monitor → HTTP / HTTPS
2. URL: http://YOUR_SERVER_IP:8080
3. Expected status: 200
4. Keyword: ok or running

For older Telegraf versions without the health plugin, the /metrics monitor from Step 2 serves as the canonical health check.

Step 5: SSL Certificate Alerts for TLS Telegraf Endpoints

If Telegraf's prometheus_client output is configured with TLS:

[[outputs.prometheus_client]]
  listen = ":9273"
  tls_cert = "/etc/telegraf/certs/server.crt"
  tls_key = "/etc/telegraf/certs/server.key"

Enable SSL monitoring in the Vigilmon monitor created in Step 2:

1. Open the monitor and scroll to the SSL section.
2. Enable Monitor SSL certificate.
3. Set Alert when certificate expires in less than 21 days.
4. Click Save.

Also update the monitor URL to https://YOUR_SERVER_IP:9273/metrics.

A 21-day alert window gives you enough time to rotate the certificate before Telegraf's metrics endpoint becomes inaccessible to your entire monitoring infrastructure.

Step 6: Heartbeat Monitoring for Telegraf Gather Cycles

The Prometheus endpoint confirms Telegraf is running, but it doesn't prove gather cycles are completing end-to-end — a stalled gather cycle can leave the HTTP endpoint responding while metrics silently stop flowing to InfluxDB or Prometheus remote write. Use a Vigilmon heartbeat to confirm the full pipeline.

Create the heartbeat:

1. In Vigilmon, click Add Monitor → Cron Heartbeat.
2. Set the expected ping interval to 60 seconds (or match your Telegraf collection interval).
3. Copy the heartbeat URL: https://vigilmon.online/heartbeat/YOUR_TOKEN

Wire the heartbeat into Telegraf's config:

Add an exec input plugin that pings Vigilmon after every successful gather cycle:

[[inputs.exec]]
  commands = ["curl -fsS https://vigilmon.online/heartbeat/YOUR_TOKEN"]
  interval = "60s"
  timeout = "5s"
  data_format = "ignore"

The data_format = "ignore" tells Telegraf not to try to parse the heartbeat response as a metric. The curl -fsS flags make curl fail silently on error (no output to parse) and exit non-zero only on HTTP errors.

Restart Telegraf:

sudo systemctl restart telegraf

Verify the heartbeat is reaching Vigilmon by checking the heartbeat monitor's last-ping timestamp in the Vigilmon dashboard. If the gather cycle stalls or the exec input fails, the heartbeat stops and Vigilmon alerts after the configured interval.

Step 7: Configure Alert Channels

1. Go to Alert Channels in Vigilmon and add Slack, email, or PagerDuty.
2. Set Consecutive failures before alert to 2 on the metrics endpoint monitor — transient network hiccups should not page on-call.
3. Set Consecutive failures before alert to 1 on the heartbeat monitor — a missed heartbeat is a strong signal that Telegraf's gather loop has stalled.

For Telegraf deployments that aggregate metrics from many remote agents (Telegraf as a pull aggregator), consider one heartbeat monitor per upstream agent cluster to detect partial failures.

Summary

| Monitor | Target | What It Catches | |---|---|---| | HTTP + keyword telegraf_internal | :9273/metrics | Telegraf daemon down, internal plugin stopped | | HTTP + keyword gather_errors 0 | :9273/metrics | Input plugin failures causing silent metric gaps | | HTTP health endpoint | :8080 | Telegraf health plugin degraded state (v1.28+) | | SSL certificate | :9273 (if TLS) | Expiring TLS certificate on metrics endpoint | | Cron heartbeat | Heartbeat URL | Stalled gather cycles; metrics no longer flowing to InfluxDB/Prometheus |

Telegraf is the silent foundation of your metrics pipeline. When it stops working, every dashboard goes stale and every threshold alert stops firing — often without any visible error. With Vigilmon watching Telegraf's own health endpoint, gather error rate, and end-to-end heartbeat, you catch a broken collection agent before it breaks your ability to catch anything else.