Monitoring Collectd with Vigilmon

Collectd is the battle-tested system statistics daemon that has quietly collected CPU, memory, disk, and network metrics on Linux servers since 2005. It runs with minimal resource overhead, supports an extensive plugin ecosystem (MySQL, PostgreSQL, Redis, Apache, SNMP, JMX, and more), and outputs metrics to RRD files, Graphite, InfluxDB, Prometheus, and Kafka. System administrators running legacy infrastructure, network devices, and environments where lightweight agents are preferred over heavyweight observability stacks often rely on Collectd as their only metrics pipeline. When Collectd stops working, that pipeline goes silent — and unlike modern agents, Collectd does not expose an HTTP health endpoint by default. Vigilmon fills that gap by monitoring Collectd's network plugin port, write_http destination, and delivery heartbeat.

What You'll Set Up

• TCP probe monitoring Collectd's network plugin port (port 25826)
• HTTP monitor for Collectd's write_http output destination
• Log-based alerting for write_http delivery failures
• SSL certificate alerts for HTTPS write_http destinations
• Cron heartbeat confirming end-to-end metric delivery

Prerequisites

• Collectd installed and running (version 5.x or later)
• At least one of: network plugin, write_http plugin, or unixsock plugin configured
• A free Vigilmon account

Step 1: Understand Collectd's Monitoring Options

Unlike modern agents, Collectd does not expose an HTTP health endpoint by default. Depending on which output plugins you have configured, choose the appropriate Vigilmon monitoring approach:

| Collectd Plugin | Vigilmon Monitor Type | What to Monitor | |---|---|---| | network plugin | TCP probe | Port 25826 on the aggregator node | | write_http plugin | HTTP monitor | The destination endpoint URL | | unixsock plugin | Heartbeat + cron | Verify socket accepts values | | Any plugin | Cron heartbeat | Metric flush cycle confirmation |

Most production Collectd deployments use write_http to forward metrics to InfluxDB or a Graphite relay, making the HTTP destination monitor the primary Vigilmon check.

Step 2: Monitor the Collectd Network Plugin Port (TCP Probe)

If Collectd is configured as a metrics aggregator receiving data from remote Collectd nodes, the network plugin listens on port 25826:

# /etc/collectd/collectd.conf
LoadPlugin network

<Plugin network>
  Listen "0.0.0.0" "25826"
</Plugin>

Add a Vigilmon TCP monitor for the network plugin:

1. Log in to vigilmon.online and click Add Monitor.
2. Set Type to TCP.
3. Enter the host: YOUR_COLLECTD_SERVER_IP
4. Enter the port: 25826
5. Set Check interval to 1 minute.
6. Click Save.

A successful TCP connection to port 25826 confirms the Collectd daemon is running and the network plugin is accepting connections. This is the fastest possible health check for a Collectd aggregator node.

Verify the port is open before adding the monitor:

# On the Collectd host
ss -ulnp | grep 25826
# Or for TCP mode:
ss -tlnp | grep 25826

# From a remote host
nc -zv YOUR_COLLECTD_SERVER_IP 25826

Note: The network plugin uses UDP by default. If Vigilmon's TCP probe is not appropriate for UDP-only deployments, use the heartbeat method in Step 5 instead.

Step 3: Monitor the Collectd write_http Destination

The write_http plugin forwards Collectd metrics to an HTTP endpoint — typically InfluxDB's write API, a Graphite relay, or a custom metrics receiver. Monitoring the destination confirms the full Collectd pipeline is functional.

A typical write_http configuration:

LoadPlugin write_http

<Plugin write_http>
  <Node "influxdb">
    URL "http://influxdb.internal:8086/write?db=collectd"
    Format "JSON"
    StoreRates false
  </Node>
</Plugin>

Add a Vigilmon HTTP monitor for the InfluxDB write endpoint:

1. Add Monitor → HTTP / HTTPS
2. URL: http://influxdb.internal:8086/health (InfluxDB's health endpoint)
3. Check interval: 1 minute
4. Expected status: 200
5. Keyword: ready
6. Click Save

Monitoring the destination (InfluxDB, Graphite, etc.) rather than Collectd itself catches the scenario where Collectd is running but the destination is rejecting its writes. A failed destination causes Collectd to buffer or silently drop metrics.

If Collectd writes to a custom HTTP endpoint or relay, monitor that endpoint directly with an appropriate keyword.

Step 4: Detect Collectd write_http Failures via Log Monitoring

When write_http fails to deliver metrics, Collectd logs an error message. Enable the logfile plugin to capture these:

LoadPlugin logfile

<Plugin logfile>
  LogLevel info
  File "/var/log/collectd.log"
  Timestamp true
  PrintSeverity true
</Plugin>

Common write_http failure log entries look like:

[2026-07-13 10:15:23] [error] plugin write_http: POST request to influxdb failed with status 503
[2026-07-13 10:15:23] [error] write_http plugin: Sending failed, will retry.

Set up a log monitoring script that triggers a Vigilmon heartbeat failure when write errors appear:

#!/bin/bash
# /usr/local/bin/collectd-log-check.sh
# Run every minute from cron

ERRORS=$(grep "write_http.*failed" /var/log/collectd.log \
  --count --include-files=/var/log/collectd.log 2>/dev/null)

if [ "${ERRORS:-0}" -eq "0" ]; then
  curl -fsS https://vigilmon.online/heartbeat/YOUR_WRITE_HTTP_TOKEN
fi

This pattern inverts the heartbeat: the heartbeat only pings Vigilmon when there are no write_http errors in the log. Silence means the delivery pipeline has failed.

Step 5: SSL Certificate Alerts for HTTPS write_http Destinations

If Collectd's write_http plugin sends metrics to an HTTPS endpoint:

<Plugin write_http>
  <Node "influxdb">
    URL "https://influxdb.yourcompany.com:8086/write?db=collectd"
    Format "JSON"
    VerifyPeer true
    VerifyHost true
    CAFile "/etc/ssl/certs/ca-certificates.crt"
  </Node>
</Plugin>

Add SSL certificate monitoring for the destination:

1. Open the InfluxDB HTTP monitor created in Step 3.
2. Ensure the URL is https://influxdb.yourcompany.com:8086/health.
3. Scroll to the SSL section.
4. Enable Monitor SSL certificate.
5. Set Alert when certificate expires in less than 21 days.
6. Click Save.

An expired certificate on the InfluxDB endpoint causes Collectd's write_http plugin to reject the connection (if VerifyPeer is true), silently dropping all metric writes. A 21-day alert window ensures you rotate the destination certificate before Collectd's delivery begins failing.

Step 6: Heartbeat Monitoring for Collectd Metric Delivery

The most robust way to monitor Collectd end-to-end is a cron-triggered heartbeat that pings Vigilmon after each successful collection interval.

Create the heartbeat:

1. In Vigilmon, click Add Monitor → Cron Heartbeat.
2. Set the expected ping interval to match your Collectd collection interval.
   • Default Collectd interval: 10 seconds → set heartbeat to 30 seconds to allow 3× grace
   • Commonly configured to 60 seconds → set heartbeat to 90 seconds
3. Copy the heartbeat URL: https://vigilmon.online/heartbeat/YOUR_TOKEN

Option A: Use Collectd's exec plugin to ping Vigilmon

LoadPlugin exec

<Plugin exec>
  Exec "collectd" "/usr/local/bin/collectd-heartbeat.sh"
</Plugin>

Create /usr/local/bin/collectd-heartbeat.sh:

#!/bin/bash
# This script is run by Collectd's exec plugin at every collection interval
# It pings Vigilmon to confirm the Collectd daemon is alive and running

# Send PUTVAL to confirm we can use the exec plugin interface
echo "PUTVAL $(hostname)/heartbeat/gauge-vigilmon interval=60 $(date +%s):1"

# Ping Vigilmon heartbeat
curl -fsS https://vigilmon.online/heartbeat/YOUR_TOKEN --max-time 10

chmod +x /usr/local/bin/collectd-heartbeat.sh
chown collectd:collectd /usr/local/bin/collectd-heartbeat.sh

Option B: Use a cron job (simpler, no exec plugin required)

# crontab -e (for root or collectd user)
* * * * * pgrep -x collectd > /dev/null && curl -fsS https://vigilmon.online/heartbeat/YOUR_TOKEN

The pgrep -x collectd check ensures the heartbeat only pings when the Collectd process is running. If Collectd crashes, pgrep fails, curl is never called, and Vigilmon alerts after the expected interval.

Option A (exec plugin) is preferred because it confirms the Collectd daemon's internal plugin loop is running, not just that the process exists. A zombie Collectd process would pass the pgrep check but fail the exec plugin check.

Step 7: Verify the Collectd unixsock Plugin (Optional)

If Collectd is configured with the unixsock plugin, you can verify the daemon is accepting values by writing a test metric:

LoadPlugin unixsock

<Plugin unixsock>
  SocketFile "/var/run/collectd.sock"
  SocketPerms "0660"
</Plugin>

Test the socket:

# Write a test metric value and check for acknowledgment
echo "PUTVAL $(hostname)/test/gauge-vigilmon $(date +%s):42" | \
  nc -U /var/run/collectd.sock

Expected response: 0 Success: 1 value has been dispatched.

A failed response or timeout means Collectd's plugin loop is stuck even though the process is running. Incorporate this check into your heartbeat script for deeper daemon health verification.

Step 8: Configure Alert Channels

1. Go to Alert Channels in Vigilmon and add Slack, email, or PagerDuty.
2. Set Consecutive failures before alert to 1 on the TCP network plugin monitor — Collectd is a daemon and should always be listening.
3. Set Consecutive failures before alert to 2 on the write_http destination monitor — the destination may have brief maintenance windows.
4. Set Consecutive failures before alert to 3 on the heartbeat monitor (if using the 30-second interval) — allow multiple intervals before alerting on transient delays.

For environments with many Collectd nodes forwarding to a central aggregator, monitor the aggregator's network plugin port and write_http destination. Individual node failures show up as gaps in the aggregated metrics rather than direct endpoint failures.

Summary

| Monitor | Target | What It Catches | |---|---|---| | TCP probe | :25826 | Collectd daemon down, network plugin stopped | | HTTP (write_http destination) | InfluxDB/Graphite health endpoint | Collectd metrics destination unavailable | | SSL certificate | HTTPS write_http destination | Expiring TLS cert causing write_http to reject connections | | Cron heartbeat | Heartbeat URL | Collectd process dead, exec plugin loop stalled | | Log-based heartbeat | write_http error pattern | Silent metric delivery failures logged by Collectd |

Collectd's lightweight design means it runs for months without intervention — which is precisely why a failed Collectd daemon goes unnoticed. With Vigilmon monitoring the network plugin port, write_http destination health, TLS certificate validity, and end-to-end delivery heartbeat, you catch a broken Collectd pipeline before days of metric gaps appear in your Graphite or InfluxDB dashboards.