tutorial

How to Monitor DeepFlow with Vigilmon (HTTP + TCP + Alerts)

DeepFlow is an open-source observability platform that uses eBPF to automatically capture network traffic, generate distributed traces, and correlate applica...

DeepFlow is an open-source observability platform that uses eBPF to automatically capture network traffic, generate distributed traces, and correlate application metrics without requiring any code instrumentation. Its server component ingests eBPF data from agents running on every node, stores the results in ClickHouse, and exposes a Grafana-integrated API for querying traces, network flows, and application metrics. When the DeepFlow server goes down, eBPF agents across the entire fleet lose their reporting target, trace data accumulates in local buffers and is eventually dropped, and the Grafana dashboards that operations teams rely on go blank. Vigilmon provides the external monitoring layer that DeepFlow cannot provide for itself: independent HTTP and TCP checks from outside your infrastructure that keep watching even when the observability platform itself has failed. You can add your first monitor for free, no credit card required.

Why DeepFlow needs external monitoring

  • Server crash halts eBPF data collection across the entire node fleet. DeepFlow agents running on every Kubernetes node maintain a persistent connection to the DeepFlow server to stream captured network flows and traces. When the server becomes unreachable, each agent enters a reconnection loop and begins buffering data locally. The local buffers are bounded — once full, captured data is dropped silently. Because the failure is in the observability layer, no alert fires inside DeepFlow, and operations teams notice only when Grafana dashboards go blank or show stale data.

  • ClickHouse storage exhaustion causes queries to return empty results. DeepFlow stores all trace and flow data in ClickHouse. When ClickHouse runs out of disk space or its merge tree compaction falls behind, write operations fail and queries return empty result sets. The DeepFlow server process itself may remain healthy and return HTTP 200 on its own health endpoint, while the underlying data layer is completely broken. An external monitor that validates query responses rather than just the process liveness catches this split state.

  • Agent reconnection storms after a restart cause cascading API overload. When the DeepFlow server restarts after an outage, all agents attempt to reconnect simultaneously. The resulting connection storm can overwhelm the server's gRPC listener, causing many agents to fail reconnection and enter an exponential backoff loop. The server appears to recover but is serving only a fraction of its agent fleet. An external HTTP monitor that checks response latency after restart detects this degraded state, which is invisible to binary up/down health checks.

  • Grafana datasource misconfiguration after upgrades silences all dashboards. DeepFlow ships with a custom Grafana plugin and datasource. After upgrades or configuration changes, the datasource connection settings can drift — wrong port, changed authentication token, or renamed query endpoint — causing every Grafana dashboard to show "No data" without generating an error visible outside the Grafana UI. An external monitor on the DeepFlow query API endpoint catches this class of misconfiguration independently of whether Grafana is working.

  • gRPC agent listener crash leaves the HTTP API healthy. DeepFlow exposes both a gRPC listener (for agents) and an HTTP API (for Grafana queries). These run as separate goroutines or services. A panic in the gRPC listener that crashes only that goroutine leaves the HTTP API healthy, so any health check on the API endpoint reports green while no agent data is being received. A TCP monitor on the gRPC port independently detects the listener failure.

What you'll need

  • A running DeepFlow server with its HTTP API accessible over HTTP or HTTPS
  • The hostname or IP for your DeepFlow server, for example https://deepflow.example.com
  • A free Vigilmon account — monitors start running in under a minute with no credit card required

Step 1: Expose DeepFlow's health endpoint

Verify your DeepFlow server is running and locate the API health endpoint before configuring Vigilmon monitors.

# Check DeepFlow server health endpoint
curl -I http://deepflow.example.com:20417/api/v1/health

# Sample output:
# HTTP/1.1 200 OK
# Content-Type: application/json

# Inspect the response body
curl -s http://deepflow.example.com:20417/api/v1/health | jq .

# Sample output:
# {
#   "status": "ok",
#   "controller": "healthy",
#   "analyzer": "healthy"
# }

# Verify the gRPC agent listener port is open
nc -zv deepflow.example.com 30035 && echo "gRPC port open" || echo "gRPC port CLOSED"

# Check the Grafana plugin query endpoint
curl -I http://deepflow.example.com:20416/v1/query/

# Sample output:
# HTTP/1.1 200 OK
# Content-Type: application/json

If DeepFlow is running in Kubernetes, verify pod and service health:

# Check DeepFlow pods
kubectl get pods -n deepflow

# Sample output:
# NAME                                READY   STATUS    RESTARTS   AGE
# deepflow-server-7d8f9b6c4-m2xk9    1/1     Running   0          7d
# deepflow-clickhouse-0               1/1     Running   0          7d
# deepflow-grafana-5b7c8d9f2-p4rn1   1/1     Running   0          7d
# deepflow-agent-ds-xxxxx             1/1     Running   0          7d  # DaemonSet on each node

# Check services
kubectl get svc -n deepflow

# Sample output:
# NAME                    TYPE           CLUSTER-IP     EXTERNAL-IP     PORT(S)
# deepflow-server         LoadBalancer   10.96.34.56    203.0.113.88    20416/TCP,20417/TCP
# deepflow-server-grpc    LoadBalancer   10.96.34.57    203.0.113.88    30035/TCP

# Test via the external IP
curl -s http://203.0.113.88:20417/api/v1/health | jq .status

Step 2: Set up HTTP monitoring in Vigilmon

Add an HTTP monitor for the DeepFlow health endpoint:

  1. Log in to vigilmon.online and click Add Monitor.
  2. Set Monitor type to HTTP.
  3. Enter the URL: https://deepflow.example.com/api/v1/health
  4. Set Check interval to 60 seconds and Expected status code to 200.
  5. Add a Keyword match for "status":"ok" to verify the response body confirms the server is healthy, not a cached stale response.
  6. Click Save Monitor — Vigilmon will run the first check within a minute.

Add a monitor for the DeepFlow Grafana query API

The Grafana query endpoint is the path that drives all operational dashboards. A failure here means all dashboards go blank even if the server health check is passing:

  1. Click Add Monitor again.
  2. Set Monitor type to HTTP.
  3. Enter the URL: https://deepflow.example.com/v1/query/
  4. Set Expected status code to 200 and Timeout to 15 seconds.
  5. Click Save Monitor.

Why external monitoring catches what internal checks miss

DeepFlow's purpose is to provide observability — but when DeepFlow itself fails, there is no observability layer left to detect and alert on the failure. eBPF agents fall back to local buffering, dashboards go dark, and operations teams may attribute the blank panels to a data rendering issue rather than a platform outage. Vigilmon polls your DeepFlow endpoints from external network nodes on a fixed schedule, completely independently of your cluster's eBPF collection, ClickHouse storage, and Grafana integration. It detects server crashes, TLS failures, DNS misconfiguration, and HTTP errors that are invisible to health checks running inside the same cluster.

Step 3: Monitor DeepFlow's TCP ports

DeepFlow exposes multiple ports for different functions. TCP monitors on these ports can isolate which component has failed when an alert fires.

Monitor the HTTP API port:

  1. In Vigilmon, click Add Monitor.
  2. Set Monitor type to TCP.
  3. Enter Host: deepflow.example.com and Port: 20417.
  4. Set Check interval to 60 seconds.
  5. Click Save Monitor.

Monitor the gRPC agent listener port:

  1. Click Add Monitor again.
  2. Set Monitor type to TCP.
  3. Enter Host: deepflow.example.com and Port: 30035.
  4. Set Check interval to 60 seconds.
  5. Click Save Monitor.

This second TCP monitor independently detects a gRPC listener crash that leaves the HTTP API healthy — a failure mode that no single HTTP health check can catch.

# Confirm which ports are exposed externally
kubectl get svc -n deepflow -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.ports[*].port}{"\n"}{end}'

# Sample output:
# deepflow-server         20416  20417
# deepflow-server-grpc    30035

Step 4: Configure alert channels

Email alerts

  1. In Vigilmon, go to Alert Channels and click Add Channel.
  2. Choose Email and enter the address for the infrastructure or SRE team (for example, sre@example.com).
  3. Assign this channel to all DeepFlow monitors created in Steps 2 and 3.
  4. Click Save — Vigilmon will send a test email to confirm delivery.

Webhook alerts

For Slack, PagerDuty, or an internal incident platform:

  1. In Alert Channels, click Add Channel and choose Webhook.
  2. Paste your endpoint URL.
  3. Vigilmon will POST the following JSON payload when a monitor transitions to down:
{
  "monitor_id": "mon_deepflow_api_health",
  "monitor_name": "DeepFlow API /api/v1/health",
  "status": "down",
  "previous_status": "up",
  "checked_at": "2026-07-12T16:44:33Z",
  "response_time_ms": null,
  "status_code": null,
  "error": "Connection timed out after 15000ms",
  "url": "https://deepflow.example.com/api/v1/health"
}
  1. Assign all DeepFlow monitors to this channel and click Save.

Runbook: kubectl diagnostics to run when an alert fires

#!/bin/bash
# DeepFlow alert runbook — run these in order when Vigilmon fires

NAMESPACE=deepflow

echo "=== Pod status ==="
kubectl get pods -n $NAMESPACE

echo "=== Server logs (last 100 lines) ==="
kubectl logs -n $NAMESPACE -l app=deepflow-server --tail=100

echo "=== Health API response ==="
curl -sv http://deepflow.example.com:20417/api/v1/health 2>&1 | tail -20

echo "=== ClickHouse status ==="
kubectl exec -n $NAMESPACE deepflow-clickhouse-0 -- \
  clickhouse-client --query "SELECT 1" 2>&1

echo "=== ClickHouse disk usage ==="
kubectl exec -n $NAMESPACE deepflow-clickhouse-0 -- \
  df -h /var/lib/clickhouse

echo "=== gRPC port connectivity ==="
nc -zv deepflow.example.com 30035 && echo "gRPC open" || echo "gRPC CLOSED"

echo "=== Agent DaemonSet status ==="
kubectl get ds deepflow-agent-ds -n $NAMESPACE

echo "=== Recent events ==="
kubectl get events -n $NAMESPACE --sort-by='.lastTimestamp' | tail -20

echo "=== Node resource pressure ==="
kubectl describe nodes | grep -A5 "Conditions:"

Step 5: Create a public status page

A Vigilmon status page gives your SRE team, operations engineers, and any teams that depend on DeepFlow's network flow and trace data a single URL to check platform health during incidents — no cluster access required.

  1. In Vigilmon, navigate to Status Pages and click New Status Page.
  2. Give it a name such as DeepFlow Observability Platform and choose a subdomain like deepflow-status.vigilmon.page.
  3. Add all monitors created in this tutorial: the /api/v1/health HTTP monitor, the Grafana query API monitor, and the TCP port monitors for ports 20417 and 30035.
  4. Optionally add a description explaining that DeepFlow provides eBPF-based distributed tracing and network flow monitoring for the infrastructure.
  5. Click Publish — the page is immediately accessible and updates in real time.

Share this URL in your SRE runbook and in the Slack channel where infrastructure alerts are posted, so engineers can check platform status during an incident without needing to log into the cluster.

Putting it all together

| Monitor | Type | What it catches | |---|---|---| | https://deepflow.example.com/api/v1/health | HTTP | Server crash, controller/analyzer failure, configuration errors | | https://deepflow.example.com/v1/query/ | HTTP | Query endpoint failure, ClickHouse connectivity loss, Grafana datasource breakage | | deepflow.example.com:20417 | TCP | HTTP API port unreachability, process crash, firewall changes | | deepflow.example.com:30035 | TCP | gRPC agent listener crash, independent of HTTP API status |

# Quick-reference diagnostic one-liners when a Vigilmon alert fires

# 1. Are all DeepFlow pods running?
kubectl get pods -n deepflow | grep -v Running

# 2. What do the server logs show?
kubectl logs -n deepflow -l app=deepflow-server --tail=50 --since=5m

# 3. Can the health endpoint be reached?
curl -sf http://deepflow.example.com:20417/api/v1/health || echo "Health FAILED"

# 4. Is ClickHouse accepting queries?
kubectl exec -n deepflow deepflow-clickhouse-0 -- \
  clickhouse-client --query "SELECT count() FROM deepflow.flow_log.l7_flow_log LIMIT 1"

# 5. Is ClickHouse disk full?
kubectl exec -n deepflow deepflow-clickhouse-0 -- df -h /var/lib/clickhouse | awk 'NR>1{print $5, $6}'

# 6. Is the gRPC agent listener open?
nc -zv deepflow.example.com 30035 && echo "gRPC port OPEN" || echo "gRPC port CLOSED"

# 7. How many agents are connected?
curl -s http://deepflow.example.com:20417/api/v1/vtap/ | jq '[.data[] | select(.enable==1)] | length'

What's next

  • SSL certificate monitoring — If your DeepFlow server uses TLS (strongly recommended in production), add a Vigilmon SSL monitor to alert you days before the certificate expires and before eBPF agents begin failing to reconnect after the next restart.
  • ClickHouse storage alerts — Add a Vigilmon HTTP monitor on the ClickHouse HTTP interface (http://deepflow-clickhouse:8123/ping) to independently verify that the storage backend is responsive. A ClickHouse failure that leaves the DeepFlow API healthy is a real failure mode; monitoring both layers gives you precise isolation during incidents.
  • Agent fleet coverage check — Create a lightweight cron job that queries the DeepFlow API for connected agent count and POSTs a heartbeat to a Vigilmon dead-man's-switch endpoint only when the count is above your minimum threshold. If agents start dropping off and the count falls below the threshold, the heartbeat stops and Vigilmon fires an alert through a channel independent of the DeepFlow stack.

Get started free at vigilmon.online — no credit card, monitors start running in under a minute.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles