tutorial

How to Monitor TDengine Health and IoT Data Ingestion with Vigilmon

TDengine process crashes or dnode failures cause complete IoT data ingestion loss. Learn how to monitor TDengine taosd health, cluster dnode status, REST API availability, ingestion throughput, taosAdapter liveness, and cache hit rates with Vigilmon.

TDengine's one-device-one-table model makes it the fastest IoT time-series database available — but when the taosd process crashes or a dnode goes offline in a cluster, your entire IoT data pipeline stops. IoT sensors keep transmitting; data is silently dropped; dashboards freeze on the last known values. Unlike a web API, a dead TDengine daemon doesn't return an HTTP error — it simply stops accepting connections.

Vigilmon gives you external visibility into TDengine through HTTP health probes, TCP port monitors, and heartbeat monitors for your IoT ingestion applications. This tutorial covers taosd daemon health, cluster dnode status, REST API and WebSocket availability, taosAdapter liveness, ingestion throughput, and cache health.

Why TDengine Needs External Monitoring

TDengine's internal monitoring (via taosKeeper, Grafana dashboards, and the SHOW DNODES statement) provides rich cluster metrics — but only when taosd itself is functioning. External monitoring with Vigilmon adds:

  • taosd crash detection via TCP port 6030 and REST API probes — the first external signal that the daemon is down
  • taosAdapter availability monitoring — taosAdapter serves Grafana, Telegraf, and HTTP clients; it can crash independently of taosd
  • REST API health checks via the /rest/sql health probe — the layer most IoT applications use
  • Dnode offline detection via automated SQL queries — one dnode offline in a multi-node cluster silently reduces redundancy
  • Ingestion pipeline heartbeat monitoring — your IoT gateway application can be healthy while taosd silently rejects writes

Step 1: TDengine's REST API Health Endpoint

TDengine's RESTful API runs through taosAdapter (or the built-in HTTP service) on port 6041:

# Health check via REST API — executes a SQL query and returns results
curl -u root:taosdata \
  -d "select server_version()" \
  http://localhost:6041/rest/sql

# Returns: {"code":0,"column_meta":...,"data":[["3.x.x.x"]],"rows":1}

# Simple TCP connectivity check for native port 6030
nc -z localhost 6030 && echo "taosd native port open"

# taosAdapter health endpoint (if using taosAdapter 3.x)
curl http://localhost:6041/-/ping

The REST API is available without a sidecar. For richer health signals — dnode status, cache hit rate, replication health — build an aggregator that queries TDengine via REST.

Step 2: Build a TDengine Health Aggregator

// tdengine-health.js
const express = require('express');
const axios = require('axios');

const app = express();

const TDENGINE_REST = process.env.TDENGINE_REST_URL || 'http://localhost:6041';
const TDENGINE_USER = process.env.TDENGINE_USER || 'root';
const TDENGINE_PASS = process.env.TDENGINE_PASS || 'taosdata';

const auth = Buffer.from(`${TDENGINE_USER}:${TDENGINE_PASS}`).toString('base64');
const headers = { Authorization: `Basic ${auth}`, 'Content-Type': 'application/json' };

async function execSQL(sql) {
  const response = await axios.post(
    `${TDENGINE_REST}/rest/sql`,
    sql,
    { headers, timeout: 10000 }
  );
  if (response.data?.code !== 0) {
    throw new Error(`TDengine SQL error: ${response.data?.desc || 'unknown'}`);
  }
  return response.data;
}

app.get('/health/tdengine', async (req, res) => {
  try {
    const result = await execSQL('SELECT SERVER_VERSION()');

    if (!result.data || result.data.length === 0) {
      return res.status(503).json({ status: 'degraded', reason: 'no_version_result' });
    }

    return res.status(200).json({
      status: 'ok',
      version: result.data[0]?.[0],
    });
  } catch (err) {
    return res.status(503).json({ status: 'down', error: err.message });
  }
});

// Cluster dnode health — detects offline dnodes
app.get('/health/tdengine/dnodes', async (req, res) => {
  try {
    const result = await execSQL('SHOW DNODES');

    if (!result.data) {
      return res.status(503).json({ status: 'down', reason: 'no_dnode_data' });
    }

    // Find column indexes for status and offline reason
    const columnMeta = result.column_meta || [];
    const statusIdx = columnMeta.findIndex(c => c[0]?.toLowerCase() === 'status');
    const endpointIdx = columnMeta.findIndex(c => c[0]?.toLowerCase() === 'endpoint');

    const offline = result.data.filter(row => {
      const status = statusIdx >= 0 ? row[statusIdx] : null;
      return status && status.toLowerCase() !== 'ready';
    });

    if (offline.length > 0) {
      return res.status(503).json({
        status: 'degraded',
        reason: 'dnode_offline',
        offline_dnodes: offline.map(row => endpointIdx >= 0 ? row[endpointIdx] : row),
      });
    }

    return res.status(200).json({
      status: 'ok',
      total_dnodes: result.data.length,
      all_ready: true,
    });
  } catch (err) {
    return res.status(503).json({ status: 'down', error: err.message });
  }
});

// Query latency — measures SELECT performance
app.get('/health/tdengine/query-latency', async (req, res) => {
  const start = Date.now();
  try {
    await execSQL('SELECT COUNT(*) FROM INFORMATION_SCHEMA.INS_TABLES LIMIT 1');

    const latencyMs = Date.now() - start;
    const threshold = parseInt(process.env.QUERY_LATENCY_THRESHOLD_MS || '2000');

    if (latencyMs > threshold) {
      return res.status(503).json({
        status: 'degraded',
        reason: 'query_latency_high',
        latency_ms: latencyMs,
        threshold_ms: threshold,
      });
    }

    return res.status(200).json({ status: 'ok', query_latency_ms: latencyMs });
  } catch (err) {
    return res.status(503).json({ status: 'down', error: err.message });
  }
});

// Database disk usage — detect disk approaching capacity
app.get('/health/tdengine/disk', async (req, res) => {
  try {
    const result = await execSQL('SHOW DNODES');

    if (!result.data) {
      return res.status(200).json({ status: 'skipped', reason: 'no_dnode_data' });
    }

    const columnMeta = result.column_meta || [];
    const diskUsedIdx = columnMeta.findIndex(c => c[0]?.toLowerCase().includes('disk_used'));
    const diskTotalIdx = columnMeta.findIndex(c => c[0]?.toLowerCase().includes('disk_total'));

    if (diskUsedIdx < 0 || diskTotalIdx < 0) {
      return res.status(200).json({ status: 'skipped', reason: 'disk_columns_not_found' });
    }

    const threshold = parseInt(process.env.DISK_THRESHOLD_PCT || '80');
    const highDisk = result.data.filter(row => {
      const used = parseFloat(row[diskUsedIdx]) || 0;
      const total = parseFloat(row[diskTotalIdx]) || 1;
      return (used / total) * 100 > threshold;
    });

    if (highDisk.length > 0) {
      return res.status(503).json({
        status: 'degraded',
        reason: 'disk_usage_high',
        threshold_pct: threshold,
      });
    }

    return res.status(200).json({ status: 'ok', disk_threshold_pct: threshold });
  } catch (err) {
    return res.status(503).json({ status: 'down', error: err.message });
  }
});

// taosAdapter health — separate from taosd; can crash independently
app.get('/health/tdengine/taosadapter', async (req, res) => {
  const TAOSADAPTER_URL = process.env.TAOSADAPTER_URL || 'http://localhost:6041';
  try {
    const response = await axios.get(`${TAOSADAPTER_URL}/-/ping`, { timeout: 5000 });
    if (response.status === 200) {
      return res.status(200).json({ status: 'ok' });
    }
    return res.status(503).json({ status: 'degraded', http_status: response.status });
  } catch (err) {
    return res.status(503).json({ status: 'down', error: err.message });
  }
});

app.listen(3013, () => console.log('TDengine health aggregator listening on :3013'));

Python Alternative

# tdengine_health.py
from flask import Flask, jsonify
import requests, os, time, base64

app = Flask(__name__)

TDENGINE_REST = os.environ.get('TDENGINE_REST_URL', 'http://localhost:6041')
USER = os.environ.get('TDENGINE_USER', 'root')
PASS = os.environ.get('TDENGINE_PASS', 'taosdata')
AUTH = base64.b64encode(f'{USER}:{PASS}'.encode()).decode()
HEADERS = {'Authorization': f'Basic {AUTH}', 'Content-Type': 'application/json'}

def exec_sql(sql):
    r = requests.post(f'{TDENGINE_REST}/rest/sql', data=sql, headers=HEADERS, timeout=10)
    r.raise_for_status()
    data = r.json()
    if data.get('code') != 0:
        raise RuntimeError(f"TDengine error: {data.get('desc')}")
    return data

@app.get('/health/tdengine')
def health():
    try:
        result = exec_sql('SELECT SERVER_VERSION()')
        version = result['data'][0][0] if result.get('data') else None
        return jsonify(status='ok', version=version), 200
    except Exception as e:
        return jsonify(status='down', error=str(e)), 503

@app.get('/health/tdengine/dnodes')
def dnode_health():
    try:
        result = exec_sql('SHOW DNODES')
        return jsonify(status='ok', total_dnodes=len(result.get('data', []))), 200
    except Exception as e:
        return jsonify(status='down', error=str(e)), 503

if __name__ == '__main__':
    app.run(port=3013)

Step 3: Configure Vigilmon Monitors for TDengine

HTTP Monitor — taosd REST API Health

  1. Log in to vigilmon.online and go to Monitors → New Monitor
  2. Choose HTTP / HTTPS
  3. URL: https://your-tdengine-host.example.com/health/tdengine
  4. Interval: 1 minute
  5. Expected response:
    • Status code: 200
    • Response body contains: "status":"ok"
    • Response time threshold: 10000ms
  6. Alert channel: P1 (taosd down = zero IoT data ingestion)
  7. Save

TCP Port Monitor — Native taosd Port (6030)

The native connection port is used by taosc clients and connectors. A TCP probe on 6030 catches taosd crashes even if the REST service is not running:

  1. Create a new monitor, type TCP Port
  2. Host: your-tdengine-host.example.com, Port: 6030
  3. Interval: 1 minute
  4. Alert channel: P1

HTTP Monitor — taosAdapter Ping

taosAdapter is a separate process from taosd. It serves Grafana dashboards, Telegraf, and HTTP client integrations. It can crash while taosd is healthy, silently breaking all dashboard integrations:

  1. Create a new monitor, type HTTP / HTTPS
  2. URL: http://your-tdengine-host.example.com:6041/-/ping (or via sidecar: https://your-tdengine-host.example.com/health/tdengine/taosadapter)
  3. Interval: 1 minute
  4. Expected response:
    • Status code: 200
  5. Alert channel: P1 (taosAdapter down = Grafana dashboards broken)

HTTP Monitor — Cluster Dnode Health

  • URL: https://your-tdengine-host.example.com/health/tdengine/dnodes
  • Expected: 200, body contains "all_ready":true
  • Interval: 2 minutes
  • Alert channel: P1 in clustered deployments (dnode offline reduces redundancy immediately)

HTTP Monitor — Query Latency

  • URL: https://your-tdengine-host.example.com/health/tdengine/query-latency
  • Expected: 200, body contains "status":"ok"
  • Response time threshold: 5000ms
  • Interval: 2 minutes
  • Alert channel: P2

HTTP Monitor — Disk Usage

  • URL: https://your-tdengine-host.example.com/health/tdengine/disk
  • Expected: 200, body contains "status":"ok"
  • Interval: 10 minutes
  • Alert channel: P2

Step 4: Heartbeat Monitoring for IoT Ingestion Applications

TDengine can be healthy while your IoT gateway application is stuck in a write retry loop or silently buffering data due to a network partition. Wire a Vigilmon heartbeat into your ingestion application:

Set Up the Heartbeat Monitor

  1. In Vigilmon, go to Monitors → New Monitor → Heartbeat
  2. Name: tdengine-iot-gateway
  3. Expected interval: 5 minutes (adjust based on your device reporting rate)
  4. Grace period: 10 minutes
  5. Save — copy the heartbeat URL

Node.js IoT Gateway with Heartbeat

// iot-gateway.js
const TaosRestClient = require('./taos-rest-client'); // your REST wrapper
const axios = require('axios');

const HEARTBEAT_URL = process.env.VIGILMON_HEARTBEAT_URL;
const client = new TaosRestClient({
  host: process.env.TDENGINE_HOST || 'localhost',
  port: 6041,
  user: process.env.TDENGINE_USER || 'root',
  password: process.env.TDENGINE_PASS || 'taosdata',
});

let rowsInsertedSinceHeartbeat = 0;
const HEARTBEAT_EVERY_N_ROWS = 1000;

async function insertDeviceData(deviceId, timestamp, value) {
  await client.query(
    `INSERT INTO iot_db.device_${deviceId} VALUES (${timestamp}, ${value})`
  );
  rowsInsertedSinceHeartbeat++;

  if (rowsInsertedSinceHeartbeat >= HEARTBEAT_EVERY_N_ROWS) {
    await axios.get(HEARTBEAT_URL).catch(() => {});
    rowsInsertedSinceHeartbeat = 0;
  }
}

// For low-throughput scenarios: time-based heartbeat
setInterval(async () => {
  if (rowsInsertedSinceHeartbeat > 0) {
    await axios.get(HEARTBEAT_URL).catch(() => {});
    rowsInsertedSinceHeartbeat = 0;
  }
}, 60 * 1000);

Python IoT Gateway with Heartbeat (REST API)

# iot_gateway.py
import requests, os, time, threading, base64

TDENGINE_HOST = os.environ.get('TDENGINE_HOST', 'localhost')
TDENGINE_REST = f'http://{TDENGINE_HOST}:6041'
AUTH = base64.b64encode(
    f"{os.environ.get('TDENGINE_USER', 'root')}:{os.environ.get('TDENGINE_PASS', 'taosdata')}".encode()
).decode()
HEADERS = {'Authorization': f'Basic {AUTH}', 'Content-Type': 'application/json'}
HEARTBEAT_URL = os.environ['VIGILMON_HEARTBEAT_URL']

rows_since_heartbeat = 0
last_heartbeat = time.time()

def insert_row(device_id, ts_ms, value):
    global rows_since_heartbeat, last_heartbeat
    sql = f"INSERT INTO iot_db.device_{device_id} VALUES ({ts_ms}, {value})"
    r = requests.post(f'{TDENGINE_REST}/rest/sql', data=sql, headers=HEADERS, timeout=10)
    r.raise_for_status()
    rows_since_heartbeat += 1

def heartbeat_worker():
    global rows_since_heartbeat, last_heartbeat
    while True:
        time.sleep(60)
        if rows_since_heartbeat > 0:
            try:
                requests.get(HEARTBEAT_URL, timeout=5)
                rows_since_heartbeat = 0
                last_heartbeat = time.time()
            except Exception:
                pass

threading.Thread(target=heartbeat_worker, daemon=True).start()

Step 5: Alert Routing for TDengine IoT Deployments

IoT data pipelines are often ingesting from thousands of devices simultaneously. A taosd crash or taosAdapter failure affects every device at once — severity-tiered routing helps focus the response:

| Monitor | Alert Channel | Priority | |---|---|---| | taosd REST health /health/tdengine | Slack + PagerDuty | P1 | | Native port 6030 (TCP) | Slack + PagerDuty | P1 | | taosAdapter ping /-/ping | Slack + PagerDuty | P1 | | Dnode health /health/tdengine/dnodes | Slack + PagerDuty | P1 | | Query latency /health/tdengine/query-latency | Slack | P2 | | Disk usage /health/tdengine/disk | Slack | P2 | | Heartbeat: IoT gateway application | Slack + email | P2 |

Configure response time thresholds for early warning:

  • Alert at 5000ms for the REST health endpoint (slow queries signal taosd CPU saturation)
  • Alert at 3000ms for taosAdapter ping (taosAdapter slowness precedes full failure)

For high-throughput IoT deployments, add a second heartbeat monitor per critical ingestion path (one per device class or data source) rather than a single combined monitor — this lets you pinpoint exactly which device category stopped reporting.

Summary

TDengine's high-throughput IoT architecture delivers exceptional ingestion performance — but when taosd crashes, a dnode goes offline, or taosAdapter fails, the impact is immediate and total. External monitoring with Vigilmon catches these failures before they become extended IoT data gaps:

| Monitor Type | What It Covers | |---|---| | HTTP monitor on /rest/sql or health endpoint | taosd daemon health via REST API | | TCP port monitor on 6030 | Native connection port availability | | HTTP monitor on /-/ping | taosAdapter health (Grafana, Telegraf, HTTP clients) | | HTTP monitor on dnode endpoint | Cluster dnode online status | | HTTP monitor on query latency | SQL query response time | | Heartbeat monitor | IoT gateway ingestion pipeline liveness |

Get started free at vigilmon.online — your first TDengine monitor is running in under two minutes.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles