How to Monitor MindsDB Server Health and ML Model Jobs with Vigilmon

MindsDB is the SQL interface to machine learning — but when the server goes down or a training job stalls, the failure is invisible to the end user. BI tools silently return no predictions; scheduled JOBS miss their window; dashboards show stale data. Unlike a web application that returns a 500 error, a dead MindsDB instance just stops answering MySQL protocol connections.

Vigilmon gives you external visibility into MindsDB through HTTP health probes, TCP port monitors, and heartbeat monitors for your scheduled prediction jobs. This tutorial covers MindsDB server health, MySQL and REST API availability, model training status, and data source connectivity.

Why MindsDB Needs External Monitoring

MindsDB's internal dashboard provides model accuracy metrics and job history — but only for users actively watching. External monitoring with Vigilmon adds:

• Server crash detection via the REST API health endpoint — /api/status returning non-200 means zero ML predictions
• MySQL protocol monitoring — port 47335 going dark blocks all SQL-based BI tool access
• Training job stall detection via heartbeat — a training job stuck for 2× expected duration silently delays model updates
• Data source connectivity alerts — upstream database loss prevents both training and real-time prediction
• Proactive TLS and certificate monitoring for secured deployments

Step 1: MindsDB Built-In Health Endpoint

MindsDB exposes a REST API on port 47334 with a built-in health check:

# Health check — returns {"status": "ok"} when server is ready
curl http://localhost:47334/api/status

# List all models (checks server + database connectivity)
curl http://localhost:47334/api/models

# List data source integrations
curl http://localhost:47334/api/databases

For deployments with authentication:

curl -H "Authorization: Bearer YOUR_API_TOKEN" http://localhost:47334/api/status

You can point Vigilmon's HTTP monitor directly at the /api/status endpoint without a sidecar. However, for richer health signals — training job status, prediction latency, data source health — build a lightweight aggregator.

Step 2: Build a MindsDB Health Aggregator

// mindsdb-health.js
const express = require('express');
const axios = require('axios');

const app = express();

const MINDSDB_API = process.env.MINDSDB_API_URL || 'http://localhost:47334';
const API_TOKEN = process.env.MINDSDB_API_TOKEN || '';

const headers = API_TOKEN ? { Authorization: `Bearer ${API_TOKEN}` } : {};

app.get('/health/mindsdb', async (req, res) => {
  try {
    const response = await axios.get(`${MINDSDB_API}/api/status`, {
      headers,
      timeout: 10000,
    });

    if (response.data?.status !== 'ok') {
      return res.status(503).json({
        status: 'degraded',
        reason: 'api_status_not_ok',
        upstream: response.data,
      });
    }

    return res.status(200).json({ status: 'ok', mindsdb: response.data });
  } catch (err) {
    return res.status(503).json({ status: 'down', error: err.message });
  }
});

// Training job health — detect stuck training
app.get('/health/mindsdb/training', async (req, res) => {
  try {
    const response = await axios.get(`${MINDSDB_API}/api/models`, { headers, timeout: 10000 });
    const models = response.data?.data || response.data || [];

    const training = models.filter(m => m.status === 'training' || m.status === 'generating');
    const stuckThresholdMs = parseInt(process.env.TRAINING_STUCK_THRESHOLD_MS || String(2 * 60 * 60 * 1000)); // 2h default

    const stuck = training.filter(m => {
      if (!m.training_time) return false;
      return m.training_time > stuckThresholdMs;
    });

    if (stuck.length > 0) {
      return res.status(503).json({
        status: 'degraded',
        reason: 'training_job_stuck',
        stuck_models: stuck.map(m => ({ name: m.name, training_time_ms: m.training_time })),
      });
    }

    return res.status(200).json({
      status: 'ok',
      total_models: models.length,
      training_count: training.length,
    });
  } catch (err) {
    return res.status(503).json({ status: 'down', error: err.message });
  }
});

// Data source connectivity health
app.get('/health/mindsdb/datasources', async (req, res) => {
  try {
    const response = await axios.get(`${MINDSDB_API}/api/databases`, { headers, timeout: 10000 });
    const databases = response.data?.data || response.data || [];

    // Filter non-internal sources (MindsDB, files, etc.)
    const externalSources = databases.filter(db => !['mindsdb', 'information_schema'].includes(db.name?.toLowerCase()));

    const disconnected = externalSources.filter(db => db.connection_status !== 'connected' && db.connection_status !== undefined);

    if (disconnected.length > 0) {
      return res.status(503).json({
        status: 'degraded',
        reason: 'datasource_disconnected',
        disconnected: disconnected.map(db => db.name),
      });
    }

    return res.status(200).json({
      status: 'ok',
      data_sources: externalSources.length,
      all_connected: true,
    });
  } catch (err) {
    return res.status(503).json({ status: 'down', error: err.message });
  }
});

// Prediction latency endpoint — executes a test prediction and measures response time
app.get('/health/mindsdb/prediction-latency', async (req, res) => {
  const testModel = process.env.MINDSDB_TEST_MODEL;
  if (!testModel) {
    return res.status(200).json({ status: 'skipped', reason: 'MINDSDB_TEST_MODEL not set' });
  }

  const start = Date.now();
  try {
    // Execute a simple prediction via the REST query endpoint
    await axios.post(
      `${MINDSDB_API}/api/sql/query`,
      { query: `SELECT * FROM mindsdb.${testModel} LIMIT 1` },
      { headers, timeout: 15000 }
    );
    const latencyMs = Date.now() - start;
    const threshold = parseInt(process.env.PREDICTION_LATENCY_THRESHOLD_MS || '5000');

    if (latencyMs > threshold) {
      return res.status(503).json({
        status: 'degraded',
        reason: 'prediction_latency_high',
        latency_ms: latencyMs,
        threshold_ms: threshold,
      });
    }

    return res.status(200).json({ status: 'ok', latency_ms: latencyMs });
  } catch (err) {
    return res.status(503).json({ status: 'down', error: err.message });
  }
});

// Disk usage for model storage
app.get('/health/mindsdb/disk', async (req, res) => {
  const { execSync } = require('child_process');
  const modelPath = process.env.MINDSDB_STORAGE_PATH || '/root/mindsdb_storage';

  try {
    const output = execSync(`df -k "${modelPath}" | tail -1`).toString().trim();
    const parts = output.split(/\s+/);
    const usePct = parseInt(parts[4]?.replace('%', '') || '0');

    const threshold = parseInt(process.env.DISK_THRESHOLD_PCT || '75');
    if (usePct > threshold) {
      return res.status(503).json({
        status: 'degraded',
        reason: 'disk_usage_high',
        disk_pct: usePct,
        threshold_pct: threshold,
        path: modelPath,
      });
    }

    return res.status(200).json({ status: 'ok', disk_pct: usePct, path: modelPath });
  } catch (err) {
    return res.status(503).json({ status: 'down', error: err.message });
  }
});

app.listen(3011, () => console.log('MindsDB health aggregator listening on :3011'));

Python Alternative

# mindsdb_health.py
from flask import Flask, jsonify
import requests, os, time

app = Flask(__name__)

MINDSDB_API = os.environ.get('MINDSDB_API_URL', 'http://localhost:47334')
TOKEN = os.environ.get('MINDSDB_API_TOKEN', '')
HEADERS = {'Authorization': f'Bearer {TOKEN}'} if TOKEN else {}

@app.get('/health/mindsdb')
def health():
    try:
        r = requests.get(f'{MINDSDB_API}/api/status', headers=HEADERS, timeout=10)
        r.raise_for_status()
        data = r.json()
        if data.get('status') != 'ok':
            return jsonify(status='degraded', upstream=data), 503
        return jsonify(status='ok', mindsdb=data), 200
    except Exception as e:
        return jsonify(status='down', error=str(e)), 503

if __name__ == '__main__':
    app.run(port=3011)

Step 3: Configure Vigilmon Monitors for MindsDB

HTTP Monitor — Server Health (Port 47334)

1. Log in to vigilmon.online and go to Monitors → New Monitor
2. Choose HTTP / HTTPS
3. URL: https://your-mindsdb-host.example.com/health/mindsdb (or directly: http://your-mindsdb-host.example.com:47334/api/status)
4. Interval: 1 minute
5. Expected response:
   • Status code: 200
   • Response body contains: "status":"ok"
   • Response time threshold: 10000ms (server startup and model loading can be slow)
6. Alert channel: P1 (server down = zero ML predictions)
7. Save

TCP Port Monitor — MySQL Protocol (Port 47335)

BI tools like Metabase, Tableau, and DBeaver connect to MindsDB on port 47335. A TCP probe ensures the MySQL-compatible listener is available even when the REST API is not responding:

1. Create a new monitor, type TCP Port
2. Host: your-mindsdb-host.example.com, Port: 47335
3. Interval: 1 minute
4. Alert channel: P1 (MySQL port down = all SQL-based BI access blocked)

HTTP Monitor — Training Job Health

• URL: https://your-mindsdb-host.example.com/health/mindsdb/training
• Expected: 200, body contains "status":"ok"
• Interval: 5 minutes (training runs for hours; checking every minute is excessive)
• Alert channel: P2 Slack channel for ML ops

HTTP Monitor — Data Source Connectivity

• URL: https://your-mindsdb-host.example.com/health/mindsdb/datasources
• Expected: 200, body contains "all_connected":true
• Interval: 2 minutes
• Alert channel: P1 (data source loss blocks both training and predictions)

HTTP Monitor — Prediction Latency

• URL: https://your-mindsdb-host.example.com/health/mindsdb/prediction-latency
• Expected: 200, body contains "status":"ok"
• Response time threshold: 15000ms
• Interval: 5 minutes
• Alert channel: P2

HTTP Monitor — Disk Usage

• URL: https://your-mindsdb-host.example.com/health/mindsdb/disk
• Expected: 200, body contains "status":"ok"
• Interval: 10 minutes
• Alert channel: P2

Step 4: Heartbeat Monitoring for MindsDB JOBs

MindsDB JOBS run on a schedule to retrain models or execute batch predictions. A job that misses its scheduled window silently lets model accuracy degrade. Wire a Vigilmon heartbeat into your MindsDB job:

Set Up the Heartbeat Monitor

1. In Vigilmon, go to Monitors → New Monitor → Heartbeat
2. Name: mindsdb-retraining-job
3. Expected interval: match your job schedule (e.g., 60 minutes for hourly retraining)
4. Grace period: 15 minutes
5. Save — copy the heartbeat URL (e.g., https://vigilmon.online/heartbeat/abc123xyz)

Trigger Heartbeat from MindsDB JOB via a Python Wrapper

MindsDB JOBS execute SQL. Use a Python wrapper that triggers the job and sends the heartbeat:

# run_mindsdb_job.py
import mysql.connector
import requests
import os
import sys

MINDSDB_HOST = os.environ.get('MINDSDB_HOST', 'localhost')
MINDSDB_PORT = int(os.environ.get('MINDSDB_PORT', 47335))
HEARTBEAT_URL = os.environ['VIGILMON_HEARTBEAT_URL']
JOB_SQL = os.environ.get('MINDSDB_JOB_SQL', 'RETRAIN mindsdb.my_model')

try:
    conn = mysql.connector.connect(
        host=MINDSDB_HOST,
        port=MINDSDB_PORT,
        user='mindsdb',
        password='',
        database='mindsdb',
    )
    cursor = conn.cursor()
    cursor.execute(JOB_SQL)
    conn.commit()
    cursor.close()
    conn.close()

    # Job succeeded — signal Vigilmon
    requests.get(HEARTBEAT_URL, timeout=10)
    print('MindsDB job completed and heartbeat sent.')
except Exception as e:
    print(f'MindsDB job failed: {e}', file=sys.stderr)
    sys.exit(1)

Schedule via cron:

# /etc/cron.d/mindsdb-job
0 * * * * mindsdb python3 /opt/mindsdb/run_mindsdb_job.py >> /var/log/mindsdb-job.log 2>&1

Step 5: Alert Routing for MindsDB

| Monitor | Alert Channel | Priority | |---|---|---| | Server health /health/mindsdb | Slack + PagerDuty | P1 | | MySQL port 47335 (TCP) | Slack + PagerDuty | P1 | | Data sources /health/mindsdb/datasources | Slack + PagerDuty | P1 | | Training jobs /health/mindsdb/training | Slack | P2 | | Prediction latency /health/mindsdb/prediction-latency | Slack | P2 | | Disk usage /health/mindsdb/disk | Slack | P2 | | Heartbeat: retraining job | Slack + email | P2 |

Configure response time thresholds for early warning:

• Alert at 8000ms for the main health endpoint (slow responses precede full server hangs)
• Alert at 10000ms for prediction latency (P95 degradation signals model serving pressure before full failure)

Summary

MindsDB's SQL interface to machine learning abstracts away pipeline complexity — but it also hides failures. A crashed server, a stalled training job, or a disconnected upstream database silently eliminates ML predictions from your BI tools. External monitoring with Vigilmon catches these failures before end users notice:

| Monitor Type | What It Covers | |---|---| | HTTP monitor on /api/status | MindsDB server process health | | TCP port monitor on 47335 | MySQL protocol availability for BI tools | | HTTP monitor on /health/mindsdb/training | Stuck model training jobs | | HTTP monitor on /health/mindsdb/datasources | Upstream database connectivity | | HTTP monitor on /health/mindsdb/prediction-latency | Model serving response time | | Heartbeat monitor | Scheduled JOBS execution liveness |

Get started free at vigilmon.online — your first MindsDB monitor is running in under two minutes.