Monitoring Your Self-Hosted Rasa Conversational AI with Vigilmon

Your Rasa chatbot stopped responding. The action server crashed. The NLU model is returning low-confidence scores. Your support bot is telling every user "I didn't understand that."

Rasa is a powerful open-source framework for building contextual AI assistants — but self-hosting it means owning the reliability of a multi-service architecture: API server on port 5005, action server on port 5055, PostgreSQL or SQLite tracker store, Redis or MongoDB lock store, and model files that need to be present and loaded correctly.

By the end of this guide you'll have:

• API server and action server availability monitoring
• NLU inference endpoint response time monitoring
• Dialogue management health checks
• Tracker store and lock store connectivity monitoring
• Webhook endpoint monitoring
• Model version management monitoring
• Heartbeat monitoring for the training pipeline
• Slack alerts and a public status page

Why monitoring matters for Rasa

Rasa's multi-service architecture has many independent failure points:

• Action server crash — custom actions silently fail; the bot falls back to default responses or loops
• Tracker store unavailability — Rasa loses conversation context; users have to repeat themselves
• Lock store failure — concurrent conversations can corrupt each other's state
• Model loading failure — Rasa starts but all NLU predictions return 0% confidence
• NLU inference degradation — slow inference (>1s) makes the bot feel broken even when technically "up"
• Training pipeline crash — model updates stop being published; the bot becomes stale

Step 1: Verify Rasa's built-in health endpoints

Rasa ships with several health-related endpoints:

# API server health
curl http://localhost:5005/
# Returns: {"version":"3.x.x","minimum_compatible_version":"...","current_moodels":...}

# Model status
curl http://localhost:5005/status
# Returns model name, loaded status, and training status

# NLU parse (inference test)
curl -X POST http://localhost:5005/model/parse \
  -H 'Content-Type: application/json' \
  -d '{"text": "hello"}'

The action server has its own health check:

curl http://localhost:5055/health
# Returns: {"status":"ok"}

These are your primary monitoring targets.

Step 2: Create a composite health check

Create a Python health sidecar that aggregates all critical checks:

#!/usr/bin/env python3
# rasa-health.py
import json
import os
import time
import urllib.request
import urllib.error
import socket
from http.server import HTTPServer, BaseHTTPRequestHandler

RASA_URL = os.environ.get('RASA_URL', 'http://localhost:5005')
ACTION_URL = os.environ.get('RASA_ACTION_URL', 'http://localhost:5055')
DB_HOST = os.environ.get('DB_HOST', 'localhost')
DB_PORT = int(os.environ.get('DB_PORT', '5432'))
REDIS_HOST = os.environ.get('REDIS_HOST', 'localhost')
REDIS_PORT = int(os.environ.get('REDIS_PORT', '6379'))

def http_get(url, timeout=5):
    try:
        req = urllib.request.Request(url)
        with urllib.request.urlopen(req, timeout=timeout) as resp:
            return resp.status, resp.read().decode()
    except urllib.error.HTTPError as e:
        return e.code, ''
    except Exception:
        return 0, ''

def check_rasa_api():
    status, body = http_get(f'{RASA_URL}/')
    if status != 200:
        return 'down'
    try:
        data = json.loads(body)
        return 'up' if 'version' in data else 'degraded'
    except Exception:
        return 'degraded'

def check_rasa_model():
    status, body = http_get(f'{RASA_URL}/status')
    if status != 200:
        return 'down'
    try:
        data = json.loads(body)
        model = data.get('model_file') or data.get('num_active_training_jobs')
        return 'up' if model is not None else 'no_model'
    except Exception:
        return 'degraded'

def check_nlu_inference():
    try:
        payload = json.dumps({'text': 'hello'}).encode()
        req = urllib.request.Request(
            f'{RASA_URL}/model/parse',
            data=payload,
            headers={'Content-Type': 'application/json'},
            method='POST'
        )
        start = time.monotonic()
        with urllib.request.urlopen(req, timeout=10) as resp:
            latency_ms = int((time.monotonic() - start) * 1000)
            if resp.status != 200:
                return 'down', 0
            data = json.loads(resp.read().decode())
            intent = data.get('intent', {})
            return 'up' if intent else 'degraded', latency_ms
    except Exception:
        return 'down', 0

def check_action_server():
    status, body = http_get(f'{ACTION_URL}/health')
    if status != 200:
        return 'down'
    try:
        data = json.loads(body)
        return 'up' if data.get('status') == 'ok' else 'degraded'
    except Exception:
        return 'degraded'

def check_tcp(host, port):
    try:
        s = socket.create_connection((host, port), timeout=3)
        s.close()
        return 'up'
    except Exception:
        return 'down'

class HealthHandler(BaseHTTPRequestHandler):
    def do_GET(self):
        if self.path != '/health':
            self.send_response(404)
            self.end_headers()
            return

        nlu_status, nlu_latency_ms = check_nlu_inference()

        checks = {
            'rasa_api': check_rasa_api(),
            'rasa_model': check_rasa_model(),
            'nlu_inference': nlu_status,
            'nlu_latency_ms': nlu_latency_ms,
            'action_server': check_action_server(),
            'tracker_store_db': check_tcp(DB_HOST, DB_PORT),
            'lock_store_redis': check_tcp(REDIS_HOST, REDIS_PORT),
        }

        healthy = all(
            v in ('up',) or isinstance(v, int)
            for v in checks.values()
        ) and checks['rasa_api'] == 'up' and checks['action_server'] == 'up'

        body = json.dumps({'status': 'ok' if healthy else 'degraded', 'checks': checks})
        self.send_response(200 if healthy else 503)
        self.send_header('Content-Type', 'application/json')
        self.end_headers()
        self.wfile.write(body.encode())

    def log_message(self, fmt, *args):
        pass

if __name__ == '__main__':
    server = HTTPServer(('0.0.0.0', 5056), HealthHandler)
    print('Rasa health on :5056')
    server.serve_forever()

Run it:

python3 rasa-health.py &
# Or as a systemd service

Verify:

curl http://localhost:5056/health
# {"status":"ok","checks":{"rasa_api":"up","rasa_model":"up","nlu_inference":"up",
#   "nlu_latency_ms":145,"action_server":"up","tracker_store_db":"up","lock_store_redis":"up"}}

Step 3: Set up monitoring in Vigilmon

Sign up at vigilmon.online (free, no card required).

Rasa API server:

1. Click New Monitor → HTTP
2. URL: https://rasa.yourdomain.com/
3. Expected status: 200
4. Keyword check: version
5. Check interval: 1 minute

Action server:

1. New Monitor → HTTP
2. URL: https://rasa.yourdomain.com:5055/health
3. Expected status: 200
4. Keyword check: "status":"ok"
5. Check interval: 1 minute

Composite health:

1. New Monitor → HTTP
2. URL: https://rasa.yourdomain.com:5056/health
3. Expected status: 200
4. Keyword check: "status":"ok"
5. Check interval: 1 minute

Step 4: Monitor NLU inference latency

NLU inference latency is a UX metric, not just a reliability metric. A Rasa instance that responds in 3 seconds feels broken to users even if it's technically "up."

In Vigilmon, set a response time alert on the NLU parse endpoint:

1. New Monitor → HTTP
2. URL: https://rasa.yourdomain.com/model/parse
3. Method: POST
4. Body: {"text":"hello"}
5. Headers: Content-Type: application/json
6. Expected status: 200
7. Response time alert: 1000ms — alert if NLU takes over 1 second

Also use the nlu_latency_ms field in the composite health check JSON to track trends. Set a keyword alert for "nlu_inference":"down" or "rasa_model":"no_model".

Step 5: Monitor webhook endpoints

Rasa webhook endpoints receive messages from external channels (Slack, Telegram, Facebook Messenger, custom REST). If these endpoints are down, messages from those channels stop being received.

| Channel | Endpoint | |---------|----------| | REST channel | POST /webhooks/rest/webhook | | Slack | POST /webhooks/slack/webhook | | Telegram | POST /webhooks/telegram/webhook | | Custom connector | POST /webhooks/{custom}/webhook |

Monitor each active webhook with a GET request expecting 405 Method Not Allowed (the route exists but only accepts POST):

1. New Monitor → HTTP
2. URL: https://rasa.yourdomain.com/webhooks/rest/webhook
3. Expected status: 405
4. Check interval: 5 minutes

A 404 means the webhook channel failed to load and isn't registered.

Step 6: Monitor tracker store and lock store connectivity

The tracker store (PostgreSQL or SQLite) persists conversation history. The lock store (Redis or MongoDB) prevents concurrent message processing from corrupting state.

The composite health check already probes these via TCP. For deeper checks, add database-level probes:

PostgreSQL tracker store:

import psycopg2

def check_postgres_tracker():
    db_url = os.environ.get('TRACKER_STORE_URL', '')
    if not db_url:
        return 'skip'
    try:
        conn = psycopg2.connect(db_url, connect_timeout=3)
        cur = conn.cursor()
        cur.execute('SELECT 1')
        conn.close()
        return 'up'
    except Exception:
        return 'down'

Redis lock store:

import redis

def check_redis_lock_store():
    redis_url = os.environ.get('LOCK_STORE_URL', '')
    if not redis_url:
        return 'skip'
    try:
        r = redis.from_url(redis_url, socket_connect_timeout=3)
        r.ping()
        return 'up'
    except Exception:
        return 'down'

Add these to the health handler's checks dict.

Step 7: Heartbeat monitoring for the training pipeline

Rasa model training is a batch process. If training stops running, the model becomes stale — intents that users start using won't be recognized until the next successful training run.

Wrap your training command with a heartbeat:

#!/bin/bash
# /opt/rasa/train-and-ping.sh
set -e

cd /opt/rasa
rasa train --domain domain.yml --data data/ --out models/

# Copy new model to the serving path
cp models/$(ls -t models/ | head -1) /opt/rasa/serving/model.tar.gz

# Signal Rasa to reload the model
curl -X POST http://localhost:5005/model \
  -H 'Content-Type: application/json' \
  -d "{\"model_file\": \"/opt/rasa/serving/model.tar.gz\"}"

# Ping heartbeat on success
curl -fsS "$VIGILMON_TRAINING_HEARTBEAT"

Run it on a schedule:

# /etc/cron.d/rasa-training
0 3 * * * /opt/rasa/train-and-ping.sh >> /var/log/rasa-training.log 2>&1

In Vigilmon:

1. New Monitor → Heartbeat
2. Expected interval: 24 hours (adjust to your training schedule)
3. Copy the ping URL to VIGILMON_TRAINING_HEARTBEAT

Step 8: Monitor model version management

Track whether the loaded model matches the expected version. Add to the health check:

def check_model_version():
    status, body = http_get(f'{RASA_URL}/status')
    if status != 200:
        return 'unknown'
    try:
        data = json.loads(body)
        model_file = data.get('model_file', '')
        # Alert if running a model older than 7 days
        if model_file:
            import os
            import time
            mtime = os.path.getmtime(model_file)
            age_days = (time.time() - mtime) / 86400
            if age_days > 7:
                return f'stale_{int(age_days)}d'
        return 'current'
    except Exception:
        return 'unknown'

Use a Vigilmon keyword alert for "model_version":"stale_ to detect when the model hasn't been retrained in a week.

Step 9: Slack alerts and status page

Slack alerts:

1. Notifications → New Channel → Slack
2. Paste your Slack incoming webhook URL
3. Enable on all Rasa monitors

When the action server goes down:

🔴 DOWN: rasa.yourdomain.com:5055/health
Status: connection refused
Region: US-East

When NLU latency spikes:

⚠️ SLOW: rasa.yourdomain.com/model/parse
Response time: 3,847ms (threshold: 1,000ms)
Region: EU-West

Status page:

1. Status Pages → New Status Page
2. Add all monitors
3. Name it "Rasa Platform Status"

What you've built

| What | How | |------|-----| | API server availability | Vigilmon HTTP → port 5005 | | Action server health | Vigilmon HTTP → port 5055 /health | | NLU inference latency | Vigilmon HTTP POST → /model/parse + 1s threshold | | Tracker store (PostgreSQL) | TCP + DB probe in health sidecar | | Lock store (Redis) | TCP + Redis ping in health sidecar | | Webhook endpoint health | Vigilmon HTTP → webhook paths (405 check) | | Training pipeline | Heartbeat from training cron | | Model version staleness | Age check in health sidecar | | Composite health | Vigilmon HTTP → port 5056 | | Slack alerts | Vigilmon Slack notification channel | | Status page | Vigilmon public status page |

Self-hosted Rasa gives you full control over your conversational AI stack. Monitoring gives you confidence that control is actually working.

Next steps

• Add separate heartbeat monitors for each Rasa background job (event brokers, reminder processors)
• Monitor conversation success rate via Rasa's built-in analytics by parsing tracker store data
• Set up SSL certificate monitoring for your Rasa domain — expired certificates are a common silent failure

Get started free at vigilmon.online.