tutorial

Monitoring Chaos Genius with Vigilmon: Open-Source Data Observability Platform Health, API Availability & Alert Delivery

How to monitor Chaos Genius open-source data observability and root-cause analysis infrastructure with Vigilmon — platform uptime, API availability, alert webhook delivery, and SSL certificate validity.

Chaos Genius is the open-source data observability and root-cause analysis platform that automatically monitors KPIs and data metrics for anomalies, then uses its proprietary DeepDrills algorithm to drill down into dimensional breakdowns and identify the root cause of unexpected changes — without requiring manual investigation of every slice and filter combination. When the Chaos Genius server becomes unavailable, data and analytics teams lose access to the automated anomaly detection and root-cause analysis workflows they rely on to understand why a metric moved. When the Chaos Genius API is degraded, integrations with your data warehouse and business intelligence tools stop receiving quality signals. When alert webhooks fail, Slack and email notifications for detected anomalies stop flowing, leaving metric regressions unacknowledged until stakeholders escalate them. Vigilmon gives you external visibility into Chaos Genius's self-hosted infrastructure: the application server, the analytics API, and your alert delivery endpoints, so you know the moment the data observability platform itself needs attention.

What You'll Build

  • An HTTP monitor on the Chaos Genius application server to detect availability failures
  • A health-check monitor on the Chaos Genius API to catch backend degradation
  • A monitor on your webhook receiver to verify Chaos Genius alert delivery is flowing
  • A monitor on the Chaos Genius task scheduler to catch KPI job failures
  • An SSL certificate monitor for your Chaos Genius domain
  • Alerting that distinguishes application server outages from API and scheduler failures

Prerequisites

  • A self-hosted Chaos Genius deployment (Docker Compose or Kubernetes)
  • The Chaos Genius application server exposed via HTTPS
  • A webhook receiver endpoint configured to accept Chaos Genius anomaly alerts
  • A free account at vigilmon.online

Step 1: Understand Chaos Genius's Service Architecture

Chaos Genius is a self-hosted platform with multiple cooperating components that each have distinct failure modes:

| Component | Default port | Role | |---|---|---| | Chaos Genius webapp | 8080 or 443 (HTTPS) | Web UI for configuring KPI monitors, reviewing anomalies, and running DeepDrills analysis | | Chaos Genius API server | 8080 or 443 (HTTPS) | Backend REST API that serves anomaly data and accepts configuration updates | | Celery task scheduler | Internal | Orchestrates KPI refresh jobs, anomaly detection runs, and scheduled report generation | | Redis | 6379 (internal) | Task queue and result backend for the Celery scheduler | | PostgreSQL | 5432 (internal) | Stores KPI configurations, anomaly history, and DeepDrills results |

External monitoring focuses on the webapp and API endpoints — these surface the overall health of the Chaos Genius deployment. Internal components like Redis and PostgreSQL are monitored indirectly: if they fail, the webapp and API begin returning errors that Vigilmon's HTTP monitors will detect.

Step 2: Monitor the Chaos Genius Application Server

The Chaos Genius webapp is the primary interface for configuring KPI monitors, reviewing detected anomalies, interpreting DeepDrills root-cause breakdowns, and managing data source connections to your warehouse or database:

# Self-hosted Chaos Genius
curl -I https://chaos-genius.your-domain.com
# Returns HTTP 200 with HTML when the webapp is healthy
  1. Log in to Vigilmon → Add Monitor → HTTP.
  2. URL: https://chaos-genius.your-domain.com (your self-hosted Chaos Genius domain).
  3. Check interval: 60 seconds.
  4. Response timeout: 15 seconds.
  5. Expected status: 200.
  6. Keyword: Chaos Genius (appears in the page title and HTML content).
  7. Click Save.

This monitor catches:

  • Chaos Genius application container crashes or failed restarts after an upgrade
  • Nginx reverse proxy misconfigurations that prevent traffic from reaching the webapp
  • Database connectivity failures that cause the application to return 5xx errors on load
  • DNS resolution failures for the Chaos Genius domain

Alert sensitivity: Set to trigger after 1 consecutive failure. When the Chaos Genius webapp is unavailable, analytics engineers cannot review active anomalies, run DeepDrills analysis on a suspicious metric movement, or configure new KPI monitors — leaving metric investigations without an automated root-cause analysis tool.

Step 3: Monitor the Chaos Genius API

The Chaos Genius REST API exposes anomaly detection results and KPI configuration endpoints that programmatic integrations and custom dashboards use to pull data quality signals. When the API degrades, webhook deliveries may continue but any integration that polls Chaos Genius for KPI status or anomaly counts fails to retrieve updated data:

curl https://chaos-genius.your-domain.com/api/v1/ping
# Returns: {"status": "ok"} when the API backend is healthy
  1. Add Monitor → HTTP.
  2. URL: https://chaos-genius.your-domain.com/api/v1/ping.
  3. Check interval: 60 seconds.
  4. Response timeout: 10 seconds.
  5. Expected status: 200.
  6. Keyword: ok (present in healthy API responses).
  7. Label: Chaos Genius API health.
  8. Click Save.

Why monitor the API separately from the webapp? The Chaos Genius webapp may display cached KPI data even when the API backend is failing to connect to the database or Celery worker. A failing API means that freshly computed anomaly results cannot be retrieved by external integrations, and configuration updates made through the UI may fail to persist.

Step 4: Monitor Your Webhook Receiver

Chaos Genius delivers anomaly detection alerts to your on-call tools (Slack, email, custom webhooks) via outbound HTTP POST requests triggered by the Celery task scheduler when a KPI anomaly is detected. If your webhook receiver becomes unavailable or returns errors, alert delivery fails silently — analysts stop receiving notifications about metric anomalies and root-cause breakdowns are not shared with the teams responsible for the underlying data:

curl https://your-webhook-receiver.example.com/health
# Returns: {"status": "ok"} when the receiver is up
  1. Add Monitor → HTTP.
  2. URL: https://your-webhook-receiver.example.com/health.
  3. Check interval: 2 minutes.
  4. Response timeout: 10 seconds.
  5. Expected status: 200.
  6. Keyword: ok.
  7. Label: Chaos Genius webhook receiver.
  8. Click Save.

For Slack incoming webhooks, verify https://slack.com/api/api.test to confirm the Slack API is reachable. This is especially important for Chaos Genius deployments where the primary notification channel is a shared Slack workspace — a Slack API outage will silently drop all anomaly alerts until the channel is restored.

Step 5: Monitor the Chaos Genius Task Scheduler

Chaos Genius uses Celery with Redis to run scheduled KPI anomaly detection jobs on a configured cadence. When the Celery scheduler or its Redis backend becomes unavailable, KPI refresh jobs stop running — meaning Chaos Genius stops detecting new anomalies even though the web interface and API continue to respond normally. The scheduler health endpoint exposes whether background jobs are being processed:

curl https://chaos-genius.your-domain.com/api/v1/scheduler/health
# Returns scheduler worker status and last successful job timestamp
  1. Add Monitor → HTTP.
  2. URL: https://chaos-genius.your-domain.com/api/v1/scheduler/health.
  3. Check interval: 5 minutes.
  4. Response timeout: 15 seconds.
  5. Expected status: 200.
  6. Keyword: health (present in the response JSON).
  7. Label: Chaos Genius scheduler health.
  8. Click Save.

Scheduler failures are the most operationally deceptive failure mode in self-hosted observability platforms: the UI looks fine, the API responds normally, but no new anomalies are being detected because the background workers have stopped. Vigilmon's external check against the scheduler health endpoint surfaces this silent failure before the gap in anomaly history is noticed.

Step 6: Monitor SSL Certificates

Chaos Genius is a self-hosted platform, which means SSL certificate management is the responsibility of your team — there is no managed certificate rotation from a SaaS provider. An expired certificate blocks browser access to the anomaly investigation UI and causes any external webhook receiver that verifies TLS to reject connections from Chaos Genius's alert dispatcher:

openssl s_client -connect chaos-genius.your-domain.com:443 2>/dev/null | openssl x509 -noout -dates
  1. Add Monitor → SSL Certificate.
  2. Domain: chaos-genius.your-domain.com.
  3. Alert when expiry is within: 30 days.
  4. Alert again: 14 days, 7 days, 3 days, 1 day.
  5. Click Save.

Self-hosted certificate management: Chaos Genius deployments often run behind an Nginx reverse proxy with a Let's Encrypt certificate managed by certbot. If the certbot renewal cron job fails (a common failure in containerized environments where certbot is not running in the correct host context), the certificate expires without warning. Vigilmon's 30-day pre-expiry alert gives your team time to diagnose and fix the renewal pipeline before the certificate lapses.

Step 7: Configure Alerting

In Vigilmon under Settings → Notifications, configure your alert channels:

| Monitor | Trigger | Action | |---|---|---| | Application server | Non-200 or Chaos Genius missing | Check container logs; verify Nginx proxy and database connectivity | | API health | Non-200 or ok missing | API down; verify database and Redis connections; check container status | | Webhook receiver | Non-200 or ok missing | Anomaly alerts not delivering; verify receiver deployment and logs | | Scheduler health | Non-200 or health missing | KPI jobs stopped; check Celery workers and Redis connectivity | | SSL certificate | < 30 days to expiry | Renew certificate; verify certbot renewal is working in production |

Alert after: 1 consecutive failure for the application server and API health monitors. 2 consecutive failures for scheduler health and webhook receiver monitors to reduce noise from brief container restarts during Celery task bursts.

Common Chaos Genius Failure Modes and What Vigilmon Catches

| Scenario | Vigilmon monitor | |---|---| | Application container crash | Server monitor fires; engineers cannot review anomalies or run DeepDrills | | API backend failure | API monitor fires; external integrations cannot pull KPI status | | Webhook receiver crash | Webhook monitor fires; Slack and email alerts stop posting | | Celery worker crash | Scheduler monitor fires; KPI jobs stop; anomalies go undetected silently | | Redis connection failure | Scheduler and API monitors may fire as task queue becomes unavailable | | TLS certificate expired | SSL monitor alerts at 30-day threshold; browser and webhook clients fail | | DNS misconfiguration | All monitors fire simultaneously | | PostgreSQL disk full | API and server monitors fire with 5xx errors as database writes fail |

Monitoring KPI Quality Itself (Beyond Infrastructure)

Vigilmon monitors Chaos Genius's self-hosted infrastructure — the application server, REST API, task scheduler, and your webhook receiver. The health of individual KPI monitors and detected anomalies is a separate concern managed within Chaos Genius itself:

  • Chaos Genius's built-in alerting: Configure notification channels within Chaos Genius (Slack, email, custom webhooks) to route anomaly alerts to the analysts and engineers responsible for each KPI — this is distinct from monitoring Chaos Genius's own availability.
  • KPI monitor coverage: Regularly audit which business metrics and data warehouse tables have Chaos Genius KPI monitors configured to ensure new metrics added to your stack are automatically included.
  • DeepDrills review cadence: Establish a regular review of Chaos Genius's anomaly feed and DeepDrills results to identify patterns in root causes — recurring anomalies in the same dimensional breakdown often indicate an upstream data pipeline issue that requires a fix rather than repeated acknowledgment.

Vigilmon catches infrastructure-level failures — when Chaos Genius itself is down or its background workers have stopped. Individual KPI anomaly coverage and root-cause analysis within a healthy Chaos Genius deployment require Chaos Genius's own monitor configuration and alert routing.

Chaos Genius is the layer that makes your KPI movements explainable — when it goes down or its task scheduler stops, anomaly detection halts, DeepDrills analysis becomes unavailable, and analytics teams lose the automated root-cause investigation tool they rely on to understand why a business metric changed. Vigilmon gives you external visibility into Chaos Genius's self-hosted availability that doesn't depend on Chaos Genius self-reporting: application server uptime, API health, scheduler status, and SSL certificate validity, so you know the moment the data observability platform itself needs attention and can restore KPI monitoring before the next anomaly goes undetected.

Start monitoring Chaos Genius in under 5 minutes — register free at vigilmon.online.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles