Monitoring Datasette with Vigilmon

Datasette is an open-source Python tool that turns SQLite databases into interactive, explorable APIs and websites. It's used for data journalism, internal analytics, open data portals, and public-facing APIs. When Datasette goes down, data consumers hit 502 errors, API integrations break, and searches stop working. Vigilmon gives you web server availability monitoring, database health checks, query latency tracking, and data refresh job alerts — all without writing a single line of monitoring code.

What You'll Set Up

• Web server availability via the Datasette HTTP health endpoint
• SQLite database file accessibility check
• Query execution health and latency monitoring
• CSV and JSON export endpoint availability
• Full-text search index health
• Authentication plugin service monitoring
• Scheduled data refresh job execution via heartbeat monitors
• API endpoint response time tracking

Prerequisites

• Datasette running on a server or VPS (default port 8001)
• Datasette accessible over HTTP or HTTPS (directly or via a reverse proxy)
• A free Vigilmon account

Step 1: Monitor the Datasette Web Server

Datasette's root endpoint (/) returns a list of databases. A 200 response confirms the Python process is up and SQLite files are accessible:

1. Log in to vigilmon.online and click Add Monitor.
2. Set Type to HTTP / HTTPS.
3. Enter the URL: https://data.yourdomain.com (or http://YOUR_SERVER_IP:8001).
4. Set Check interval to 1 minute.
5. Set Expected HTTP status to 200.
6. Click Save.

If you use a reverse proxy (nginx, Caddy), also set up an SSL certificate monitor for the domain — Datasette is often used to serve public data and a broken certificate causes immediate trust loss.

Step 2: Monitor the JSON API Endpoint

Datasette's primary data delivery mechanism is its JSON API. Probe a specific table endpoint to confirm SQLite access and query execution are both healthy:

1. Add Monitor → HTTP / HTTPS.
2. URL: https://data.yourdomain.com/DATABASE_NAME/TABLE_NAME.json?_shape=array&_size=1
3. Expected HTTP status: 200
4. Keyword match: [ — confirms the response is a JSON array (not an error page).
5. Check interval: 2 minutes
6. Save.

Replace DATABASE_NAME and TABLE_NAME with a table you know is always populated. This monitor will fail if:

• The Datasette process crashes
• The SQLite file becomes unreadable (permissions, disk full)
• A query hangs indefinitely (disk I/O problem)

Step 3: Monitor the CSV Export Endpoint

Datasette's CSV export is used by many data pipelines and spreadsheet integrations. Monitor it separately because it exercises a different code path than the JSON API:

1. Add Monitor → HTTP / HTTPS.
2. URL: https://data.yourdomain.com/DATABASE_NAME/TABLE_NAME.csv?_size=1
3. Expected HTTP status: 200
4. Keyword match: , — confirms the response is CSV-formatted, not an error page.
5. Check interval: 5 minutes
6. Save.

Step 4: Monitor Full-Text Search Health

If your Datasette instance has full-text search enabled (via SQLite FTS5 tables), probe the search endpoint to confirm the index is intact:

1. Add Monitor → HTTP / HTTPS.
2. URL: https://data.yourdomain.com/DATABASE_NAME/TABLE_NAME.json?_search=test&_size=1
3. Expected HTTP status: 200
4. Check interval: 5 minutes
5. Save.

Use a search term that you know returns at least one result from your data. If the FTS index is corrupted or the search module fails to load, this monitor catches it.

Step 5: Monitor the Datasette Metadata Endpoint

Datasette exposes a /-/metadata endpoint that lists all databases and their configuration. It's a fast, lightweight health probe that confirms the app is fully initialized:

1. Add Monitor → HTTP / HTTPS.
2. URL: https://data.yourdomain.com/-/metadata
3. Expected HTTP status: 200
4. Keyword match: databases — confirms the metadata object is populated.
5. Check interval: 2 minutes
6. Save.

Step 6: Monitor Plugin Load Status

Datasette plugins (datasette-auth-github, datasette-vega, datasette-cluster-map, etc.) can fail silently if their dependencies are missing. The /-/plugins endpoint lists all loaded plugins:

1. Add Monitor → HTTP / HTTPS.
2. URL: https://data.yourdomain.com/-/plugins
3. Expected HTTP status: 200
4. Keyword match: the name of a critical plugin (e.g., datasette-auth-github).
5. Check interval: 10 minutes
6. Save.

If the keyword disappears from the response, the plugin failed to load — perhaps after a pip install that introduced a conflict.

Step 7: Heartbeat Monitor for Scheduled Data Refresh Jobs

Many Datasette deployments refresh their underlying SQLite databases on a schedule — pulling from a PostgreSQL replica, an API, or a CSV pipeline. A failed refresh means Datasette serves stale data without any visible error. Use a heartbeat monitor to track it:

Create the heartbeat

1. Click Add Monitor → Heartbeat.
2. Name: Datasette data refresh
3. Grace period: Set to your refresh interval plus 30 minutes (e.g., 90 minutes for an hourly job).
4. Copy the heartbeat URL.

Ping from your refresh script

#!/bin/bash
# /usr/local/bin/refresh-datasette.sh
set -e

HEARTBEAT_URL="https://vigilmon.online/api/heartbeat/YOUR_HEARTBEAT_TOKEN"
DB_PATH="/var/lib/datasette/mydata.db"
TEMP_PATH="/var/lib/datasette/mydata.db.tmp"

# Your data pull logic here
python3 /usr/local/bin/build-database.py --output "$TEMP_PATH"

# Atomic swap to avoid serving a half-written database
mv "$TEMP_PATH" "$DB_PATH"

# Signal success to Vigilmon
curl -s "$HEARTBEAT_URL"

Schedule via cron:

# /etc/cron.d/datasette-refresh
0 * * * * datasette /usr/local/bin/refresh-datasette.sh >> /var/log/datasette-refresh.log 2>&1

If build-database.py or the mv fails, the heartbeat is not pinged and Vigilmon alerts you after the grace period.

Step 8: Monitor Authentication Plugin Service

If you use datasette-auth-tokens, datasette-auth-github, or a similar authentication plugin, verify that protected endpoints still require authentication — a misconfiguration could accidentally expose private data:

1. Add Monitor → HTTP / HTTPS.
2. URL: a protected endpoint, e.g., https://data.yourdomain.com/private-db/users.json
3. Expected HTTP status: 403 or 302 (redirect to login)
4. Check interval: 10 minutes
5. Save.

If the expected status changes to 200, the authentication layer has broken and private data is publicly accessible — this is a critical alert.

Step 9: Configure Alerting

1. Go to Settings → Notifications in Vigilmon.
2. Add your notification channel: email, Slack, PagerDuty, or webhook.
3. For the core web server monitor (Step 1), set alert after 1 failed check.
4. For the JSON API and CSV export monitors, set alert after 2 failed checks.
5. For the auth endpoint monitor (Step 8), set alert after 1 failed check — an unexpected 200 on a protected route is a security event.
6. Enable recovery notifications so you know when service resumes.

Key Metrics to Watch

| Signal | Monitor Type | Alert Threshold | |---|---|---| | Root endpoint / | HTTP | 1 failure | | JSON API .json | HTTP | 2 failures | | CSV export .csv | HTTP | 2 failures | | Full-text search ?_search= | HTTP | 2 failures | | Metadata /-/metadata | HTTP | 2 failures | | Plugin list /-/plugins | HTTP | Plugin keyword missing | | Data refresh job | Heartbeat | Grace period exceeded | | Auth-protected endpoint | HTTP | Unexpected 200 |

Conclusion

Datasette is often the public face of your organization's data — journalists, researchers, and API consumers depend on it. With Vigilmon you get server availability monitoring, SQLite query health checks, export endpoint probes, plugin load tracking, and data refresh job heartbeats all in a single dashboard. Whether you're running a public open-data portal or an internal analytics tool, you'll know about problems before your users do.

Sign up for Vigilmon — free tier covers all the monitors in this tutorial.