How to Monitor devpi with Vigilmon

devpi is the self-hosted Python package server that keeps your pip install commands working even when PyPI is unavailable, a package gets yanked, or your team needs to distribute internal packages privately. When devpi goes down, every developer and every CI job that runs pip install against your internal index immediately fails.

In this tutorial you'll set up comprehensive monitoring for devpi-server using Vigilmon, covering the HTTP API, PyPI mirror sync, package upload/download health, and backup job verification.

Why devpi monitoring matters

devpi outages have a cascading blast radius:

• All pip installs fail — any developer or CI job configured to use your devpi index as the primary or only index gets Connection refused or 404 on every package install
• Mirror sync lag — devpi falls behind syncing PyPI; developers start getting old package versions or install failures for recently-released packages
• Storage exhaustion — devpi's key-value file store fills the disk; uploads fail and the server may start returning 500 on downloads too
• Backup missed — devpi's data store isn't in PostgreSQL; it's a custom key-value file format that must be backed up separately; a missed backup means data loss after any disk failure
• Index inheritance broken — your dev/staging/prod index chain breaks silently; packages from upstream indexes stop resolving

Vigilmon catches all of these before they affect your team.

What you'll need

• A running devpi-server instance (default port 3141)
• A free Vigilmon account

Step 1: Verify devpi API health endpoint

devpi exposes a JSON API. The root API info endpoint is a reliable health probe:

curl http://your-devpi:3141/+api

A healthy response:

{
  "login": "http://your-devpi:3141/+login",
  "authstatus": ["noauth", ""],
  "result": {...},
  "type": "apiconfig",
  "status": 200
}

devpi-server also exposes /+status which provides server status details:

curl http://your-devpi:3141/+status

For an HTTPS setup (recommended in production), use nginx or Caddy as a reverse proxy and terminate TLS there. The devpi-server itself only speaks HTTP.

Step 2: Set up HTTP monitoring for the devpi API

1. Log in to vigilmon.online and go to Monitors → New Monitor
2. Choose HTTP / HTTPS as the type
3. Set the URL to http://your-devpi:3141/+api
4. Set check interval to 1 minute
5. Under Expected response, set:
   • Status code: 200
   • Response body contains: "type": "apiconfig" or just "status" to confirm valid API response
6. Save the monitor

When this monitor fails, all pip installs against your devpi instance are broken. This is a P1 alert.

Step 3: Monitor the PyPI mirror sync health

devpi mirrors PyPI packages to a local cache. If the sync job falls behind, developers may get stale package versions or install failures for recently-published packages.

Check mirror sync status via the devpi API:

# Check the root/pypi index which is the PyPI mirror
curl http://your-devpi:3141/root/pypi/+status

A healthy mirror shows recent serial numbers and sync timestamps. Set up a heartbeat script:

#!/bin/bash
# /usr/local/bin/check-devpi-mirror.sh

DEVPI_URL="http://localhost:3141"
PING_URL="https://vigilmon.online/heartbeat/YOUR_HEARTBEAT_ID"
MAX_LAG_HOURS=2

# Get mirror status
mirror_status=$(curl -fs "${DEVPI_URL}/root/pypi/+status" 2>/dev/null)

if [ -z "$mirror_status" ]; then
  echo "ERROR: devpi mirror status endpoint unreachable"
  exit 1
fi

# Check if mirroring is active (basic check)
is_mirroring=$(echo "$mirror_status" | python3 -c "
import json, sys
try:
    d = json.load(sys.stdin)
    result = d.get('result', {})
    # If we get a result, devpi mirror is responsive
    print('ok')
except:
    print('error')
")

if [ "$is_mirroring" != "ok" ]; then
  echo "ERROR: devpi mirror status malformed"
  exit 1
fi

curl -fsS "$PING_URL" > /dev/null 2>&1
echo "OK: devpi PyPI mirror is active"

Add to cron to run every 15 minutes:

*/15 * * * * /usr/local/bin/check-devpi-mirror.sh >> /var/log/devpi-mirror-check.log 2>&1

In Vigilmon, create a Heartbeat monitor for the PyPI mirror check with a 30-minute expected interval. If the mirror check stops reporting in, you'll be alerted before your developers notice stale packages.

Step 4: Monitor package download health

Test that packages are actually downloadable through your devpi instance:

# Probe a known package download through devpi (e.g., requests)
# This tests the full path: devpi → mirror lookup → blob serve
curl -fsS \
  "http://your-devpi:3141/root/pypi/+simple/requests/" \
  | grep -q "requests-" && echo "OK" || echo "FAIL"

Add an HTTP monitor for the simple index:

1. New Monitor → HTTP / HTTPS
2. URL: http://your-devpi:3141/root/pypi/+simple/pip/
3. Expected status: 200
4. Response body contains: pip- (confirms the simple index returns actual package entries)
5. Check interval: 5 minutes

This monitor catches storage backend failures and mirror database corruption that would cause silent download failures.

Step 5: Monitor disk/storage utilization

devpi stores all package blobs in a local key-value file store (.devpi directory by default). Storage exhaustion causes upload failures and can corrupt the database.

#!/bin/bash
# /usr/local/bin/check-devpi-disk.sh

DEVPI_DIR="/home/devpi/.devpi"
THRESHOLD=80
PING_URL="https://vigilmon.online/heartbeat/YOUR_DISK_HEARTBEAT_ID"

# Check disk usage of devpi data directory
usage=$(df -h "$DEVPI_DIR" | awk 'NR==2 {gsub(/%/,""); print $5}')

if [ "$usage" -gt "$THRESHOLD" ]; then
  echo "ALERT: devpi storage at ${usage}% (threshold: ${THRESHOLD}%)"
  exit 1
fi

curl -fsS "$PING_URL" > /dev/null 2>&1
echo "OK: devpi storage at ${usage}%"

Set the heartbeat interval to 10 minutes and the alert condition to "missed more than one check" — this gives you a 20-minute window to respond before the disk fills completely.

Step 6: Monitor user authentication health

devpi has user-level access control. Authentication failures can lock out developers or break CI credentials silently.

Test authentication health:

# Test login endpoint
curl -X POST http://your-devpi:3141/+login \
  -H "Content-Type: application/json" \
  -d '{"user": "your-ci-user", "password": "your-ci-password"}' \
  | python3 -c "import json,sys; d=json.load(sys.stdin); print(d.get('type','error'))"

A healthy response returns "userconfig" type. An auth failure returns a 401 with an error message.

Add an HTTP monitor targeting the login endpoint:

1. New Monitor → HTTP / HTTPS
2. URL: http://your-devpi:3141/+login
3. Method: POST
4. Headers: Content-Type: application/json
5. Body: {"user": "monitor-user", "password": "monitor-password"}
6. Expected status: 200
7. Response body contains: "userconfig"

Create a dedicated read-only monitor user in devpi for this check so you don't expose production credentials in the monitor configuration.

Step 7: Monitor the backup job

devpi's key-value file store is not a traditional database — you must run devpi-server --export to create a portable backup. If backups stop running, any disk failure means data loss.

#!/bin/bash
# /usr/local/bin/devpi-backup.sh

DEVPI_DIR="/home/devpi/.devpi"
BACKUP_DIR="/backup/devpi"
PING_URL="https://vigilmon.online/heartbeat/YOUR_BACKUP_HEARTBEAT_ID"
DATE=$(date +%Y%m%d-%H%M%S)
BACKUP_PATH="${BACKUP_DIR}/devpi-${DATE}"

# Create backup directory
mkdir -p "$BACKUP_DIR"

# Export devpi data
devpi-server --serverdir "$DEVPI_DIR" --export "$BACKUP_PATH"

if [ $? -eq 0 ]; then
  # Remove backups older than 7 days
  find "$BACKUP_DIR" -name "devpi-*" -mtime +7 -exec rm -rf {} + 2>/dev/null
  
  # Ping Vigilmon heartbeat on success
  curl -fsS "$PING_URL" > /dev/null 2>&1
  echo "Backup completed: $BACKUP_PATH"
else
  echo "ERROR: devpi backup failed" >&2
  exit 1
fi

Add to cron for daily backups:

0 2 * * * /usr/local/bin/devpi-backup.sh >> /var/log/devpi-backup.log 2>&1

Set the Vigilmon heartbeat to a 25-hour expected interval. If the backup job skips a day, you'll be alerted before you're at risk of a 48-hour data exposure window.

Step 8: Configure alerting

1. Go to Alert Channels → Add Channel → Webhook
2. Set up Slack, PagerDuty, or email
3. Assign the channel to all your devpi monitors

Recommended alert policy:

• devpi API down → page immediately (P1 — all pip installs broken)
• Package download monitor fails → page immediately (P1 — pip installs failing)
• PyPI mirror heartbeat missed → warn (P2 — stale packages possible)
• Disk >80% → warn (P2 — act before uploads start failing)
• Backup heartbeat missed → warn (P2 — data loss risk if disk fails)
• Auth health monitor fails → warn (P2 — CI credentials may be broken)

Putting it all together

| Monitor | Type | Alert level | What it catches | |---------|------|-------------|-----------------| | http://devpi:3141/+api | HTTP | P1 | devpi server down | | http://devpi:3141/root/pypi/+simple/pip/ | HTTP | P1 | Package download / mirror failure | | PyPI mirror sync script | Heartbeat | P2 | Mirror sync stalled | | Disk usage script | Heartbeat | P2 | Storage >80% | | http://devpi:3141/+login POST | HTTP | P2 | Auth system broken | | Backup script | Heartbeat | P2 | Daily backup missed |

What's next

• Index staging chain — add HTTP monitors for each of your dev/staging/prod index URLs (http://devpi:3141/your-org/dev/+simple/, /staging/, /prod/) to verify the full promotion pipeline is accessible
• Replication health — if you run a devpi primary-replica setup for HA, add a heartbeat that verifies the replica is within a configurable lag of the primary by comparing serial numbers from each instance's /+api response
• TLS certificate — if you front devpi with nginx over HTTPS, add Vigilmon's SSL certificate monitor to alert 30 days before the certificate expires

Get started free at vigilmon.online — no credit card, monitors start running in under a minute.