Monitoring devenv with Vigilmon

devenv is an open source developer environment tool built on Nix that lets teams define language versions, system dependencies, background services (PostgreSQL, Redis, Elasticsearch), pre-commit hooks, and scripts in a single devenv.nix file. Environments are hermetic — they work identically on macOS, Linux, and in CI. But when the Nix binary cache is slow, a background service fails to start, or the flake.lock drifts between dev and CI, developers lose hours to environment noise instead of shipping features. Vigilmon gives you continuous visibility into every layer of devenv's health — from Nix cache hit rates to service port bindings to direnv activation latency.

What You'll Set Up

• Cron heartbeats for Nix binary cache health and cache hit rate
• Service health monitors for PostgreSQL, Redis, and other devenv-managed services
• Process supervisor health heartbeat
• Nix store integrity and disk space alerts
• CI/CD environment parity monitoring via flake.lock freshness

Prerequisites

• devenv installed and configured (devenv.nix in your project)
• devenv up running background services during development
• A free Vigilmon account

Why Monitoring devenv Matters

devenv failures are subtle. Unlike a web server that clearly goes down, devenv failures tend to silently degrade the developer experience:

• Binary cache misses: A cold Nix build with no cache can take 20–60 minutes. If cache.nixos.org is unreachable or your self-hosted Cachix is down, every devenv shell rebuild becomes a blocking event.
• Service startup failures: If PostgreSQL fails to bind its port, your application silently fails to connect — often misdiagnosed as an application bug.
• Process supervisor instability: devenv runs multiple services as a process group. A supervisor crash silently kills all background services.
• Stale flake.lock: When devenv's flake.lock differs between developer machines and CI, builds diverge. Tests pass locally, fail in CI, and the root cause is invisible.
• Nix store disk exhaustion: The local Nix store grows with every shell rebuild. Unchecked, it fills the disk and breaks all builds.

Step 1: Monitor the Nix Binary Cache

The Nix binary cache (cache.nixos.org or your self-hosted Cachix) is the difference between a 5-second shell activation and a 30-minute source build. Monitor it as an HTTP endpoint.

1. Log in to vigilmon.online and click Add Monitor.
2. Set Type to HTTP / HTTPS.
3. Set URL to https://cache.nixos.org/nix-cache-info (returns StoreDir: /nix/store when healthy).
4. Set Check interval to 5 minutes.
5. Set Expected HTTP status to 200.
6. Click Save.

For a self-hosted Cachix or custom binary cache server:

https://your-cache.yourdomain.com/nix-cache-info

Add a second monitor for SSL certificate expiry on the cache server — a certificate error will silently fall back to source builds.

Step 2: Heartbeat for Cache Hit Rate

Connectivity to the cache server is necessary but not sufficient — you also need to know that builds are actually hitting the cache rather than rebuilding from source. Set up a heartbeat that only pings when cache hit rate is healthy.

Add a script that runs after each devenv shell initialization:

#!/bin/bash
# devenv-cache-check.sh — run this in your devenv postShellHook or as a CI step

# Count cache hits in the last nix build log
NIX_LOG=$(nix-store --query --references /nix/var/nix/profiles/default 2>&1)
TOTAL=$(echo "$NIX_LOG" | wc -l)
BUILT=$(nix log /nix/var/nix/profiles/default 2>/dev/null | grep -c "building" || echo 0)

HIT_RATE=$(( (TOTAL - BUILT) * 100 / TOTAL ))

if [ "$HIT_RATE" -ge 80 ]; then
  curl -sf https://vigilmon.online/heartbeat/nix-cache-hit-xyz
fi

Set the heartbeat interval to your CI run frequency. If hit rate drops below 80%, the heartbeat stops arriving and Vigilmon alerts you.

Step 3: Monitor devenv-Managed Services

devenv runs PostgreSQL, Redis, and other services as background processes during development. Each service has a TCP port that Vigilmon can probe directly.

For each service, add a TCP monitor:

PostgreSQL (default port 5432):

1. Click Add Monitor → TCP Port.
2. Set Host to localhost (or the dev machine's LAN IP if monitoring from an external Vigilmon probe).
3. Set Port to 5432.
4. Set Check interval to 1 minute.
5. Click Save.

Redis (default port 6379):

Repeat the same steps with port 6379.

For the devenv HTTP API server (port 3000 when using devenv up with web services):

1. Add an HTTP / HTTPS monitor for http://localhost:3000/health.
2. Set the expected status to 200.

If Vigilmon is running on a separate monitoring host rather than localhost, expose service ports via SSH tunnel or monitor from a script on the dev machine that posts to Vigilmon's inbound API.

Step 4: Heartbeat for Process Supervisor Health

devenv uses a built-in process supervisor to manage multiple concurrent services. If the supervisor crashes, all background services die silently with no individual port failure — the ports stop responding one by one. A supervisor-level heartbeat catches this before you diagnose service by service.

Add a heartbeat from your devenv process health check:

#!/bin/bash
# devenv-supervisor-health.sh — run every 2 minutes via cron or launchd

# Check if devenv process supervisor is running
if pgrep -f "devenv processes up" > /dev/null 2>&1; then
  # Check that at least one managed service is running
  if devenv processes list 2>/dev/null | grep -q "running"; then
    curl -sf https://vigilmon.online/heartbeat/devenv-supervisor-xyz
  fi
fi

1. In Vigilmon, add a Cron Heartbeat with interval 5 minutes.
2. Run the script every 2 minutes via cron: */2 * * * * /path/to/devenv-supervisor-health.sh

Step 5: Nix Store Disk Space Alert

The Nix store at /nix/store grows unboundedly without garbage collection. When it fills the disk, all builds fail with cryptic errors.

Monitor disk usage with a heartbeat that only pings when the store is within safe limits:

#!/bin/bash
# nix-store-disk.sh — run every 30 minutes

NIX_USAGE=$(df /nix/store | awk 'NR==2 {print $5}' | tr -d '%')

if [ "$NIX_USAGE" -lt 85 ]; then
  curl -sf https://vigilmon.online/heartbeat/nix-store-disk-xyz
fi

1. Add a Cron Heartbeat in Vigilmon with interval 60 minutes.
2. Schedule the script to run every 30 minutes: */30 * * * * /path/to/nix-store-disk.sh

When the store hits 85% capacity, the heartbeat stops arriving and Vigilmon alerts you before builds start failing.

Also schedule periodic garbage collection to keep the store healthy:

# Weekly Nix garbage collection — keep last 7 days
0 2 * * 0 nix-collect-garbage --delete-older-than 7d

Step 6: Monitor flake.lock Freshness for CI Parity

When devenv's flake.lock is committed to the repo, it pins the exact Nix revisions used. If developers update their local flake.lock but don't commit it, or vice versa, dev and CI environments diverge — tests pass in one environment and fail in the other.

Add a heartbeat from your CI pipeline that validates flake.lock freshness:

#!/bin/bash
# In CI — check flake.lock age
LOCK_AGE_DAYS=$(( ($(date +%s) - $(git log -1 --format=%ct -- flake.lock)) / 86400 ))

# Alert if flake.lock hasn't been updated in over 30 days (staleness risk)
if [ "$LOCK_AGE_DAYS" -le 30 ]; then
  curl -sf https://vigilmon.online/heartbeat/flakelock-freshness-xyz
fi

Set the heartbeat interval to 720 minutes (12 hours). If no CI run checks in within 24 hours, something is wrong with your CI pipeline or your flake.lock is severely stale.

Step 7: Monitor Self-Hosted Binary Cache (Cachix)

If you run a self-hosted Cachix instance or custom Nix binary cache server for private packages, that server is a critical dependency. Add a full health monitor:

1. HTTP monitor for https://your-cache.yourdomain.com/nix-cache-info (as in Step 1).
2. SSL certificate alert — set expiry warning to 21 days.
3. Disk space heartbeat — the cache server's storage fills up as packages are uploaded:

#!/bin/bash
CACHE_USAGE=$(df /var/cache/nix | awk 'NR==2 {print $5}' | tr -d '%')
if [ "$CACHE_USAGE" -lt 80 ]; then
  curl -sf https://vigilmon.online/heartbeat/nix-cache-disk-xyz
fi

Step 8: Configure Alert Channels

1. Go to Alert Channels in Vigilmon.
2. Add your team Slack channel (so environment failures surface immediately).
3. For the binary cache monitor, set Consecutive failures before alert to 3 — transient CDN hiccups at cache.nixos.org are common.
4. For service port monitors, set Consecutive failures before alert to 2 — service restarts can take 5–15 seconds.
5. Leave supervisor and disk heartbeats at 1 missed ping — those are always urgent.

Summary

| Monitor | Target | What It Catches | |---|---|---| | HTTP uptime | https://cache.nixos.org/nix-cache-info | Nix cache unreachable | | TCP port | localhost:5432 | PostgreSQL down | | TCP port | localhost:6379 | Redis down | | Supervisor heartbeat | Heartbeat URL (5m interval) | Process supervisor crash | | Cache hit rate heartbeat | Heartbeat URL (CI frequency) | Cache regression | | Nix store disk heartbeat | Heartbeat URL (60m interval) | Disk space exhaustion | | flake.lock heartbeat | Heartbeat URL (12h interval) | Environment parity drift | | SSL certificate | Self-hosted cache endpoint | Certificate expiry |

devenv eliminates "works on my machine" for your team — but only when every layer of its infrastructure is healthy. With Vigilmon watching your Nix cache, background services, process supervisor, and environment parity, you catch devenv failures before they interrupt a developer's flow state.