tutorial

Monitoring Nix Build Systems and NixOS Services with Vigilmon

Nix's purely functional build system and NixOS make deployments reproducible — but the Nix daemon, binary cache, and deployed services still need runtime monitoring. Here's how to do it with Vigilmon.

Nix takes a fundamentally different approach to package management and system configuration: every build is a pure function of its inputs, outputs live in an immutable store, and system state is declared rather than imperative. This gives you reproducible builds, atomic upgrades, and easy rollbacks. But reproducibility is a build-time guarantee, not a runtime guarantee. The Nix daemon can crash, your binary cache can become unavailable, and NixOS services configured via systemd can fail to start after a rebuild. Vigilmon adds the runtime layer that Nix intentionally does not include: uptime monitoring for your NixOS services, heartbeat checks for scheduled builds and garbage collection runs, and HTTP monitors for any Nix binary caches you operate.

What You'll Set Up

  • HTTP uptime monitors for NixOS-managed services
  • Cron heartbeat monitors for nix build pipelines and garbage collection jobs
  • HTTP monitors for self-hosted Nix binary caches (Cachix, attic, or custom)
  • SSL certificate alerts for cache and service endpoints
  • TCP monitors for network services managed by NixOS

Prerequisites

  • A NixOS system or a Linux/macOS machine with Nix installed
  • At least one service deployed and running (via NixOS systemd or direct Nix builds)
  • A free Vigilmon account

Step 1: Add Health Endpoints to NixOS Services

NixOS manages services through systemd using declarative configuration in /etc/nixos/configuration.nix or flake-based system configurations. Any HTTP service you configure should expose a health endpoint.

For a service defined in your NixOS configuration:

# configuration.nix
services.myapp = {
  enable = true;
  port = 8080;
  healthPath = "/healthz";
};

If the service is a custom application in your Nix flake, add a health endpoint in the application code:

Go

mux.HandleFunc("/healthz", func(w http.ResponseWriter, r *http.Request) {
    w.WriteHeader(http.StatusOK)
    fmt.Fprint(w, "ok")
})

Python (Starlette/FastAPI)

@app.get("/healthz")
async def health():
    return {"status": "ok"}

Verify the endpoint is reachable after a nixos-rebuild switch:

curl -sf http://localhost:8080/healthz

Step 2: Monitor NixOS Services with Vigilmon

With your service's health endpoint confirmed, add a Vigilmon monitor:

  1. Log in to vigilmon.online and click Add Monitor.
  2. Set Type to HTTP / HTTPS.
  3. Enter your service URL: https://myservice.yourdomain.com/healthz.
  4. Set Check interval to 1 minute.
  5. Set Expected HTTP status to 200.
  6. Click Save.

NixOS services restart via systemd, and a nixos-rebuild switch can cause a brief service interruption while the new derivation activates. Set Consecutive failures before alert to 2 to avoid false positives during expected system updates.

For services that listen only on localhost (common for database-backed apps behind nginx reverse proxy), monitor the public nginx-proxied endpoint — this checks both the service and the nginx configuration that Nix manages.

Step 3: Monitor Your Nix Binary Cache

Nix binary caches dramatically speed up builds by serving pre-built derivations. Whether you self-host with attic, run a Cachix organization cache, or operate a custom Nix HTTP binary cache, its availability directly affects your CI/CD build times.

Add an HTTP monitor for your binary cache:

  1. Click Add Monitor → HTTP / HTTPS.
  2. Enter the cache URL: https://cache.yourdomain.com/nix-cache-info.
  3. Set Expected HTTP status to 200.
  4. Set Check interval to 5 minutes.
  5. Click Save.

The /nix-cache-info endpoint is standardized across all Nix binary caches — it returns the cache's store path prefix and priority. A 200 response confirms the cache is reachable and correctly configured as a Nix binary cache.

For caches secured with authentication (private Cachix caches or attic caches behind a token), Vigilmon's HTTP monitor supports custom headers:

  1. Enable Custom request headers on the monitor.
  2. Add Authorization: Bearer YOUR_CACHE_TOKEN.
  3. Save.

A binary cache outage during CI will not break your builds — Nix falls back to building from source — but it will dramatically increase build times and may exhaust CI runner timeouts on large derivations. An alert lets you restore the cache before it silently degrades your pipeline.

Step 4: Heartbeat Monitoring for Nix Build Pipelines

Nix builds are often scheduled: nightly CI runs that produce and push derivations to your binary cache, weekly NixOS system rebuild tests, or periodic nix flake update runs that refresh inputs. Use Vigilmon heartbeats to confirm these jobs complete.

  1. In Vigilmon, click Add Monitor → Cron Heartbeat.
  2. Set Expected ping interval to match your build schedule (e.g. 1440 minutes for a daily build).
  3. Copy the heartbeat URL.

Add the ping to your build script:

#!/usr/bin/env bash
set -euo pipefail

# Update flake inputs
nix flake update

# Build and push to binary cache
nix build .#packages.x86_64-linux.myapp
nix copy --to "s3://mybucket?region=us-east-1" ./result

# Ping Vigilmon on success
curl -s https://vigilmon.online/heartbeat/YOUR_ID

For GitHub Actions CI:

- name: Build Nix derivation
  run: nix build .#packages.x86_64-linux.myapp

- name: Push to binary cache
  run: nix copy --to "s3://mybucket?region=us-east-1" ./result

- name: Ping Vigilmon heartbeat
  if: success()
  run: curl -s https://vigilmon.online/heartbeat/YOUR_ID

Step 5: Monitor Nix Garbage Collection

Nix's store grows indefinitely until you run garbage collection. A common pattern is a weekly nix-collect-garbage job run via systemd timers in NixOS:

# configuration.nix
systemd.services.nix-gc = {
  description = "Nix Garbage Collection";
  serviceConfig = {
    Type = "oneshot";
    ExecStart = "${pkgs.nix}/bin/nix-collect-garbage --delete-older-than 30d";
    ExecStartPost = "${pkgs.curl}/bin/curl -s https://vigilmon.online/heartbeat/GC_HEARTBEAT_ID";
  };
};

systemd.timers.nix-gc = {
  wantedBy = ["timers.target"];
  timerConfig = {
    OnCalendar = "weekly";
    Persistent = true;
  };
};

The ExecStartPost line pings Vigilmon after the garbage collection completes. Create a separate heartbeat monitor for this with an interval of 10080 minutes (7 days).

Without monitoring, a silent garbage collection failure can allow /nix/store to fill the filesystem, causing all subsequent Nix operations to fail with no-space-left errors. The heartbeat catches this before the disk fills.

Step 6: TCP Port Monitoring for NixOS Network Services

For NixOS services that don't expose HTTP — PostgreSQL databases, Redis caches, SSH servers, custom TCP daemons — use TCP port monitoring:

  1. Click Add Monitor → TCP Port.
  2. Enter your server's IP or hostname and port (e.g. db.internal:5432).
  3. Set Check interval to 1 minute.
  4. Click Save.

Common NixOS service ports to monitor:

| NixOS Service | Port | |---|---| | services.postgresql | 5432 | | services.redis | 6379 | | services.openssh | 22 | | services.gitea | 3000 | | services.hydra (Nix CI) | 3000 | | services.harmonia (Nix cache) | 5000 |

For Hydra, the Nix-native CI system, also add an HTTP monitor for its web interface and a heartbeat for nightly evaluation jobs.

Step 7: SSL Certificate Alerts and Alert Configuration

For any HTTPS endpoints in your NixOS stack, add SSL certificate monitoring:

  1. Open the HTTP monitor for the endpoint.
  2. Enable Monitor SSL certificate under the SSL section.
  3. Set Alert when certificate expires in less than 21 days.
  4. Click Save.

NixOS configurations that use security.acme for Let's Encrypt certificate management will auto-renew, but the ACME renewal can fail silently if port 80 is blocked or the ACME service is misconfigured. A 21-day SSL alert gives you three weeks to diagnose and fix the renewal.

Configure alert channels:

  1. Go to Alert Channels and add your Slack workspace or email.
  2. For the binary cache HTTP monitor, route alerts to your infra channel — a cache outage is high-impact but not an emergency.
  3. For NixOS service health checks, route to PagerDuty or your primary on-call channel.
  4. Set Consecutive failures before alert to 2 for HTTP and TCP monitors — NixOS nixos-rebuild switch can cause a brief interruption.

Summary

| Monitor | Target | What It Catches | |---|---|---| | HTTP health check | NixOS service /healthz | Service crash after rebuild, activation failure | | HTTP cache check | /nix-cache-info on binary cache | Cache unreachable, slow builds | | Cron heartbeat | Nix build pipeline | Missed build, flake update failure | | Cron heartbeat | nix-collect-garbage systemd timer | GC failure, disk filling | | TCP port | Database, SSH, custom NixOS services | Service not listening | | SSL certificate | Exposed HTTPS endpoints | ACME renewal failure |

Nix and NixOS make your build and deployment pipeline reproducible and auditable. What they don't provide is runtime observability — that's your job. Vigilmon watches the runtime half: are your services up, is your binary cache reachable, did the nightly build run, and is the disk safe from a GC failure? With Vigilmon alongside Nix, reproducibility covers both what you ship and the system that confirms it's running.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles