YaCy Monitoring with Vigilmon: P2P Network, Crawl Scheduler & Search API Health Checks

Your YaCy node lost contact with the peer network five hours ago. The index has stopped growing. Users searching your instance are seeing results that haven't been updated since the node isolated itself. The Java heap hit its limit overnight and nobody got an alert.

YaCy is an open-source distributed peer-to-peer web search engine written in Java. Each YaCy instance runs a web admin interface on port 8090 and participates in a global P2P network of search nodes, sharing crawl data and index contributions. Self-hosting YaCy means you own the full pipeline from peer connectivity through crawl scheduling, page indexing, and search result serving — and you own the responsibility of knowing when any part of it fails.

Vigilmon probes your YaCy node from outside, the way a real user or peer node would. If the admin interface goes offline, if the crawl scheduler stalls, or if P2P connectivity drops below the minimum peer threshold, Vigilmon alerts you before your index grows stale and users start noticing degraded results.

What You'll Build

• An HTTP monitor on the YaCy web admin interface
• A crawl and indexing scheduler health check
• A P2P network connectivity monitor
• A search API response time probe
• An index database size and growth monitor
• A seed list peer synchronization check
• Alert channels for your search infrastructure team

Prerequisites

• A self-hosted YaCy installation (latest release recommended)
• Web admin interface default port: 8090
• A free account at vigilmon.online

Step 1: Monitor the YaCy Web Admin Interface

The YaCy web admin interface on port 8090 is the primary control surface for your node. If it goes down, you lose the ability to manage crawls, inspect index health, or diagnose P2P connectivity issues.

Test the admin interface is responding:

curl -o /dev/null -s -w "%{http_code}" http://your-yacy-host:8090/

Expected: a 200 response with the YaCy interface HTML.

Set up the Vigilmon monitor:

1. Log in to Vigilmon and click New Monitor → HTTP.
2. Set URL to http://your-yacy-host:8090/.
3. Set Check interval to 60 seconds.
4. Set Expected status code to 200.
5. Under Advanced → Keyword check, add:
   • Keyword present: YaCy
6. Save the monitor.

This baseline check catches Java process crashes, heap OOM kills, and port binding failures. A YaCy node that has crashed is completely disconnected from the P2P network and no longer contributing to shared index data.

Step 2: Monitor the Crawl and Indexing Scheduler

YaCy's crawl scheduler continuously fetches new pages and feeds them to the indexer. If the scheduler stalls — due to queue overflow, network timeouts, or Java thread exhaustion — your local index stops growing and your contribution to the P2P network stops.

Test the crawl job status via the YaCy API:

curl -s "http://your-yacy-host:8090/yacy/crawler.json?auth=YOUR_ADMIN_PASSWORD"

A healthy response returns JSON with crawl job counts and queue depths. The activeCount field should be non-zero during active crawl sessions. An activeCount of zero during a scheduled crawl window indicates the scheduler has stalled.

Set up the Vigilmon monitor:

1. Click New Monitor → HTTP in Vigilmon.
2. Set URL to http://your-yacy-host:8090/yacy/crawler.json.
3. Set Expected status code to 200.
4. Set Check interval to 300 seconds.
5. Under Advanced → Keyword check, add:
   • Keyword present: count
   • Keyword absent: error
6. Save the monitor.

The YaCy crawl status API responds immediately from memory — a slow or failed response indicates the Java servlet container is overwhelmed, not just the crawler thread.

Step 3: Monitor P2P Network Connectivity

YaCy's value comes from its peer network. Each node maintains connections to other YaCy peers to share index data and crawl assignments. If your node loses P2P connectivity — due to firewall changes, IP blocklisting, or network isolation — it becomes a standalone search engine with a static, growing-stale index.

Test peer connectivity via the network API:

curl -s "http://your-yacy-host:8090/yacy/network.json" \
  | python3 -c "import sys,json; d=json.load(sys.stdin); print(d.get('peers', {}).get('count', 0))"

A healthy YaCy node maintains connections to dozens or hundreds of peers depending on your network configuration. A peer count near zero or rapidly declining indicates network isolation.

Set up the Vigilmon monitor:

1. Click New Monitor → HTTP in Vigilmon.
2. Set URL to http://your-yacy-host:8090/yacy/network.json.
3. Set Expected status code to 200.
4. Set Check interval to 120 seconds.
5. Under Advanced → Keyword check, add:
   • Keyword present: peers
   • Keyword absent: error
6. Save the monitor.

Combine this Vigilmon check with a peer count threshold alert using Vigilmon's webhook integration to trigger an alert when the count field drops below your minimum viable peer threshold (typically 10 peers for a functioning P2P node).

Step 4: Monitor the Search API Response Times

YaCy serves search results through both its web interface and a JSON API. Slow or failing search responses indicate Java heap pressure, disk I/O saturation from index reads, or peer query aggregation timeouts.

Test a search query:

curl -o /dev/null -s -w "%{time_total}" \
  "http://your-yacy-host:8090/yacysearch.json?query=test&maximumRecords=5&nav=none"

A healthy YaCy local search completes in under 2 seconds for a simple query against the local index. Distributed P2P search can take 5–10 seconds. If local search is taking more than 3 seconds consistently, the index reader is under pressure.

Set up the Vigilmon monitor:

1. Click New Monitor → HTTP in Vigilmon.
2. Set URL to http://your-yacy-host:8090/yacysearch.json?query=test&maximumRecords=5&nav=none.
3. Set Expected status code to 200.
4. Set Timeout to 10000 ms — a query exceeding 10 seconds is a functional failure.
5. Set Check interval to 120 seconds.
6. Under Advanced → Keyword check, add:
   • Keyword present: channels
7. Save the monitor.

The channels field is present in every valid YaCy search JSON response. Its absence at a 200 status indicates the search layer returned a malformed or empty response.

Step 5: Monitor Index Database Size and Growth

YaCy stores its local index in an embedded Lucene database. If the index becomes corrupted, runs out of disk space, or stops growing due to indexer failure, the quality and freshness of search results degrades silently over time.

Test index statistics via the status API:

curl -s "http://your-yacy-host:8090/yacy/status.json" \
  | python3 -m json.tool | grep -E "urlCount|wordCount"

A healthy YaCy index shows a growing urlCount (total URLs indexed) and wordCount (total terms in the inverted index). Flat counts over 24 hours indicate the indexer has stopped processing crawled pages.

Set up the Vigilmon monitor:

1. Click New Monitor → HTTP in Vigilmon.
2. Set URL to http://your-yacy-host:8090/yacy/status.json.
3. Set Expected status code to 200.
4. Set Check interval to 600 seconds.
5. Under Advanced → Keyword check, add:
   • Keyword present: urlCount
   • Keyword absent: error
6. Save the monitor.

Step 6: Monitor Seed List Peer Synchronization

YaCy nodes use a seed list — a curated set of bootstrap peers — to discover and reconnect to the P2P network after restarts. If seed list synchronization fails, your node cannot join the network after a restart and operates in isolation indefinitely.

Test seed list accessibility:

curl -s "http://your-yacy-host:8090/yacy/seedlist.txt" | head -5

A healthy YaCy node serves its seed list at this URL, containing peer addresses in hash,ip,port format. An empty or error response means the node cannot provide seed data to joining peers and may itself be using a stale or unreachable seed source.

Set up the Vigilmon monitor:

1. Click New Monitor → HTTP in Vigilmon.
2. Set URL to http://your-yacy-host:8090/yacy/seedlist.txt.
3. Set Expected status code to 200.
4. Set Check interval to 600 seconds.
5. Under Advanced → Keyword check, add:
   • Keyword present: .
6. Save the monitor.

The presence of any content in the seedlist response (the . keyword check) confirms the node is serving peer discovery data. An empty response or 404 means the P2P bootstrap mechanism has broken.

Step 7: Alert Channels

Go to Notifications → New Channel in Vigilmon and configure:

• Email — immediate alerts to your search infrastructure team
• Webhook — Slack or Discord for team-wide visibility

A failing P2P connectivity alert looks like:

🔴 DOWN: yacy-p2p-network (HTTP monitor)
Expected status 200, got 500 Internal Server Error
Region: EU-Central
Triggered: 2026-07-13 01:33 UTC

When the node recovers:

✅ RECOVERED: yacy-p2p-network
Downtime: 2 hours 14 minutes

Set critical severity for the web admin interface and search API monitors — these represent complete service loss. Use warning severity for P2P connectivity, crawl scheduler, and seed list monitors, which represent gradual degradation. Enable Alerting → Escalation if the admin interface is down for more than 5 minutes; a crashed YaCy node is silent in the peer network and cannot be diagnosed remotely.

What You've Built

| Scenario | How Vigilmon catches it | |---|---| | Java process crash or OOM kill | HTTP monitor on :8090 fails with connection refused | | Crawl scheduler stalled | Crawler API returns zero active jobs | | P2P network isolation | Network API returns zero or very low peer count | | Search response timeout | Search query exceeds 10-second timeout | | Index not growing | Status API shows flat urlCount over time | | Seed list not served | Seedlist URL returns empty or 404 | | Disk full — index writes failing | Indexer stops growing and status API shows errors |

A YaCy node that has isolated itself from the P2P network is invisible to the peer ecosystem — it stops receiving shared crawl data and its index ages in place. External monitoring with Vigilmon gives you a real-time signal when your YaCy admin interface, crawl scheduler, or peer network connectivity breaks — before your search results grow too stale for users to trust.

Start monitoring your YaCy node today — register free at vigilmon.online.