tutorial

How to Monitor Apache Any23 Semantic Web Data Extraction Health with Vigilmon

Apache Any23 RDF extraction hangs on malformed HTML, MIME type detection failures silently return empty triples, and format converter queues back up when parsing slow remote URLs. Learn how to monitor Any23 extraction service availability, parser health, and batch job liveness with Vigilmon HTTP probes and heartbeat monitors.

Apache Any23 (Anything To Triples) is an open-source library and web service that extracts structured semantic data from web documents in RDF, Microdata, JSON-LD, RDFa, Microformats, and CSV formats for knowledge graph ingestion pipelines and linked data workflows. Any23 wraps a chain of MIME type detectors and format parsers behind a single HTTP extraction endpoint — when a parser in the chain hangs on a malformed HTML document with a pathological CSS selector, the extraction thread blocks indefinitely while Any23's embedded Jetty server queues subsequent requests that never execute, exhausting the thread pool without logging a single error; when the Tika MIME type detector misidentifies a large CSV as binary data, Any23 silently returns an empty RDF graph instead of 503 — the calling pipeline receives a valid but empty response and ingests nothing, corrupting downstream knowledge graph completeness; when the Any23 service is deployed behind a reverse proxy that strips the Content-Type header, every extraction request falls through to the fallback text/html parser which returns syntactically valid but semantically meaningless triples. These are data pipeline integrity failures disguised as successful HTTP responses.

Vigilmon gives you external visibility into Any23's extraction layer through HTTP probe monitoring and heartbeat monitors. This tutorial covers both.

Why Any23 Needs External Monitoring

Any23's failure modes are subtle and data-corrupting:

  • Parser thread stall on malformed documents: Any23's HTML parser uses a CSS-based extractor chain; a circular reference in a malformed HTML document's inline JSON-LD causes the JSONLDExtractor to loop indefinitely, holding a Jetty worker thread — under concurrent extraction load, all threads stall and Any23 stops processing requests with no JVM crash or error log entry
  • Empty triple response on MIME detection failure: the Tika-based MIME detector occasionally misidentifies structured HTML documents with unusual encoding as binary/octet-stream; Any23 then skips all extractors and returns an empty NTriples response with HTTP 200 — the upstream pipeline sees a successful empty extraction, not an error
  • Memory exhaustion on large RDF documents: Any23 holds the full parsed RDF graph in memory before writing the response; a 50MB RDF/XML document parsed in a constrained heap causes an OutOfMemoryError that kills the JVM while the reverse proxy returns 502 to callers with no error body
  • Format converter queue backup: when Any23 is used as a batch converter via its CLI, large files fed faster than the converter threads can process cause the input queue to back up in memory — the CLI process appears running but conversion output stops being written
  • Service restart on configuration reload: Any23's Jetty-based service restarts when any23-core.properties is modified at runtime; during the restart window, all in-flight extraction requests receive TCP RST rather than a graceful 503, causing the upstream pipeline to treat the extraction as successful (no exception, just empty response body)
  • Extractor chain misconfiguration after upgrade: Any23 upgrades occasionally change the default extractor list in any23-default.properties; a missing html-mf-hcard extractor entry after an upgrade causes all hCard extractions to silently return empty triples

External monitoring with Vigilmon adds:

  • Proactive alerting when Any23's extraction endpoints return errors or slow responses
  • Parser health monitoring independent of the HTTP layer status
  • Heartbeat monitoring so you know when scheduled batch extraction and ETL pipeline jobs stop completing
  • Multi-region probe consensus that filters transient Tika detection blips from genuine Any23 service failures

Step 1: Build an Any23 Health Endpoint

Any23 ships with a REST extraction endpoint at /any23/. Build a health wrapper that exercises the parser chain end-to-end.

Java Health Servlet

// health/Any23HealthServlet.java
package com.yourorg.any23.health;

import org.apache.any23.Any23;
import org.apache.any23.extractor.ExtractionContext;
import org.apache.any23.http.AcceptHeaderBuilder;
import org.apache.any23.source.StringDocumentSource;
import org.apache.any23.writer.NTriplesWriter;
import javax.servlet.http.*;
import java.io.*;
import java.util.*;

public class Any23HealthServlet extends HttpServlet {

    private static final String PROBE_HTML =
        "<html><head>" +
        "<script type=\"application/ld+json\">" +
        "{\"@context\":\"https://schema.org\",\"@type\":\"Organization\",\"name\":\"Probe\"}" +
        "</script></head><body>Health probe</body></html>";

    @Override
    protected void doGet(HttpServletRequest req, HttpServletResponse resp)
            throws IOException {
        Map<String, String> checks = new LinkedHashMap<>();
        boolean anyFailed = false;

        // Test JSON-LD extractor with a known-good document
        try {
            Any23 any23 = new Any23();
            ByteArrayOutputStream out = new ByteArrayOutputStream();
            any23.extract(
                new StringDocumentSource(PROBE_HTML, "http://probe.local/", "text/html"),
                new NTriplesWriter(out)
            );
            String triples = out.toString("UTF-8");
            if (triples.contains("Organization") || triples.length() > 0) {
                checks.put("jsonld_extractor", "ok");
            } else {
                checks.put("jsonld_extractor", "empty_response");
                anyFailed = true;
            }
        } catch (Exception e) {
            checks.put("jsonld_extractor", "down:" + e.getClass().getSimpleName());
            anyFailed = true;
        }

        resp.setContentType("application/json");
        resp.setStatus(anyFailed ? 503 : 200);

        PrintWriter out = resp.getWriter();
        out.write("{\"status\":\"" + (anyFailed ? "degraded" : "ok") + "\",\"checks\":{");
        boolean first = true;
        for (Map.Entry<String, String> e : checks.entrySet()) {
            if (!first) out.write(",");
            out.write("\"" + e.getKey() + "\":\"" + e.getValue() + "\"");
            first = false;
        }
        out.write("}}");
    }
}

Node.js Sidecar

// health/any23.js
const express = require('express');
const http = require('http');
const querystring = require('querystring');

const app = express();
const ANY23_HOST = process.env.ANY23_HOST || 'localhost';
const ANY23_PORT = parseInt(process.env.ANY23_PORT || '8080');

const PROBE_BODY = querystring.stringify({
  url: 'data:text/html,<script type="application/ld+json">{"@context":"https://schema.org","@type":"Organization","name":"Probe"}</script>'
});

function probeAny23() {
  return new Promise((resolve, reject) => {
    const req = http.request(
      {
        host: ANY23_HOST,
        port: ANY23_PORT,
        path: '/any23/',
        method: 'POST',
        timeout: 10000,
        headers: {
          'Content-Type': 'application/x-www-form-urlencoded',
          'Accept': 'text/plain',
          'Content-Length': Buffer.byteLength(PROBE_BODY),
        },
      },
      res => {
        let body = '';
        res.on('data', chunk => { body += chunk; });
        res.on('end', () => resolve({ status: res.statusCode, body }));
      }
    );
    req.on('error', reject);
    req.on('timeout', () => { req.destroy(); reject(new Error('timeout')); });
    req.write(PROBE_BODY);
    req.end();
  });
}

app.get('/health/any23', async (req, res) => {
  try {
    const result = await probeAny23();
    const up = result.status === 200;
    const hasTriples = result.body && result.body.includes('<');

    if (!up) {
      return res.status(503).json({
        status: 'down',
        upstream_status: result.status,
      });
    }

    if (!hasTriples) {
      return res.status(503).json({
        status: 'degraded',
        detail: 'extractor returned empty response for known-good probe document',
      });
    }

    return res.status(200).json({ status: 'ok' });
  } catch (err) {
    return res.status(503).json({ status: 'down', error: err.message });
  }
});

app.listen(3020);

Python Health Sidecar

# health_any23.py
import os, requests
from flask import Flask, jsonify

app = Flask(__name__)

ANY23_BASE = f"http://{os.environ.get('ANY23_HOST', 'localhost')}:{os.environ.get('ANY23_PORT', '8080')}"

PROBE_HTML = (
    '<html><head><script type="application/ld+json">'
    '{"@context":"https://schema.org","@type":"Organization","name":"Probe"}'
    '</script></head><body>Health probe</body></html>'
)

@app.route('/health/any23')
def health():
    checks = {}
    any_failed = False

    # Test extraction endpoint with a known-good JSON-LD document
    try:
        resp = requests.post(
            f'{ANY23_BASE}/any23/',
            data={'body': PROBE_HTML},
            headers={'Accept': 'text/plain'},
            timeout=10
        )
        if resp.status_code != 200:
            checks['extraction'] = f'down:{resp.status_code}'
            any_failed = True
        elif not resp.text.strip():
            checks['extraction'] = 'degraded:empty_response'
            any_failed = True
        else:
            checks['extraction'] = f'ok:{len(resp.text.splitlines())}_triples'
    except Exception as e:
        checks['extraction'] = f'down:{e}'
        any_failed = True

    # Test service info endpoint
    try:
        resp = requests.get(f'{ANY23_BASE}/any23/info', timeout=5)
        checks['info_endpoint'] = 'ok' if resp.status_code == 200 else f'down:{resp.status_code}'
    except Exception as e:
        checks['info_endpoint'] = f'down:{e}'
        any_failed = True

    status = 'degraded' if any_failed else 'ok'
    return jsonify({'status': status, 'checks': checks}), 503 if any_failed else 200

if __name__ == '__main__':
    app.run(host='0.0.0.0', port=3020)

Step 2: Configure Vigilmon HTTP Monitor for Any23

  1. Log in to vigilmon.online and go to Monitors → New Monitor
  2. Choose HTTP / HTTPS
  3. Set the URL to your Any23 health endpoint: https://any23.your-org.example.com/health/any23
  4. Set the check interval to 1 minute
  5. Under Expected response, configure:
    • Status code: 200
    • Response body contains: "status":"ok"
    • Response time threshold: 10000ms
  6. Under Alert channels, assign your Slack or PagerDuty channel
  7. Save the monitor

Add a second monitor for the Any23 info endpoint to detect service restart cycles:

  • URL: https://any23.your-org.example.com/any23/info
  • Expected: status 200
  • Interval: 2 minutes
  • Alert channel: P1 pager — info endpoint down means the Jetty server is restarting or crashed

For high-throughput extraction deployments, add a third monitor targeting the extraction endpoint directly with a minimal probe:

  • URL: https://any23.your-org.example.com/any23/
  • Method: POST with body url=https://schema.org/Organization
  • Expected: status 200
  • Response time threshold: 15000ms
  • Interval: 5 minutes
  • Alert channel: Slack — slow extraction indicates thread pool pressure

Vigilmon's multi-region probe consensus filters transient Tika initialization warmup delays from genuine Any23 extraction service failures.

Step 3: Heartbeat Monitoring for ETL and Batch Extraction Jobs

Health endpoint monitoring catches service availability failures, but not end-to-end batch extraction pipeline problems. Any23 can respond to the info endpoint while a batch ETL job that processes thousands of documents per hour has silently stalled on a pathological URL that holds a parser thread indefinitely.

Vigilmon heartbeat monitors catch these: your Any23 ETL and batch extraction jobs ping Vigilmon after each successful execution cycle.

Set Up Heartbeat Monitors

For batch extraction pipeline liveness:

  1. In Vigilmon, go to Monitors → New Monitor → Heartbeat
  2. Set the name: any23-batch-extraction
  3. Set the expected interval: 30 minutes
  4. Set the grace period: 60 minutes
  5. Save — copy the heartbeat URL

For knowledge graph ingestion pipeline liveness:

  1. Create another Heartbeat monitor named any23-kg-ingestion
  2. Set the expected interval: 1 hour
  3. Set the grace period: 2 hours
  4. Save — copy the heartbeat URL

Wire Heartbeats to Any23 Batch Jobs

Add heartbeat pings to your Any23 batch extraction driver:

// batch/Any23BatchDriver.java
package com.yourorg.any23.batch;

import org.apache.any23.Any23;
import org.apache.any23.source.HTTPDocumentSource;
import org.apache.any23.writer.NTriplesWriter;
import java.net.http.*;
import java.net.URI;
import java.util.List;

public class Any23BatchDriver {

    private final HttpClient http = HttpClient.newHttpClient();
    private final String heartbeatUrl = System.getenv("VIGILMON_BATCH_HEARTBEAT_URL");
    private final Any23 any23 = new Any23();

    public void processBatch(List<String> urls) throws Exception {
        int processed = 0;
        for (String url : urls) {
            try {
                any23.extract(
                    new HTTPDocumentSource(any23.getHTTPClient(), url),
                    new NTriplesWriter(System.out)
                );
                processed++;
            } catch (Exception e) {
                System.err.println("Extraction failed for " + url + ": " + e.getMessage());
            }
        }
        System.out.println("Processed " + processed + "/" + urls.size() + " URLs");
        pingHeartbeat(heartbeatUrl);
    }

    private void pingHeartbeat(String url) {
        if (url == null) return;
        try {
            http.sendAsync(
                HttpRequest.newBuilder(URI.create(url)).GET().build(),
                HttpResponse.BodyHandlers.discarding()
            );
        } catch (Exception ignored) {}
    }
}

For shell-based ETL pipelines:

#!/bin/bash
# any23-etl-cron.sh — run by cron for periodic batch extraction
/opt/any23/bin/any23-batch.sh --input /data/urls.txt --output /data/triples/

if [ $? -eq 0 ] && [ -n "$VIGILMON_KG_HEARTBEAT_URL" ]; then
  curl -s "$VIGILMON_KG_HEARTBEAT_URL" > /dev/null 2>&1 || true
fi

Step 4: Alert Routing for Any23 Failures

Any23 failures range from empty triple responses that silently corrupt knowledge graph completeness to full service outages. Route alerts by severity:

| Monitor | Alert Channel | Priority | |---|---|---| | HTTP: health endpoint (extraction probe) | Slack + PagerDuty | P1 | | HTTP: Any23 info endpoint | Slack + PagerDuty | P1 | | HTTP: extraction endpoint response time | Slack | P2 | | Heartbeat: batch extraction pipeline | Slack + PagerDuty | P1 | | Heartbeat: knowledge graph ingestion | Slack + PagerDuty | P1 |

Set response time thresholds as early warning signals:

  • Alert at 8000ms for the health endpoint extraction probe (slow extraction indicates thread pool pressure or parser stall)
  • Alert at 1000ms for the info endpoint (slow info responses indicate Jetty restart or JVM GC pause)
  • Alert at 60 minutes gap in batch extraction heartbeat (stalled extraction means knowledge graph data goes stale)
  • Alert at 2 hours gap in knowledge graph ingestion heartbeat during business hours

For production linked data pipelines, set up a status page in Vigilmon showing Any23 service availability and batch extraction liveness — this lets data engineering teams identify extraction failures before they surface as missing or stale knowledge graph entries.

Summary

Any23 failures present as empty triple responses or silent parser stalls before they escalate to full extraction service outages — upstream knowledge graph pipelines receive HTTP 200 with empty RDF bodies while Any23's thread pool is exhausted by stalled parsers. External monitoring catches these before they corrupt knowledge graph completeness:

| Monitor Type | What It Covers | |---|---| | HTTP monitor on health endpoint | End-to-end extraction chain availability with known-good probe | | HTTP monitor on info endpoint | Any23 Jetty service availability and restart detection | | HTTP monitor on extraction endpoint | Parser response time and thread pool health | | Heartbeat monitor (batch extraction) | End-to-end batch ETL pipeline execution success | | Heartbeat monitor (KG ingestion) | Knowledge graph ingestion pipeline liveness |

Get started free at vigilmon.online — your first Any23 semantic extraction monitor is running in under two minutes.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles