tutorial

Monitoring Apache Tika with Vigilmon

Apache Tika's content detection and extraction pipeline is invisible until it stalls — when the Tika server stops responding, content type detection fails, or metadata extraction times out, downstream pipelines silently break. Here's how to monitor Apache Tika availability, extraction throughput, and response latency with Vigilmon.

Apache Tika is a toolkit for detecting and extracting metadata and structured content from over a thousand file types — PDFs, Word documents, spreadsheets, images, archives, and more. It underpins search indexing pipelines, data lake ingestion jobs, digital asset management systems, and any workflow where you need to know what a file contains without opening it manually. When the Tika server process stops responding, runs out of memory parsing a malformed document, or its REST endpoint becomes saturated under heavy extraction load, every upstream pipeline that depends on it either stalls or receives empty extraction results — often without surfacing an explicit error to the caller. Vigilmon provides external monitoring for the Tika server REST API, extraction health, and response latency so you catch Tika failures before silent data loss accumulates in your indexing pipeline.

What You'll Set Up

  • Tika server REST API availability monitoring
  • Content detection endpoint health check
  • Extraction throughput and latency tracking
  • Memory pressure detection via JVM stats
  • Process liveness monitoring

Prerequisites

  • Apache Tika 2.x server running with the REST API (java -jar tika-server.jar)
  • Tika server accessible at a known URL (e.g., http://tika.internal:9998)
  • A sample test document available for extraction checks
  • A free Vigilmon account

Step 1: Monitor the Tika Server REST API

Tika's REST API exposes a root endpoint at / and a version endpoint at /version that can be used to confirm the server is alive. When the Tika JVM crashes, runs out of heap memory parsing a corrupt file, or the server process is killed by the OS OOM killer, these endpoints stop responding. Set up a basic availability monitor:

  1. In Vigilmon, click Add Monitor → HTTP/HTTPS.
  2. Set the URL to http://tika.internal:9998/version.
  3. Set Check interval to 1 minute.
  4. Under Expected response, add keyword Apache Tika — the version endpoint always returns this string in a healthy response.
  5. Set Timeout to 15 seconds.

For the full health baseline, also monitor the /detectors endpoint which lists all registered content detectors:

# Verify Tika server is serving detector metadata
TIKA_URL="http://tika.internal:9998"

DETECTORS=$(curl -sf --max-time 15 \
  -H "Accept: application/json" \
  "${TIKA_URL}/detectors" 2>/dev/null)

if echo "$DETECTORS" | python3 -c "
import sys, json
data = json.load(sys.stdin)
# Tika returns a nested structure with detector classes
sys.exit(0 if isinstance(data, dict) and len(data) > 0 else 1)
" 2>/dev/null; then
  echo "Tika detector registry healthy"
  curl -s "https://vigilmon.online/heartbeat/REPLACE_WITH_YOUR_TOKEN"
else
  echo "Tika detector registry unavailable or empty"
  exit 1
fi

Step 2: Monitor Content Detection

Tika's content type detection is a fast, stateless operation that should always succeed within milliseconds. Monitoring detection separately from full extraction lets you distinguish between a crashed Tika server and a server that's alive but too busy to accept new extraction jobs. Set up a synthetic detection check:

  1. In Vigilmon, click Add Monitor → Cron Heartbeat.
  2. Set Expected ping interval to 5 minutes.
  3. Copy the heartbeat URL (e.g., https://vigilmon.online/heartbeat/abc123).

Create the detection health check script:

#!/bin/bash
# /usr/local/bin/tika-detect-check.sh

HEARTBEAT_URL="https://vigilmon.online/heartbeat/abc123"
TIKA_URL="http://tika.internal:9998"
TIMEOUT=15

# Use a small inline test payload — a minimal PDF header
# Tika should detect this as application/pdf
TEST_PAYLOAD=$'\x25\x50\x44\x46\x2D\x31\x2E\x34'

DETECTED_TYPE=$(echo -n "$TEST_PAYLOAD" | curl -sf \
  --max-time "$TIMEOUT" \
  -X PUT \
  -H "Content-Disposition: attachment; filename=test.pdf" \
  "${TIKA_URL}/detect/stream" 2>/dev/null)

if [ -n "$DETECTED_TYPE" ]; then
  echo "Tika detection OK: detected type '${DETECTED_TYPE}'"
  curl -s "$HEARTBEAT_URL"
else
  echo "Tika detection failed: no response from /detect/stream"
  exit 1
fi

Install the cron job:

chmod +x /usr/local/bin/tika-detect-check.sh
(crontab -l 2>/dev/null; echo "*/5 * * * * /usr/local/bin/tika-detect-check.sh >> /var/log/tika-health.log 2>&1") | crontab -

Detection failures indicate a Tika server that's running but unable to process requests — this often precedes a full crash by 10–15 minutes as heap memory fills with partially-parsed documents.

Step 3: Monitor Extraction Latency

Full text extraction is computationally expensive for Tika — parsing a large PDF or a deeply nested Office document can take several seconds. When Tika is healthy, extraction for typical documents should complete in well under 10 seconds. When the server is under memory pressure or a malformed document is consuming all available threads, extraction requests queue up and eventually time out. Monitor extraction latency with a known-good test document:

#!/bin/bash
# /usr/local/bin/tika-extraction-check.sh

HEARTBEAT_URL="https://vigilmon.online/heartbeat/def456"
TIKA_URL="http://tika.internal:9998"
TEST_FILE="/opt/tika-health/test-document.txt"
MAX_LATENCY_SECONDS=10
TIMEOUT=30

# Create a small test document if it doesn't exist
if [ ! -f "$TEST_FILE" ]; then
  mkdir -p /opt/tika-health
  echo "This is a Tika health check test document. Apache Tika extraction monitoring." > "$TEST_FILE"
fi

START_TIME=$(date +%s%3N)

EXTRACTED=$(curl -sf \
  --max-time "$TIMEOUT" \
  -X PUT \
  -H "Content-Type: text/plain" \
  --data-binary @"$TEST_FILE" \
  "${TIKA_URL}/tika" 2>/dev/null)

END_TIME=$(date +%s%3N)
LATENCY_MS=$(( END_TIME - START_TIME ))
LATENCY_S=$(( LATENCY_MS / 1000 ))

if [ -z "$EXTRACTED" ]; then
  echo "Tika extraction failed: no response (latency: ${LATENCY_MS}ms)"
  exit 1
fi

if [ "$LATENCY_S" -le "$MAX_LATENCY_SECONDS" ]; then
  echo "Tika extraction OK: ${LATENCY_MS}ms"
  curl -s "$HEARTBEAT_URL"
else
  echo "Tika extraction slow: ${LATENCY_MS}ms (threshold: ${MAX_LATENCY_SECONDS}s)"
  exit 1
fi

Set the Vigilmon heartbeat to 15 minutes. Extraction latency spikes above 10 seconds for simple documents are reliable early indicators of Tika JVM heap exhaustion.

Step 4: Monitor JVM Memory Pressure

Tika's most common production failure mode is running out of JVM heap memory. When parsing large or malformed documents — especially encrypted PDFs, nested ZIP files, or documents with embedded objects — Tika's parser threads hold large byte arrays in memory. If the server is exposed to arbitrary user-uploaded content without document size limits, heap exhaustion is inevitable. Monitor JVM memory via the Tika server metrics endpoint:

#!/bin/bash
# /usr/local/bin/tika-memory-check.sh

HEARTBEAT_URL="https://vigilmon.online/heartbeat/ghi789"
TIKA_URL="http://tika.internal:9998"
MAX_HEAP_PERCENT=85   # Alert if heap is over 85% used
TIMEOUT=15

# Tika server exposes JVM stats at /tika endpoint with Accept: application/json
# This can also be gathered via JMX if enabled
TIKA_STATS=$(curl -sf \
  --max-time "$TIMEOUT" \
  -H "Accept: application/json" \
  "${TIKA_URL}/version" 2>/dev/null)

if [ -z "$TIKA_STATS" ]; then
  echo "Tika stats endpoint unreachable"
  exit 1
fi

# For deployments with JMX REST bridge (e.g., Jolokia), check heap usage directly
# If not available, use process-level monitoring:
TIKA_PID=$(pgrep -f "tika-server" | head -1)

if [ -n "$TIKA_PID" ]; then
  # Check if process is consuming too much memory (over 80% of its RSS limit)
  PROCESS_MEM_KB=$(cat /proc/"$TIKA_PID"/status 2>/dev/null | grep VmRSS | awk '{print $2}')
  TOTAL_MEM_KB=$(grep MemTotal /proc/meminfo | awk '{print $2}')

  if [ -n "$PROCESS_MEM_KB" ] && [ -n "$TOTAL_MEM_KB" ]; then
    MEM_PERCENT=$(( PROCESS_MEM_KB * 100 / TOTAL_MEM_KB ))
    if [ "$MEM_PERCENT" -le "$MAX_HEAP_PERCENT" ]; then
      echo "Tika memory OK: ${MEM_PERCENT}% of system RAM"
      curl -s "$HEARTBEAT_URL"
    else
      echo "Tika memory pressure: ${MEM_PERCENT}% of system RAM (threshold: ${MAX_HEAP_PERCENT}%)"
      exit 1
    fi
  else
    echo "Tika process found (PID: ${TIKA_PID}) but could not read memory stats"
    curl -s "$HEARTBEAT_URL"
  fi
else
  echo "Tika server process not found"
  exit 1
fi

Set the Vigilmon heartbeat to 10 minutes. On Linux deployments, the OOM killer will terminate Tika before it reports errors — process disappearance is often the first and only signal.

Step 5: Monitor Process Liveness

When Tika is deployed as a system service (systemd, supervisor, or a container), the process manager may restart it automatically after a crash. Without external monitoring, these restarts are silent — your indexing pipeline may be degraded for 30–60 seconds on every restart without anyone being alerted. Set up a process liveness monitor:

#!/bin/bash
# /usr/local/bin/tika-liveness-check.sh

HEARTBEAT_URL="https://vigilmon.online/heartbeat/jkl012"
TIKA_URL="http://tika.internal:9998"
TIMEOUT=10

# Check the root endpoint — fastest possible liveness test
HTTP_CODE=$(curl -o /dev/null -sf \
  --max-time "$TIMEOUT" \
  -w "%{http_code}" \
  "${TIKA_URL}/" 2>/dev/null)

if [ "$HTTP_CODE" = "200" ]; then
  echo "Tika server alive: HTTP ${HTTP_CODE}"
  curl -s "$HEARTBEAT_URL"
else
  echo "Tika server not responding: HTTP ${HTTP_CODE:-timeout}"
  exit 1
fi

Run this every 2 minutes. For containerized Tika deployments, add a Docker health check as well:

HEALTHCHECK --interval=30s --timeout=10s --start-period=30s --retries=3 \
  CMD curl -f http://localhost:9998/version | grep -q "Apache Tika" || exit 1

Step 6: Set Up Alert Channels

Configure Vigilmon to route Tika alerts appropriately:

  1. Go to Alert Channels in Vigilmon.
  2. Add a Slack webhook to #data-engineering — Tika failures affect all ingestion and indexing pipelines.
  3. Add PagerDuty for the process liveness monitor — Tika going down is immediately breaking for upstream pipelines.
  4. Add email for the memory pressure monitor — this gives time to investigate before a crash.
  5. Set Consecutive failures before alert to 2 for extraction latency — a single slow check during GC pause is normal.

Add maintenance windows when restarting Tika to reload parser configurations:

curl -X POST https://vigilmon.online/api/maintenance \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -d '{
    "monitor_id": "TIKA_LIVENESS_MONITOR_ID",
    "duration_minutes": 5,
    "reason": "Tika server restart for configuration update"
  }'

sudo systemctl restart tika-server

Summary

| Monitor | Target | What It Catches | |---|---|---| | HTTP monitor (API) | /version keyword check | Server crash, JVM OOM kill, port unavailable | | Cron heartbeat (detection) | /detect/stream response | Server alive but unable to process, thread exhaustion | | Cron heartbeat (extraction) | /tika latency check | Heap pressure, slow GC, malformed-document backlog | | Cron heartbeat (memory) | Process RSS vs system RAM | Pre-crash memory saturation | | Cron heartbeat (liveness) | Root endpoint HTTP 200 | Fast crash detection, restart loops |

Apache Tika is a silent dependency for every pipeline that handles file content — search indexes, data lakes, digital asset managers, and content management systems all rely on it producing correct extraction results. When Tika fails silently, pipelines continue running but produce empty or incorrect metadata, which is often much harder to detect than an outright failure. Vigilmon gives you real-time visibility into Tika's health before silent data corruption becomes a production incident.

Start monitoring for free at vigilmon.online.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles