Monitoring Apache POI with Vigilmon

Apache POI is a Java API for reading and writing Microsoft Office formats — it handles XLSX spreadsheets (XSSF), DOCX word documents (XWPF), PPTX presentations (XSLF), and the older binary OLE2 formats (XLS, DOC, PPT). POI is embedded in report generation services, data export pipelines, document conversion systems, ETL workflows, and enterprise applications that need to produce or consume Office files programmatically. When a large XLSX workbook pushes the JVM to its heap limit, a corrupted OLE2 compound document hangs the parser indefinitely, or the POI streaming writer (SXSSF) runs out of temporary disk space while writing a multi-million-row spreadsheet, the service continues accepting requests while delivering corrupted or truncated documents with no visible HTTP error. Vigilmon provides external monitoring for POI-based services, document generation endpoints, JVM memory pressure, and temporary disk usage so you detect pipeline degradation before your users receive broken Office files.

What You'll Set Up

• Document generation endpoint availability and output validation
• Excel workbook generation health checks (XLSX/XSSF)
• Word document generation health checks (DOCX/XWPF)
• JVM heap monitoring for POI services
• Temporary disk space monitoring for SXSSF streaming writer

Prerequisites

• An Apache POI-based service exposing HTTP endpoints for document generation or conversion
• JVM metrics accessible via Spring Boot Actuator, JMX, or a custom health endpoint
• A free Vigilmon account

Step 1: Monitor the Document Service HTTP Endpoint

POI is embedded in application servers — Spring Boot, Jakarta EE, or Quarkus services — that expose document generation over HTTP. The application's health endpoint provides the first availability signal. When the JVM runs out of heap during a large workbook generation, the whole process crashes and the HTTP endpoint stops responding. Set up a basic availability monitor:

1. In Vigilmon, click Add Monitor → HTTP/HTTPS.
2. Set the URL to your document service's health endpoint (e.g., http://docservice.internal:8080/actuator/health).
3. Set Check interval to 1 minute.
4. Under Expected response, add keyword UP — Spring Boot Actuator returns {"status":"UP"} when healthy.
5. Set Timeout to 15 seconds.

For a service without Spring Boot Actuator, monitor a lightweight ping endpoint:

# Verify document service is responding
DOC_SERVICE="http://docservice.internal:8080"

HTTP_CODE=$(curl -sf \
  --max-time 15 \
  -o /dev/null \
  -w "%{http_code}" \
  "${DOC_SERVICE}/health" 2>/dev/null)

if [ "$HTTP_CODE" = "200" ]; then
  echo "Document service available"
  curl -s "https://vigilmon.online/heartbeat/REPLACE_WITH_YOUR_TOKEN"
else
  echo "Document service unavailable: HTTP ${HTTP_CODE:-timeout}"
  exit 1
fi

Step 2: Monitor Excel Workbook Generation (XLSX)

POI's XSSF API generates XLSX workbooks in memory — a workbook with thousands of rows and formatted cells can require several hundred megabytes of heap. A silent failure mode is the service returning HTTP 200 with a zero-byte file or a file smaller than any valid XLSX (which is a ZIP archive containing XML files and has a minimum size even when empty). Set up an end-to-end generation check:

1. In Vigilmon, click Add Monitor → Cron Heartbeat.
2. Set Expected ping interval to 5 minutes.
3. Copy the heartbeat URL (e.g., https://vigilmon.online/heartbeat/abc123).

Create the XLSX generation health check:

#!/bin/bash
# /usr/local/bin/poi-xlsx-check.sh

HEARTBEAT_URL="https://vigilmon.online/heartbeat/abc123"
DOC_SERVICE="http://docservice.internal:8080"
TIMEOUT=45
MIN_XLSX_SIZE=1000  # bytes — any valid XLSX is larger than 1KB (it's a ZIP file)

TEMP_FILE=$(mktemp /tmp/poi-check-XXXXXX.xlsx)

HTTP_CODE=$(curl -sf \
  --max-time "$TIMEOUT" \
  -X POST \
  -H "Content-Type: application/json" \
  -H "Accept: application/vnd.openxmlformats-officedocument.spreadsheetml.sheet" \
  -d '{"template": "health-check", "rows": 10}' \
  -o "$TEMP_FILE" \
  -w "%{http_code}" \
  "${DOC_SERVICE}/api/excel/generate" 2>/dev/null)

FILE_SIZE=$(stat -c%s "$TEMP_FILE" 2>/dev/null || echo "0")

# Verify the file has a ZIP magic number (XLSX is a ZIP archive)
MAGIC=$(xxd -l 4 "$TEMP_FILE" 2>/dev/null | awk '{print $2$3}' | head -1)
rm -f "$TEMP_FILE"

if [ "$HTTP_CODE" = "200" ] && [ "$FILE_SIZE" -ge "$MIN_XLSX_SIZE" ] && [ "$MAGIC" = "504b0304" ]; then
  echo "XLSX generation healthy: HTTP ${HTTP_CODE}, size ${FILE_SIZE} bytes"
  curl -s "$HEARTBEAT_URL"
else
  echo "XLSX generation failed: HTTP ${HTTP_CODE}, size ${FILE_SIZE} bytes, magic ${MAGIC}"
  exit 1
fi

Install the cron job:

chmod +x /usr/local/bin/poi-xlsx-check.sh
(crontab -l 2>/dev/null; echo "*/5 * * * * /usr/local/bin/poi-xlsx-check.sh >> /var/log/poi-health.log 2>&1") | crontab -

The ZIP magic number check (504b0304) catches a common POI failure mode where the service writes a partial ZIP file — the HTTP response is 200 and the file is non-empty, but the XLSX cannot be opened by Excel because the ZIP structure is incomplete.

Step 3: Monitor Word Document Generation (DOCX)

POI's XWPF API generates DOCX word documents. DOCX is also a ZIP-based format (Office Open XML). The same truncation failure modes apply — a DOCX with a ZIP magic number check is a reliable minimum validity test. For services that generate both XLSX and DOCX, monitor them separately since they use different POI code paths:

#!/bin/bash
# /usr/local/bin/poi-docx-check.sh

HEARTBEAT_URL="https://vigilmon.online/heartbeat/def456"
DOC_SERVICE="http://docservice.internal:8080"
TIMEOUT=30
MIN_DOCX_SIZE=1000  # bytes

TEMP_FILE=$(mktemp /tmp/poi-docx-check-XXXXXX.docx)

HTTP_CODE=$(curl -sf \
  --max-time "$TIMEOUT" \
  -X POST \
  -H "Content-Type: application/json" \
  -H "Accept: application/vnd.openxmlformats-officedocument.wordprocessingml.document" \
  -d '{"template": "health-check", "content": "monitoring test document"}' \
  -o "$TEMP_FILE" \
  -w "%{http_code}" \
  "${DOC_SERVICE}/api/word/generate" 2>/dev/null)

FILE_SIZE=$(stat -c%s "$TEMP_FILE" 2>/dev/null || echo "0")
MAGIC=$(xxd -l 4 "$TEMP_FILE" 2>/dev/null | awk '{print $2$3}' | head -1)
rm -f "$TEMP_FILE"

if [ "$HTTP_CODE" = "200" ] && [ "$FILE_SIZE" -ge "$MIN_DOCX_SIZE" ] && [ "$MAGIC" = "504b0304" ]; then
  echo "DOCX generation healthy: HTTP ${HTTP_CODE}, size ${FILE_SIZE} bytes"
  curl -s "$HEARTBEAT_URL"
else
  echo "DOCX generation failed: HTTP ${HTTP_CODE}, size ${FILE_SIZE} bytes, magic ${MAGIC}"
  exit 1
fi

Set the heartbeat to 5 minutes.

Step 4: Monitor JVM Heap Usage

POI's XSSF API is heap-intensive by design — it builds the entire workbook object graph in memory before serializing to ZIP. A 10,000-row spreadsheet with formatting and formulas can consume 200-400MB of heap. Multiple concurrent generation requests can push the JVM into GC thrash or OOM. Monitor heap usage for early warning:

#!/bin/bash
# /usr/local/bin/poi-jvm-check.sh

HEARTBEAT_URL="https://vigilmon.online/heartbeat/ghi789"
DOC_SERVICE="http://docservice.internal:8080"
TIMEOUT=10
MAX_HEAP_PCT=80  # Alert when heap exceeds 80% — POI needs headroom for large documents

HEAP_RESPONSE=$(curl -sf \
  --max-time "$TIMEOUT" \
  "${DOC_SERVICE}/actuator/metrics/jvm.memory.used?tag=area:heap" 2>/dev/null)

HEAP_MAX_RESPONSE=$(curl -sf \
  --max-time "$TIMEOUT" \
  "${DOC_SERVICE}/actuator/metrics/jvm.memory.max?tag=area:heap" 2>/dev/null)

if [ -z "$HEAP_RESPONSE" ] || [ -z "$HEAP_MAX_RESPONSE" ]; then
  echo "JVM heap metrics not available"
  exit 1
fi

python3 << EOF
import json, sys
try:
    used = json.loads('''${HEAP_RESPONSE}''')['measurements'][0]['value']
    maximum = json.loads('''${HEAP_MAX_RESPONSE}''')['measurements'][0]['value']
    pct = (used / maximum) * 100
    print(f'JVM heap: {pct:.1f}% ({used/1024/1024:.0f}MB / {maximum/1024/1024:.0f}MB)')
    if pct > ${MAX_HEAP_PCT}:
        print(f'Heap pressure: {pct:.1f}% > ${MAX_HEAP_PCT}% threshold', file=sys.stderr)
        sys.exit(1)
    sys.exit(0)
except Exception as e:
    print(f'Metric parse error: {e}', file=sys.stderr)
    sys.exit(1)
EOF

if [ $? -eq 0 ]; then
  curl -s "$HEARTBEAT_URL"
else
  exit 1
fi

Install the cron job:

chmod +x /usr/local/bin/poi-jvm-check.sh
(crontab -l 2>/dev/null; echo "*/3 * * * * /usr/local/bin/poi-jvm-check.sh >> /var/log/poi-jvm.log 2>&1") | crontab -

Step 5: Monitor Temporary Disk for SXSSF Streaming Writer

POI's SXSSF (Streaming XLSX) API is designed for large spreadsheets — it flushes rows to a temporary file on disk rather than keeping the entire workbook in memory, keeping heap usage bounded. The tradeoff is that SXSSF requires disk space for temporary files proportional to the size of the final XLSX. When the temporary directory fills up (usually /tmp), SXSSF throws an IOException mid-write, producing a corrupted ZIP file without an HTTP error status. Monitor temporary disk space:

#!/bin/bash
# /usr/local/bin/poi-disk-check.sh

HEARTBEAT_URL="https://vigilmon.online/heartbeat/jkl012"
TEMP_DIR="/tmp"
MIN_FREE_MB=500  # Minimum 500MB free in tmp for SXSSF to work safely
TIMEOUT=5

FREE_MB=$(df -m "$TEMP_DIR" 2>/dev/null | awk 'NR==2 {print $4}')

if [ -z "$FREE_MB" ]; then
  echo "Cannot determine free space in ${TEMP_DIR}"
  exit 1
fi

if [ "$FREE_MB" -ge "$MIN_FREE_MB" ]; then
  echo "Temporary disk space OK: ${FREE_MB}MB free in ${TEMP_DIR}"
  curl -s "$HEARTBEAT_URL"
else
  echo "Temporary disk space low: ${FREE_MB}MB free in ${TEMP_DIR} (minimum: ${MIN_FREE_MB}MB)"
  exit 1
fi

Install the cron job:

chmod +x /usr/local/bin/poi-disk-check.sh
(crontab -l 2>/dev/null; echo "*/5 * * * * /usr/local/bin/poi-disk-check.sh >> /var/log/poi-disk.log 2>&1") | crontab -

Set the heartbeat to 5 minutes.

Step 6: Alert Configuration

Configure Vigilmon alerts for POI service incidents:

1. Open your document service monitors in the Vigilmon dashboard.
2. Click Alerts → Add alert channel.
3. Add your notification channels (email, Slack, PagerDuty).
4. For the HTTP endpoint monitor: set Alert after to 2 consecutive failures.
5. For the XLSX/DOCX generation monitors: set Alert after to 2 consecutive failures — a single corrupted output may be a transient issue.
6. For the JVM heap monitor: set Alert after to 1 failure — heap pressure above 80% in a POI service escalates quickly.
7. For the disk space monitor: set Alert after to 1 failure — disk full causes immediate document corruption.
8. Enable Recovery alerts to confirm the service is healthy after disk cleanup or a process restart.

Step 7: Validate Your Monitoring Setup

Test each monitor before your first production incident:

# Test XLSX generation check
chmod +x /usr/local/bin/poi-xlsx-check.sh
/usr/local/bin/poi-xlsx-check.sh
# Should print "XLSX generation healthy" with size and exit 0

# Test DOCX generation check
chmod +x /usr/local/bin/poi-docx-check.sh
/usr/local/bin/poi-docx-check.sh
# Should print "DOCX generation healthy" with size and exit 0

# Test disk space check
chmod +x /usr/local/bin/poi-disk-check.sh
/usr/local/bin/poi-disk-check.sh
# Should print free MB and exit 0 (unless /tmp is genuinely low)

Simulate a failure by temporarily restricting /tmp permissions:

# Simulate disk full by creating a test file that fills remaining space
# dd if=/dev/zero of=/tmp/fill bs=1M count=<remaining_mb>
# Run poi-disk-check.sh — should exit 1 and NOT ping the heartbeat
# Remove the fill file and verify the check passes again

What to Monitor in Production

| Monitor | Type | Interval | What it catches | |---------|------|----------|-----------------| | HTTP health endpoint | HTTP/HTTPS | 1 min | JVM crash, app server failure | | XLSX generation | Cron heartbeat | 5 min | XSSF failure, truncated ZIP, zero-byte output | | DOCX generation | Cron heartbeat | 5 min | XWPF failure, truncated ZIP | | JVM heap usage | Cron heartbeat | 3 min | Memory pressure, GC thrash, OOM risk | | Temporary disk space | Cron heartbeat | 5 min | SXSSF disk full, partial file writes |

Summary

Apache POI-based services fail in ways that standard availability monitoring misses: the HTTP layer stays healthy while the POI writer produces truncated ZIP files, heap exhaustion silently slows processing until a crash, and SXSSF disk exhaustion corrupts large spreadsheets mid-write with no HTTP error signal. External monitoring with Vigilmon covers the full failure surface: HTTP monitoring catches process-level failures, ZIP magic number checks on synthetic generation requests catch silent output corruption, JVM heap monitoring provides early warning before OOM crashes, and disk space monitoring prevents SXSSF write failures before users encounter corrupted Excel files. Deploy the HTTP and XLSX generation monitors first, add heap monitoring before heavy load, and add the disk check if your service uses SXSSF for large report generation.

Start monitoring Apache POI services for free with Vigilmon →