tutorial

Monitoring Apache Joshua with Vigilmon

Apache Joshua's statistical machine translation pipeline is computationally intensive and failure-prone — when the decoder stalls, language model loading fails, or translation throughput drops, the system continues accepting requests while silently producing degraded or empty output. Here's how to monitor Apache Joshua availability, decoder health, and translation latency with Vigilmon.

Apache Joshua is a statistical machine translation (SMT) decoder that implements phrase-based, hierarchical phrase-based, and syntax-based translation models. It powers multilingual content pipelines, real-time translation APIs, and batch document translation workflows in research institutions and content-heavy enterprises. Joshua's translation quality depends on loading large language models and translation tables into memory — models that can exceed several gigabytes per language pair. When Joshua's server crashes, runs out of memory loading a language model, or its decoder threads become saturated with long-running translation jobs, the service either stops responding or continues accepting requests while producing garbage translations with no automatic error signaling. Vigilmon provides external monitoring for the Joshua HTTP server, decoder availability, and translation throughput so you detect service degradation before your translation pipeline silently fails.

What You'll Set Up

  • Joshua HTTP server availability monitoring
  • Decoder configuration and model load health
  • Translation throughput and latency tracking
  • Memory pressure from language model loading
  • Process liveness monitoring

Prerequisites

  • Apache Joshua 6.x with the server module enabled (--server-port)
  • Joshua accessible at a known endpoint (e.g., http://joshua.internal:5674)
  • A valid language pair configured (e.g., en-de)
  • A free Vigilmon account

Step 1: Monitor the Joshua HTTP Server

Joshua's built-in HTTP server accepts translation requests and returns JSON responses. When the Joshua process crashes — typically due to JVM heap exhaustion loading a large phrase table or language model — the port stops responding immediately. Since Joshua often takes several minutes to start up due to model loading, detecting crashes early is essential to minimizing total downtime. Set up a basic availability monitor:

  1. In Vigilmon, click Add Monitor → HTTP/HTTPS.
  2. Set the URL to http://joshua.internal:5674/ (or your configured server port).
  3. Set Check interval to 1 minute.
  4. Under Expected response, set HTTP status code to 200.
  5. Set Timeout to 20 seconds.

For a richer health check that verifies the Joshua server can produce translation output, add a keyword check on the translation endpoint:

# Verify Joshua server is responding to translation requests
JOSHUA_URL="http://joshua.internal:5674"

RESPONSE=$(curl -sf --max-time 20 \
  -G \
  --data-urlencode "q=hello" \
  "${JOSHUA_URL}/translate" 2>/dev/null)

if echo "$RESPONSE" | python3 -c "
import sys, json
data = json.load(sys.stdin)
# Joshua returns a 'data' key with translations
sys.exit(0 if 'data' in data and len(data['data'].get('translations', [])) > 0 else 1)
" 2>/dev/null; then
  echo "Joshua translation endpoint healthy"
  curl -s "https://vigilmon.online/heartbeat/REPLACE_WITH_YOUR_TOKEN"
else
  echo "Joshua translation endpoint not returning valid translations"
  exit 1
fi

Step 2: Monitor Translation Latency

Joshua's translation speed varies dramatically by input length, decoder beam width, and language pair complexity. For a healthy server under normal load, short sentences (under 30 words) should translate in well under 5 seconds. When the server is under memory pressure, when the language model has been partially evicted from cache, or when many concurrent translation requests are competing for decoder threads, latency climbs sharply before the service becomes unresponsive. Set up a latency monitor:

  1. In Vigilmon, click Add Monitor → Cron Heartbeat.
  2. Set Expected ping interval to 5 minutes.
  3. Copy the heartbeat URL (e.g., https://vigilmon.online/heartbeat/abc123).

Create the translation latency check:

#!/bin/bash
# /usr/local/bin/joshua-latency-check.sh

HEARTBEAT_URL="https://vigilmon.online/heartbeat/abc123"
JOSHUA_URL="http://joshua.internal:5674"
TEST_SENTENCE="The system is running correctly."
MAX_LATENCY_SECONDS=10
TIMEOUT=30

START_TIME=$(date +%s%3N)

RESPONSE=$(curl -sf \
  --max-time "$TIMEOUT" \
  -G \
  --data-urlencode "q=${TEST_SENTENCE}" \
  "${JOSHUA_URL}/translate" 2>/dev/null)

END_TIME=$(date +%s%3N)
LATENCY_MS=$(( END_TIME - START_TIME ))
LATENCY_S=$(( LATENCY_MS / 1000 ))

if [ -z "$RESPONSE" ]; then
  echo "Joshua translation request failed: no response (${LATENCY_MS}ms)"
  exit 1
fi

TRANSLATION_PRESENT=$(echo "$RESPONSE" | python3 -c "
import sys, json
try:
    data = json.load(sys.stdin)
    translations = data.get('data', {}).get('translations', [])
    print('yes' if len(translations) > 0 else 'no')
except:
    print('no')
" 2>/dev/null)

if [ "$TRANSLATION_PRESENT" = "yes" ] && [ "$LATENCY_S" -le "$MAX_LATENCY_SECONDS" ]; then
  echo "Joshua translation OK: ${LATENCY_MS}ms"
  curl -s "$HEARTBEAT_URL"
elif [ "$LATENCY_S" -gt "$MAX_LATENCY_SECONDS" ]; then
  echo "Joshua translation slow: ${LATENCY_MS}ms (threshold: ${MAX_LATENCY_SECONDS}s)"
  exit 1
else
  echo "Joshua returned response but no translations"
  exit 1
fi

Install the cron job:

chmod +x /usr/local/bin/joshua-latency-check.sh
(crontab -l 2>/dev/null; echo "*/5 * * * * /usr/local/bin/joshua-latency-check.sh >> /var/log/joshua-health.log 2>&1") | crontab -

Step 3: Monitor Language Model Loading

Joshua loads translation models (phrase tables, language models, and reordering tables) at startup. On large language pair configurations, this can take 5–15 minutes. If Joshua restarts due to a crash, the translation service is completely unavailable while models reload — but the process appears "alive" to basic process monitors because it has started. Detect prolonged model-loading states:

#!/bin/bash
# /usr/local/bin/joshua-ready-check.sh

HEARTBEAT_URL="https://vigilmon.online/heartbeat/def456"
JOSHUA_URL="http://joshua.internal:5674"
TIMEOUT=15

# A Joshua server that is loading models will either not accept connections
# or will respond with an error before it's ready to translate
RESPONSE=$(curl -sf \
  --max-time "$TIMEOUT" \
  -G \
  --data-urlencode "q=test" \
  "${JOSHUA_URL}/translate" 2>/dev/null)

EXIT_CODE=$?

if [ $EXIT_CODE -eq 0 ] && [ -n "$RESPONSE" ]; then
  # Check that the response is valid JSON with translation data
  VALID=$(echo "$RESPONSE" | python3 -c "
import sys, json
try:
    data = json.load(sys.stdin)
    print('yes' if 'data' in data else 'no')
except:
    print('no')
" 2>/dev/null)

  if [ "$VALID" = "yes" ]; then
    echo "Joshua server ready and translating"
    curl -s "$HEARTBEAT_URL"
  else
    echo "Joshua server responding but not ready: ${RESPONSE:0:100}"
    exit 1
  fi
else
  echo "Joshua server not ready: connection failed or timed out"
  exit 1
fi

Set the Vigilmon heartbeat to 10 minutes. If Joshua misses two heartbeats (20 minutes), it's either crashed or has been restarting and failing to load models — both require immediate investigation.

Step 4: Monitor Memory Usage

Joshua's memory footprint is dominated by loaded language models and phrase tables. A French–English system with a 5-gram language model and a large phrase table can easily consume 8–16 GB of RAM. On shared infrastructure or containers with memory limits, Joshua is routinely killed by the OOM killer — which is silent unless you are watching for the process to disappear. Monitor memory consumption:

#!/bin/bash
# /usr/local/bin/joshua-memory-check.sh

HEARTBEAT_URL="https://vigilmon.online/heartbeat/ghi789"
MAX_RSS_GB=14         # Alert if Joshua consumes more than 14 GB RSS
TIMEOUT=10

JOSHUA_PID=$(pgrep -f "joshua.decoder.JoshuaDecoder" | head -1)

if [ -z "$JOSHUA_PID" ]; then
  echo "Joshua process not found"
  exit 1
fi

# Read RSS in KB from /proc
RSS_KB=$(cat /proc/"$JOSHUA_PID"/status 2>/dev/null | grep "^VmRSS:" | awk '{print $2}')
RSS_GB=$(echo "scale=2; ${RSS_KB:-0} / 1048576" | bc 2>/dev/null || echo "0")
RSS_GB_INT=$(( ${RSS_KB:-0} / 1048576 ))

if [ "$RSS_GB_INT" -le "$MAX_RSS_GB" ]; then
  echo "Joshua memory OK: ${RSS_GB} GB RSS (PID: ${JOSHUA_PID})"
  curl -s "$HEARTBEAT_URL"
else
  echo "Joshua memory high: ${RSS_GB} GB RSS (threshold: ${MAX_RSS_GB} GB)"
  exit 1
fi

Set the Vigilmon heartbeat to 10 minutes. On systems with less RAM than the combined model footprint, Joshua will page-fault constantly, causing extreme latency spikes before the OOM killer terminates it.

Step 5: Monitor Translation Throughput

In batch translation workloads, Joshua processes a queue of documents and writes output to a file or database. A healthy Joshua server should sustain a predictable sentences-per-second rate for a given language pair and beam width. When throughput drops — due to garbage collection pressure, network I/O to a remote language model server, or decoder thread contention — batch jobs fall behind their SLA. Monitor throughput indirectly via a timed batch translation:

#!/bin/bash
# /usr/local/bin/joshua-throughput-check.sh

HEARTBEAT_URL="https://vigilmon.online/heartbeat/jkl012"
JOSHUA_URL="http://joshua.internal:5674"
MIN_SENTENCES_PER_SECOND=0.5  # Alert if throughput drops below 0.5 sentences/sec
BATCH_SIZE=10
TIMEOUT=60

# Build a small batch of test sentences
SENTENCES=(
  "The server is available."
  "All systems are operational."
  "The health check passed successfully."
  "Translation services are running normally."
  "The decoder is processing requests."
  "Language model loading completed."
  "The system responded within acceptable limits."
  "All monitors are reporting healthy status."
  "The service is ready to process requests."
  "Throughput is within normal parameters."
)

START_TIME=$(date +%s%3N)
SUCCESS_COUNT=0

for SENTENCE in "${SENTENCES[@]}"; do
  RESP=$(curl -sf \
    --max-time 10 \
    -G \
    --data-urlencode "q=${SENTENCE}" \
    "${JOSHUA_URL}/translate" 2>/dev/null)

  if echo "$RESP" | python3 -c "
import sys, json
try:
    d = json.load(sys.stdin)
    sys.exit(0 if len(d.get('data',{}).get('translations',[])) > 0 else 1)
except: sys.exit(1)
" 2>/dev/null; then
    SUCCESS_COUNT=$(( SUCCESS_COUNT + 1 ))
  fi
done

END_TIME=$(date +%s%3N)
ELAPSED_S=$(( (END_TIME - START_TIME) / 1000 ))
ELAPSED_S=$(( ELAPSED_S > 0 ? ELAPSED_S : 1 ))

THROUGHPUT=$(echo "scale=2; ${SUCCESS_COUNT} / ${ELAPSED_S}" | bc 2>/dev/null || echo "0")
THROUGHPUT_INT_X10=$(echo "scale=0; ${SUCCESS_COUNT} * 10 / ${ELAPSED_S}" | bc 2>/dev/null || echo "0")
MIN_X10=$(echo "scale=0; ${MIN_SENTENCES_PER_SECOND} * 10" | bc 2>/dev/null | cut -d. -f1 || echo "5")

if [ "$SUCCESS_COUNT" -eq "$BATCH_SIZE" ] && [ "${THROUGHPUT_INT_X10:-0}" -ge "${MIN_X10:-5}" ]; then
  echo "Joshua throughput OK: ${SUCCESS_COUNT}/${BATCH_SIZE} in ${ELAPSED_S}s (${THROUGHPUT}/s)"
  curl -s "$HEARTBEAT_URL"
else
  echo "Joshua throughput degraded: ${SUCCESS_COUNT}/${BATCH_SIZE} in ${ELAPSED_S}s (${THROUGHPUT}/s, min: ${MIN_SENTENCES_PER_SECOND}/s)"
  exit 1
fi

Run this every 10 minutes. Throughput below 0.5 sentences per second for simple sentences indicates severe decoder degradation.

Step 6: Set Up Alert Channels

Configure Vigilmon to route Joshua alerts to the right teams:

  1. Go to Alert Channels in Vigilmon.
  2. Add a Slack webhook to #translation-services — Joshua failures affect all multilingual content pipelines.
  3. Add PagerDuty for the server availability and liveness monitors — Joshua downtime means translation requests are failing for end users.
  4. Add email for the memory and throughput monitors — these allow proactive capacity planning before a crash.
  5. Set Consecutive failures before alert to 2 for latency — single slow checks during model cache warming are expected.

Include Joshua restart context in alerts:

ALERT_MSG=":globe_with_meridians: Apache Joshua translation service is unresponsive. Model reload takes 5-15 minutes after restart. Check: http://joshua.internal:5674/"

curl -X POST https://hooks.slack.com/services/YOUR/WEBHOOK/URL \
  -H 'Content-type: application/json' \
  --data "{\"text\": \"$ALERT_MSG\", \"channel\": \"#translation-services\"}"

Add a maintenance window before reloading Joshua with updated language models:

curl -X POST https://vigilmon.online/api/maintenance \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -d '{
    "monitor_id": "JOSHUA_SERVER_MONITOR_ID",
    "duration_minutes": 20,
    "reason": "Joshua restart to reload updated translation model"
  }'

sudo systemctl restart joshua-server

Summary

| Monitor | Target | What It Catches | |---|---|---| | HTTP monitor (server) | Root endpoint HTTP 200 | Server crash, port unavailable, JVM OOM kill | | Cron heartbeat (latency) | /translate round-trip time | Heap pressure, thread saturation, slow model cache | | Cron heartbeat (readiness) | /translate valid JSON response | Model still loading after restart | | Cron heartbeat (memory) | Process RSS vs threshold | Pre-crash memory growth, model size exceeding capacity | | Cron heartbeat (throughput) | Batch sentence translation rate | Decoder degradation, GC pressure, thread contention |

Apache Joshua's operational challenge is that translation failures are silent by design — a busy or degraded decoder continues returning responses, but the quality and completeness of those translations deteriorates without raising any explicit error. Vigilmon gives you external visibility into Joshua's responsiveness and throughput so you catch service degradation before your multilingual pipeline produces incorrect output at scale.

Start monitoring for free at vigilmon.online.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles