tutorial

Monitoring LabKey Server with Vigilmon

LabKey Server manages clinical study data, omics pipelines, and assay imports for biomedical research — a silent Tomcat crash or failed ETL pipeline can block critical study workflows. Here's how to monitor every layer with Vigilmon.

LabKey Server is an open-source platform for biomedical research data management, used by academic medical centers, pharmaceutical companies, and clinical study networks to handle patient data, assay imports, omics analysis, and regulatory submissions. It runs as a Java web application inside Apache Tomcat on port 8080. When LabKey Server's pipeline jobs fail silently or its Tomcat process crashes, study coordinators lose access to trial data and ETL pipelines stall — potentially blocking patient enrollment decisions. Vigilmon gives you full-stack visibility from Tomcat availability down to individual pipeline job health.

What You'll Set Up

  • Tomcat application server availability monitor
  • Pipeline job execution framework health check
  • Database connectivity monitor (PostgreSQL or MS SQL Server)
  • File storage module health check
  • ETL pipeline job status heartbeat
  • Assay data import service probe
  • R/Python scripting service health check
  • Study data versioning service monitor
  • Scheduled pipeline job heartbeat
  • Authentication service (LDAP/SSO) health check

Prerequisites

  • LabKey Server 23.x+ installed and running
  • Tomcat accessible on port 8080 (or your configured port)
  • Admin access to LabKey Server for health endpoint configuration
  • A free Vigilmon account

Step 1: Monitor the Tomcat Application Server

LabKey Server's main application runs inside Tomcat. The root URL returns the LabKey login page when healthy:

  1. Log in to vigilmon.online and click Add Monitor.
  2. Set Type to HTTP / HTTPS.
  3. Enter http://your-labkey-host:8080/labkey/home/project-begin.view.
  4. Set Check interval to 1 minute.
  5. Set Expected HTTP status to 200 (or 302 redirect to login).
  6. Click Save.

If you run LabKey behind a reverse proxy (nginx/Apache) with SSL:

  • Monitor https://your-labkey-domain.example.com instead.
  • Add a separate TCP Port check on 8080 (internal) to isolate Tomcat vs. proxy failures.

Step 2: Monitor the Pipeline Job Execution Framework

LabKey's pipeline framework runs ETL jobs, import tasks, and analysis workflows asynchronously. Monitor its health via the built-in status API:

  1. Add monitor — Type: HTTP / HTTPS.
  2. URL: http://your-labkey-host:8080/labkey/pipeline/getStatus.api.
  3. Expected status: 200.
  4. Body contains: "success":true.
  5. Check interval: 2 minutes.

This API call requires authentication. Create a service account with Reader permissions and pass Basic Auth credentials in the monitor's Authentication settings, or append an API key as a query parameter:

http://your-labkey-host:8080/labkey/pipeline/getStatus.api?apikey=YOUR_API_KEY

Step 3: Monitor Database Connectivity

LabKey Server requires a running PostgreSQL or MS SQL Server database. Monitor the database port directly:

PostgreSQL

  1. Add monitor — Type: TCP Port.
  2. Host: your-postgres-host, Port: 5432.
  3. Check interval: 1 minute.

MS SQL Server

  1. Add monitor — Type: TCP Port.
  2. Host: your-mssql-host, Port: 1433.
  3. Check interval: 1 minute.

For a deeper database connectivity check, use LabKey's admin console health endpoint (requires admin session):

http://your-labkey-host:8080/labkey/admin/healthCheck.view

This page reports database status, module state, and server info. Add it as an HTTP monitor with Body contains: Database: UP.

Step 4: Monitor the File Storage Module

LabKey stores pipeline inputs, assay files, and study documents in a configured file root. Disk full or mount failure breaks all file-dependent operations silently. Add a storage health heartbeat:

#!/bin/bash
# probe-labkey-storage.sh
LABKEY_FILES="/path/to/labkey/files"
THRESHOLD=90

USAGE=$(df "$LABKEY_FILES" | awk 'NR==2 {print $5}' | tr -d '%')
if [ "$USAGE" -lt "$THRESHOLD" ]; then
  touch "$LABKEY_FILES/.vigilmon_probe" && rm "$LABKEY_FILES/.vigilmon_probe"
  curl -fsS "https://vigilmon.online/api/push/YOUR_STORAGE_HEARTBEAT_KEY" > /dev/null
fi

*/5 * * * * /usr/local/bin/probe-labkey-storage.sh

In Vigilmon, add a Heartbeat monitor with Expected interval: 10 minutes.

Step 5: Monitor ETL Pipeline Job Status

Long-running ETL jobs that import data from source systems can fail without surfacing errors in the UI. Schedule a probe that checks for stuck or failed jobs via the API:

#!/bin/bash
# probe-labkey-etl.sh
LABKEY_URL="http://your-labkey-host:8080"
API_KEY="your-api-key"

# Get jobs with ERROR status created in the last 24h
ERRORS=$(curl -fsS \
  "$LABKEY_URL/labkey/pipeline-status/list.api?apikey=$API_KEY&StatusFile=ERROR" \
  | python3 -c "import sys,json; d=json.load(sys.stdin); print(d.get('rowCount',0))")

if [ "$ERRORS" -eq "0" ]; then
  curl -fsS "https://vigilmon.online/api/push/YOUR_ETL_HEARTBEAT_KEY" > /dev/null
fi

*/15 * * * * /usr/local/bin/probe-labkey-etl.sh

In Vigilmon, add a Heartbeat monitor with Expected interval: 20 minutes. If the heartbeat stops, it means a pipeline job has errored.

Step 6: Monitor the Assay Data Import Service

Assay data import (flow cytometry, mass spec, qPCR) goes through LabKey's assay transform scripts. Verify the assay module is responding:

  1. Add monitor — Type: HTTP / HTTPS.
  2. URL: http://your-labkey-host:8080/labkey/assay/assayList.api?apikey=YOUR_API_KEY.
  3. Expected status: 200.
  4. Body contains: "definitions".
  5. Check interval: 5 minutes.

A successful response with "definitions" confirms the assay module is loaded and database-accessible.

Step 7: Monitor R/Python Scripting Services

LabKey pipelines often invoke R or Python for statistical analysis. These scripting engines are configured separately and can silently stop working:

Add a LabKey admin check that validates the R engine:

http://your-labkey-host:8080/labkey/reports/validateDesign.view

More practically, schedule a test script run via the pipeline API and push a heartbeat on success:

# probe-labkey-rscript.sh
curl -fsS -X POST \
  "http://your-labkey-host:8080/labkey/pipeline/startAnalysis.api?apikey=YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"taskId":"org.labkey.api.pipeline.cmd.ScriptTask","path":"/labkey/server/test/rTest.R"}' \
  && curl -fsS "https://vigilmon.online/api/push/YOUR_RSCRIPT_HEARTBEAT_KEY" > /dev/null

Run this via cron every 30 minutes. In Vigilmon, set Expected interval to 35 minutes.

Step 8: Monitor the Study Data Versioning Service

LabKey Study modules version participant data snapshots for regulatory compliance. Monitor the study module's availability:

  1. Add monitor — Type: HTTP / HTTPS.
  2. URL: http://your-labkey-host:8080/labkey/study/getStudy.api?apikey=YOUR_API_KEY&containerPath=/StudyContainer.
  3. Expected status: 200.
  4. Body contains: "success":true.
  5. Check interval: 5 minutes.

Replace /StudyContainer with your study's container path. A failed response indicates the study module or its database tables are inaccessible.

Step 9: Scheduled Pipeline Job Heartbeats

LabKey can schedule recurring pipeline jobs (nightly data imports, report refreshes). Each critical scheduled job should push a heartbeat:

Add to each scheduled pipeline job's post-execution step:

# After successful pipeline job completion
curl -fsS "https://vigilmon.online/api/push/YOUR_JOB_HEARTBEAT_KEY"

In LabKey's pipeline XML task definitions:

<task id="postJobHeartbeat">
  <location>webserver</location>
  <run>
    <command>curl</command>
    <params>-fsS https://vigilmon.online/api/push/YOUR_JOB_HEARTBEAT_KEY</params>
  </run>
</task>

In Vigilmon, create one Heartbeat monitor per critical scheduled job with the expected job frequency plus 1 hour of grace.

Step 10: Monitor Authentication Services (LDAP/SSO)

LabKey supports LDAP and SAML SSO for institutional authentication. Monitor the LDAP port:

  1. Add monitor — Type: TCP Port.
  2. Host: your-ldap-server, Port: 389 (or 636 for LDAPS).
  3. Check interval: 2 minutes.

For SAML/SSO, monitor the identity provider metadata endpoint:

  1. Add monitor — Type: HTTP / HTTPS.
  2. URL: https://your-idp.example.com/saml/metadata.
  3. Expected status: 200.
  4. Body contains: EntityDescriptor.
  5. Check interval: 2 minutes.

Alerting Configuration

  1. In Vigilmon, go to Alert Channels and add email, Slack, or PagerDuty.
  2. Set Alert after: 2 consecutive failures for Tomcat and database monitors.
  3. Set Alert after: 1 failure for ETL pipeline heartbeats — a stuck pipeline is always urgent.
  4. Set Alert after: 1 missed heartbeat for scheduled job monitors.
  5. Enable Recovery alerts so study coordinators know when LabKey is back online.

Summary

| Monitor | Type | Interval | |---|---|---| | Tomcat / web UI | HTTP | 1 min | | Pipeline API | HTTP + keyword | 2 min | | PostgreSQL / MSSQL | TCP Port | 1 min | | File storage write test | Heartbeat | 10 min | | ETL job error check | Heartbeat | 20 min | | Assay import API | HTTP + keyword | 5 min | | R/Python scripting | Heartbeat | 35 min | | Study versioning API | HTTP + keyword | 5 min | | Scheduled pipeline jobs | Heartbeat | per job | | LDAP port | TCP Port | 2 min | | SAML IdP metadata | HTTP + keyword | 2 min |

LabKey Server sits at the center of clinical data workflows where a silent failure can delay trial results or block regulatory submissions. These monitors cover the full stack from Tomcat availability through individual pipeline job execution, giving your team the visibility to catch and fix issues before they affect study operations.

Sign up for a free Vigilmon account to get started.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles