How to Monitor Hopsworks Feature Store with Vigilmon

Hopsworks is a full-featured open-source ML platform built around a feature store. It bundles feature engineering, storage (both online and offline), model training, and serving into a single deployable platform. Because Hopsworks runs significant infrastructure — Kafka for stream ingestion, RonDB for online serving, Hive for offline storage, and multiple web services — there are many components that can independently fail.

Vigilmon gives you external uptime monitoring that watches Hopsworks from the outside, the same way your users and inference services see it. This tutorial walks through setting up comprehensive monitoring for a self-hosted Hopsworks deployment.

Why Hopsworks needs external monitoring

Hopsworks is a multi-component platform, and failure can occur at any layer:

• Hopsworks web application goes down — the main UI and REST API used by data scientists becomes unreachable; all platform operations halt
• Feature Store REST API fails — the HSFS (Hopsworks Feature Store) Python client communicates with a REST API; if this goes down, feature reads and writes from any client fail silently or with unhelpful network errors
• RonDB (online feature store) becomes unavailable — RonDB provides the sub-millisecond feature retrieval for online inference; an outage means real-time models can't fetch features
• Kafka lag grows unbounded — streaming feature pipelines write to Kafka topics that Hopsworks consumes; if consumer lag grows, online features become stale
• Feature materialization jobs fail — scheduled jobs that compute and write feature values stop running; features go stale
• Hopsworks Model Serving goes down — if you use Hopsworks as a model serving platform, inference endpoints become unreachable

Internal Hopsworks health dashboards tell you about internal state. External monitoring tells you whether your ML systems can actually reach and use the platform — which is the only thing that matters in production.

What you'll need

• A running Hopsworks instance (self-hosted on Kubernetes, AWS, Azure, or GCP)
• The Hopsworks REST API accessible at a known hostname
• A free Vigilmon account

Step 1: Confirm Hopsworks health endpoints

Hopsworks exposes health and status endpoints that you can probe directly:

# Check the main Hopsworks web application
curl -I https://hopsworks.your-domain.com/

# Check the Feature Store REST API
curl https://hopsworks.your-domain.com/hopsworks-api/api/version
# Returns: {"version": "3.x.x", ...}

# Check Hopsworks project API (requires API key)
curl -H "Authorization: ApiKey $HOPSWORKS_API_KEY" \
  https://hopsworks.your-domain.com/hopsworks-api/api/project

The /hopsworks-api/api/version endpoint is unauthenticated and is a good lightweight probe for API availability.

Step 2: Build a feature store health probe

For deeper monitoring, write a small Python health service that wraps hsfs (the Hopsworks Feature Store client):

from fastapi import FastAPI, Response
import hopsworks
import os

app = FastAPI()

@app.get("/health")
async def health(response: Response):
    try:
        project = hopsworks.login(
            host=os.environ["HOPSWORKS_HOST"],
            port=int(os.environ.get("HOPSWORKS_PORT", 443)),
            api_key_value=os.environ["HOPSWORKS_API_KEY"],
            project=os.environ["HOPSWORKS_PROJECT"]
        )
        return {"status": "ok", "project": project.name}
    except Exception as e:
        response.status_code = 503
        return {"status": "down", "error": str(e)}

@app.get("/health/feature-store")
async def feature_store_health(response: Response):
    try:
        project = hopsworks.login(
            host=os.environ["HOPSWORKS_HOST"],
            api_key_value=os.environ["HOPSWORKS_API_KEY"],
            project=os.environ["HOPSWORKS_PROJECT"]
        )
        fs = project.get_feature_store()
        feature_groups = fs.get_feature_groups("*")
        return {
            "status": "ok",
            "feature_groups": len(list(feature_groups))
        }
    except Exception as e:
        response.status_code = 503
        return {"status": "down", "error": str(e)}

@app.get("/health/online-store")
async def online_store_health(response: Response):
    """Verify RonDB online feature retrieval works end-to-end."""
    try:
        project = hopsworks.login(
            host=os.environ["HOPSWORKS_HOST"],
            api_key_value=os.environ["HOPSWORKS_API_KEY"],
            project=os.environ["HOPSWORKS_PROJECT"]
        )
        fs = project.get_feature_store()
        # Retrieve from a known feature view with a test entity
        fv = fs.get_feature_view("health_check_features", version=1)
        result = fv.get_feature_vector({"entity_id": "test-health-entity"})
        return {"status": "ok", "features_returned": len(result)}
    except Exception as e:
        response.status_code = 503
        return {"status": "down", "error": str(e)}

Deploy this as a lightweight sidecar service. It gives you three independently probeable health endpoints.

Step 3: Add monitors in Vigilmon

With your health probe running, add the following monitors:

Hopsworks API availability:

1. Log in to vigilmon.online → Monitors → New Monitor
2. Choose HTTP / HTTPS
3. URL: https://hopsworks.your-domain.com/hopsworks-api/api/version
4. Interval: 1 minute
5. Expected status: 200
6. Save

Feature Store connectivity:

• URL: https://health-proxy.internal/health/feature-store
• Interval: 2 minutes
• Expected status: 200
• Expected body contains: "ok"

Online store (RonDB) availability:

• URL: https://health-proxy.internal/health/online-store
• Interval: 2 minutes
• Expected status: 200

Multi-region monitoring for inference services

If your ML inference services are deployed across multiple cloud regions or availability zones, Vigilmon's multi-region probing confirms that Hopsworks is reachable from each location — not just from a single monitoring agent. Regional network partitions can silently isolate one deployment of your inference service from Hopsworks while the rest remain unaffected.

Step 4: Monitor Kafka with TCP probes

Hopsworks uses Kafka as the backbone for streaming feature ingestion. If Kafka brokers become unreachable, streaming feature pipelines stall and online feature freshness degrades.

Monitor Kafka broker availability at the TCP layer:

1. In Vigilmon → Monitors → New Monitor
2. Choose TCP Port
3. Enter your Kafka broker host and port: kafka-broker-1.hopsworks.internal / 9092
4. Interval: 1 minute
5. Save

Add TCP monitors for each Kafka broker in your cluster. For TLS-enabled Kafka:

# Verify Kafka broker port before adding to Vigilmon
nc -zv kafka-broker-1.hopsworks.internal 9093
# Connection to kafka-broker-1.hopsworks.internal 9093 port [tcp] succeeded!

Also monitor the Zookeeper ensemble (or KRaft controller) if applicable:

• zookeeper.hopsworks.internal:2181 — TCP

Step 5: Monitor RonDB with a TCP check

RonDB is the MySQL NDB Cluster used by Hopsworks for online feature serving. Monitor its SQL port directly:

1. In Vigilmon → Monitors → New Monitor → TCP Port
2. Host: rondb.hopsworks.internal
3. Port: 3306
4. Interval: 1 minute
5. Save

This catches RonDB failures at the network layer before they propagate to slow or silent failures in the feature serving layer.

Step 6: Monitor feature pipeline freshness with heartbeats

Hopsworks runs scheduled feature computation jobs (Spark jobs, Python jobs) that write feature values to both the offline and online stores. When these jobs stop, features go stale.

Configure your pipeline jobs to ping a Vigilmon heartbeat after successful completion:

import requests
import hopsworks
from datetime import datetime, timedelta

def run_user_activity_features_pipeline():
    project = hopsworks.login(
        host="hopsworks.your-domain.com",
        api_key_value=os.environ["HOPSWORKS_API_KEY"],
        project="production-ml"
    )
    fs = project.get_feature_store()
    fg = fs.get_feature_group("user_activity_features", version=1)

    # Read from your data source
    df = spark.sql("""
        SELECT user_id, COUNT(*) as event_count, ...
        FROM events
        WHERE event_date >= current_date() - 1
        GROUP BY user_id
    """)

    # Write to Hopsworks (both online and offline store)
    fg.insert(df, write_options={"online_enabled": True})

    # Only ping Vigilmon after successful insert
    requests.get(
        "https://vigilmon.online/api/heartbeat/your-heartbeat-id",
        timeout=10
    )

if __name__ == "__main__":
    run_user_activity_features_pipeline()

In Vigilmon:

1. Monitors → New Monitor → Heartbeat
2. Name: hopsworks-pipeline-user-activity-features
3. Expected interval: match your cron schedule (e.g., 1 hour)
4. Grace period: 20 minutes
5. Save and copy the ping URL

Create one heartbeat per critical feature group or pipeline. This way you know which specific pipeline stopped running when an alert fires, not just that "something broke."

Step 7: Monitor Hopsworks Model Serving

If you use Hopsworks to serve models (via KServe or its built-in serving infrastructure), each serving endpoint needs monitoring too:

# Check a deployed model endpoint
curl -H "Authorization: Bearer $SERVING_TOKEN" \
  https://hopsworks.your-domain.com/serving/v1/models/your_model:predict \
  -d '{"instances": [{"feature_1": 0.5}]}'

Add an HTTP monitor for each critical model endpoint:

• URL: https://hopsworks.your-domain.com/serving/v1/models/your_model
• Interval: 1 minute
• Expected status: 200

Step 8: Configure alert channels

1. In Vigilmon → Alert Channels → Add Channel → Email
2. Add your ML platform team on-call email
3. Assign to all Hopsworks monitors

For Slack:

1. Alert Channels → Add Channel → Webhook
2. Add your #ml-infra-alerts webhook URL
3. Assign to your monitors

Sample alert for a Hopsworks pipeline heartbeat miss:

{
  "monitor_name": "hopsworks-pipeline-user-activity-features",
  "status": "missing",
  "expected_every": "3600s",
  "last_seen_at": "2026-07-02T04:00:00Z",
  "missing_for_seconds": 5400
}

This fires 90 minutes after the last successful heartbeat — giving the pipeline 20 minutes of grace before alerting.

Step 9: Create a status page for your feature platform

1. In Vigilmon → Status Pages → New Status Page
2. Name: "Hopsworks ML Feature Platform"
3. Add monitors: Hopsworks API, Feature Store, Online Store, Kafka TCP, RonDB TCP, per-pipeline heartbeats, model serving endpoints
4. Group by: "Feature Serving", "Feature Pipelines", "Infrastructure", "Model Serving"
5. Publish

Share this page with data science teams and stakeholders. When a model behaves unexpectedly, the first question is usually "is the feature platform up?" — a public status page answers that immediately.

Debugging when Hopsworks alerts fire

# 1. Check Hopsworks services status
kubectl get pods -n hopsworks
# or for bare-metal/VM deployments:
sudo systemctl status hopsworks glassfish

# 2. Check Hopsworks application logs
kubectl logs -n hopsworks deployment/hopsworks -f
# or:
sudo tail -f /srv/hops/glassfish/domain1/logs/server.log

# 3. Verify RonDB is running
mysql -h rondb.hopsworks.internal -u root -p -e "SHOW STATUS LIKE 'Ndb_number_of_ready_data_nodes';"

# 4. Check Kafka consumer lag for feature pipelines
kafka-consumer-groups.sh \
  --bootstrap-server kafka-broker-1.hopsworks.internal:9092 \
  --group hopsworks-feature-ingestion \
  --describe

# 5. Test HSFS connectivity directly
python -c "
import hopsworks
project = hopsworks.login(
    host='hopsworks.your-domain.com',
    api_key_value='your-api-key',
    project='production-ml'
)
print('Connected:', project.name)
"

Recommended monitoring setup

| Monitor | Type | Interval | What it catches | |---------|------|----------|-----------------| | Hopsworks API /api/version | HTTP | 1 min | Platform web service down | | Feature Store connectivity | HTTP | 2 min | HSFS API unreachable | | Online store (RonDB) probe | HTTP | 2 min | Online feature retrieval broken | | Kafka broker TCP | TCP | 1 min | Streaming ingestion broken | | RonDB port TCP | TCP | 1 min | Online store port unreachable | | Per-pipeline heartbeats | Heartbeat | Per schedule | Feature pipeline stopped | | Model serving endpoints | HTTP | 1 min | Inference endpoint down |

What's next

• SSL certificate monitoring — Hopsworks uses TLS everywhere; Vigilmon alerts you before certificates expire to prevent silent outages
• Response time history — Vigilmon tracks feature serving latency over time; rising latency trends often predict failures
• Multi-environment separation — create separate monitor groups for development, staging, and production Hopsworks deployments

Get started free at vigilmon.online — no credit card, and your first monitor is live in under 90 seconds.