tutorial

How to Monitor Feast Feature Store with Vigilmon

Feast is the most widely deployed open-source feature store for machine learning. It provides a centralized registry for feature definitions, a pipeline for ...

Feast is the most widely deployed open-source feature store for machine learning. It provides a centralized registry for feature definitions, a pipeline for materializing features from offline sources into an online store, and a low-latency serving layer for real-time inference. When Feast breaks, it doesn't always fail loudly — models can silently serve stale features, feature pipelines can fall behind schedule, and the online store can become unreachable without any in-process exception being raised.

External monitoring with Vigilmon gives you a safety net that doesn't depend on Feast's own health reporting. This tutorial walks through setting up comprehensive monitoring for a Feast deployment — from the online feature server to materialization pipelines.

Why Feast needs external monitoring

Feast consists of several independently deployable components, and each can fail in its own way:

  • Online feature server crashes — the gRPC or HTTP server Feast runs for low-latency inference goes down; model inference requests fail or fall back to defaults silently
  • Materialization pipeline falls behind — the job that syncs features from your offline store (BigQuery, Snowflake, S3) to the online store (Redis, DynamoDB) stops running or lags hours behind; models serve features from the wrong time window
  • Feature registry becomes unreachable — SDK calls to store.get_online_features() throw exceptions because the registry (file, GCS, S3, SQL) can't be read
  • Redis / DynamoDB online store degraded — the backend that actually stores online feature values becomes slow or unavailable; feature retrieval latency spikes or requests time out
  • Feast UI stops responding — the web dashboard used by data scientists to browse features goes down; team productivity drops even though model serving still works

None of these failures are caught by simply checking if the Feast Python process is running. You need active probes against the serving endpoints and health routes.

What you'll need

  • A running Feast feature store (self-hosted or Feast on Kubernetes)
  • The Feast online feature server running (either the built-in server or a custom wrapper)
  • A free Vigilmon account — no credit card required

Step 1: Expose a health endpoint on the Feast online server

Feast's built-in online feature server exposes a /health endpoint when you start it with feast serve. Confirm it's reachable:

# Start the Feast online feature server
feast serve --host 0.0.0.0 --port 6566

# Verify the health endpoint
curl http://localhost:6566/health
# {"status": "up"}

If you're wrapping Feast in a FastAPI or Flask application for custom feature serving, add a health route that also checks the underlying online store:

from fastapi import FastAPI
from feast import FeatureStore
import redis

app = FastAPI()
store = FeatureStore(repo_path=".")

@app.get("/health")
async def health():
    try:
        # Attempt a lightweight connectivity check
        store.list_feature_views()
        return {"status": "ok", "registry": "reachable"}
    except Exception as e:
        return {"status": "degraded", "error": str(e)}, 503

@app.get("/health/online-store")
async def online_store_health():
    try:
        r = redis.Redis(host="your-redis-host", port=6379, socket_connect_timeout=2)
        r.ping()
        return {"status": "ok", "online_store": "reachable"}
    except Exception as e:
        return {"status": "down", "error": str(e)}, 503

This gives you two probe targets: one for the application layer and one for the online store backend.

Step 2: Add HTTP monitors in Vigilmon

With health endpoints in place, add monitors for each Feast component:

  1. Log in to vigilmon.online and go to Monitors → New Monitor
  2. Choose HTTP / HTTPS
  3. Set the URL to your Feast online server health endpoint: http://feast.internal:6566/health
  4. Set the check interval to 1 minute
  5. Under Expected response, set status code 200
  6. Save the monitor

Repeat for the online store health check:

  • URL: http://feast.internal:8000/health/online-store
  • Expected status: 200
  • Expected body contains: "ok"

Multi-region probing for serving infrastructure

The Feast online server is on the critical path for every model inference request. Vigilmon checks from multiple geographic regions simultaneously — if your Feast server becomes unreachable from any region where your inference services run, you'll know within 60 seconds.

Step 3: Monitor Redis (the online feature store backend)

Most production Feast deployments use Redis as the online store because of its sub-millisecond read latency. If Redis goes down, every feature lookup fails.

Monitor Redis availability at the TCP layer:

  1. In Vigilmon, go to Monitors → New Monitor
  2. Choose TCP Port
  3. Enter your Redis host and port: redis.internal / 6379
  4. Set interval to 1 minute
  5. Save the monitor

For Redis Cluster or Redis Sentinel, add TCP monitors for each node or the sentinel endpoints:

# Confirm Redis port is reachable before adding to Vigilmon
nc -zv redis.internal 6379
# Connection to redis.internal 6379 port [tcp/redis] succeeded!

The TCP monitor won't catch data corruption or replication lag, but it immediately alerts you when the port is unreachable — which is the failure mode that kills Feast feature serving completely.

Step 4: Monitor materialization pipeline freshness with heartbeats

Feast materialization jobs run on a schedule — typically via cron, Airflow, Prefect, or a Kubernetes CronJob — to sync features from your offline store to Redis. If a materialization job stops running, the online store silently serves stale features. Models keep running, but they're making predictions on outdated data.

Vigilmon's heartbeat monitors detect this. Configure your materialization job to ping a Vigilmon heartbeat URL after each successful run:

import requests
from feast import FeatureStore

def run_materialization():
    store = FeatureStore(repo_path="/app/feature_repo")

    # Run materialization for the last 24 hours
    store.materialize_incremental(end_date=datetime.utcnow())

    # Ping Vigilmon heartbeat — only reached on successful completion
    requests.get(
        "https://vigilmon.online/api/heartbeat/your-heartbeat-id",
        timeout=10
    )

if __name__ == "__main__":
    run_materialization()

In Vigilmon:

  1. Go to Monitors → New Monitor → Heartbeat
  2. Name it feast-materialization
  3. Set the expected interval to match your cron schedule (e.g., 24 hours)
  4. Set the grace period to 30 minutes
  5. Save and copy the heartbeat ping URL

Now if your materialization job fails, crashes, or simply stops being scheduled, Vigilmon alerts you — before data scientists notice their models drifting.

Step 5: Monitor the Feast UI

The Feast UI is a Streamlit or standalone web app that lets your team browse the feature registry, inspect feature statistics, and understand lineage. While not on the inference critical path, keeping it available improves team productivity and reduces friction for data scientists onboarding to your feature platform.

# Feast UI runs on port 8888 by default
feast ui --port 8888

# Check it's responding
curl -I http://feast-ui.internal:8888/
# HTTP/1.1 200 OK

Add an HTTP monitor in Vigilmon for http://feast-ui.internal:8888/ with expected status 200. Set the interval to 5 minutes since this is lower priority than inference serving.

Step 6: Configure alert channels

  1. In Vigilmon, go to Alert Channels → Add Channel → Email
  2. Add your ML platform team's on-call email
  3. Assign the channel to all Feast monitors

For Slack integration:

  1. Go to Alert Channels → Add Channel → Webhook
  2. Paste your Slack webhook URL
  3. Add the channel to your monitors

Example Vigilmon alert payload when Feast goes down:

{
  "monitor_name": "feast-online-server /health",
  "status": "down",
  "url": "http://feast.internal:6566/health",
  "started_at": "2026-07-02T08:15:00Z",
  "duration_seconds": 180
}

Route these alerts to your #ml-platform-alerts Slack channel so the ML infra team sees them immediately, even if the data science team hasn't yet noticed degraded model performance.

Step 7: Create a status page for your ML platform

If multiple teams consume your feature store, a status page lets data scientists and ML engineers self-serve incident information without filing tickets.

  1. In Vigilmon, go to Status Pages → New Status Page
  2. Name it "ML Feature Platform"
  3. Add your monitors: Feast online server, Redis, materialization heartbeat, Feast UI
  4. Group them logically: "Feature Serving", "Feature Pipeline", "Developer Tools"
  5. Publish the page

Share the status page URL in your ML platform documentation and Slack channel. When something breaks, teams immediately know whether it's a Feast issue or a problem in their own code.

Debugging with Vigilmon alert context

When a Feast alert fires, use this triage checklist:

# 1. Check if the Feast server process is running
ps aux | grep "feast serve"
systemctl status feast-server

# 2. Inspect the Feast server logs
journalctl -u feast-server -n 100 --no-pager

# 3. Check Redis connectivity
redis-cli -h your-redis-host ping
redis-cli -h your-redis-host info replication

# 4. Verify feature store registry is readable
cd /app/feature_repo && python -c "
from feast import FeatureStore
store = FeatureStore('.')
print(store.list_feature_views())
"

# 5. Check last successful materialization
redis-cli -h your-redis-host hgetall "__feast_meta__:driver_hourly_stats"

The combination of Vigilmon's timestamp (when the outage started) and these commands lets you quickly determine whether the issue is in the server process, the Redis backend, or the feature registry.

Recommended monitoring setup summary

| Monitor | Type | Interval | What it catches | |---------|------|----------|-----------------| | http://feast.internal:6566/health | HTTP | 1 min | Online server down, process crash | | http://feast.internal:8000/health/online-store | HTTP | 1 min | Online store connectivity | | redis.internal:6379 | TCP | 1 min | Redis port unreachable | | feast-materialization | Heartbeat | 24h | Materialization job stopped | | http://feast-ui.internal:8888/ | HTTP | 5 min | Feature registry UI down |

What's next

  • SSL monitoring — if your Feast online server is exposed over HTTPS, Vigilmon monitors the TLS certificate and alerts you before it expires
  • Response time tracking — Vigilmon records response time history so you can spot feature server latency trending up before it breaks SLAs
  • Multi-environment coverage — add separate monitors for your staging and production Feast deployments to catch issues in staging before they reach production

Get started free at vigilmon.online — no credit card required, and your first monitors are live in under two minutes.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles