tutorial

How to Monitor Tecton Feature Platform with Vigilmon

Tecton is a managed feature platform for machine learning that automates the entire feature lifecycle — from defining features in Python, through orchestrati...

Tecton is a managed feature platform for machine learning that automates the entire feature lifecycle — from defining features in Python, through orchestrating data pipelines that materialize features on schedule, to serving features at low latency for real-time inference. As a managed service, Tecton abstracts away most of the infrastructure, but the serving endpoints and pipeline health are still things your team needs to monitor from the outside.

This tutorial covers how to set up production monitoring for Tecton using Vigilmon — catching serving outages, pipeline delays, and API degradation before they silently corrupt your model predictions.

Why Tecton deployments need external monitoring

Tecton manages pipelines and infrastructure on your behalf, but there are several failure modes that Tecton's own observability won't surface quickly enough for production ML systems:

  • Feature serving API becomes unreachable — network routing issues, VPC peering misconfigurations, or temporary cloud provider issues can make the Tecton serving endpoint unreachable from your inference service even when Tecton itself reports healthy
  • Feature freshness violations — batch and stream pipelines can fall behind their expected schedules, causing models to serve features from stale time windows without raising any obvious error
  • Tecton SDK connectivity failures — your inference service uses the Tecton Python SDK to retrieve features at request time; if the SDK can't reach the Tecton API due to credential rotation or network changes, requests fail silently or raise exceptions that get swallowed by try/except blocks
  • Workspace API quota exhaustion — heavy feature retrieval traffic can exhaust API quotas; requests start failing with 429s that your application may not handle gracefully
  • Environment-specific outages — you may have separate Tecton workspaces for development, staging, and production; an issue in one doesn't mean the others are affected, and you need per-environment monitoring

External uptime monitoring from Vigilmon provides an independent view of availability that doesn't depend on Tecton's own dashboards or internal health checks.

What you'll need

  • A Tecton workspace with feature services deployed
  • Access to your Tecton serving endpoint URL (typically https://your-org.tecton.ai)
  • A lightweight health proxy (described below) or access to Tecton's HTTP API
  • A free Vigilmon account

Step 1: Create a feature serving health check wrapper

Tecton's feature serving uses a gRPC or HTTP API. The cleanest approach for external monitoring is a lightweight health check service that wraps Tecton SDK calls and exposes a standard /health endpoint:

from fastapi import FastAPI, Response
import tecton
import os

app = FastAPI()

# Initialize Tecton workspace connection
workspace = tecton.get_workspace(os.environ["TECTON_WORKSPACE"])

@app.get("/health")
async def health(response: Response):
    try:
        # List feature services as a lightweight connectivity probe
        services = workspace.list_feature_services()
        return {
            "status": "ok",
            "workspace": os.environ["TECTON_WORKSPACE"],
            "feature_services": len(services)
        }
    except tecton.TectonValidationError as e:
        response.status_code = 503
        return {"status": "degraded", "error": "validation_error", "detail": str(e)}
    except Exception as e:
        response.status_code = 503
        return {"status": "down", "error": str(e)}

@app.get("/health/serving")
async def serving_health(response: Response):
    """Test an actual feature retrieval to verify end-to-end serving."""
    try:
        feature_service = workspace.get_feature_service("your_feature_service")
        # Retrieve features for a known test entity
        result = feature_service.get_online_features(
            join_keys={"user_id": "test_health_check_user"}
        )
        return {"status": "ok", "features_returned": len(result.to_dict())}
    except Exception as e:
        response.status_code = 503
        return {"status": "down", "error": str(e)}

Deploy this as a sidecar to your inference service or as a standalone health proxy. It gives you two probe targets:

  • /health — can we connect to the Tecton workspace at all?
  • /health/serving — can we actually retrieve features end-to-end?

Step 2: Add HTTP monitors in Vigilmon

Once your health proxy is running:

  1. Log in to vigilmon.online and go to Monitors → New Monitor
  2. Choose HTTP / HTTPS
  3. URL: https://your-health-proxy.internal/health
  4. Interval: 1 minute
  5. Expected status: 200
  6. Optional: expected body contains "ok"
  7. Save

Add a second monitor for the deeper serving check:

  • URL: https://your-health-proxy.internal/health/serving
  • Interval: 2 minutes (longer interval since this makes a real Tecton API call)
  • Expected status: 200

Why multi-region consensus matters for ML serving

Vigilmon checks from multiple geographic regions simultaneously. For Tecton, this is valuable because:

  • A regional cloud incident might affect probes from one region while others still succeed — multi-region consensus prevents false alerts
  • If your inference services are deployed in multiple regions, each needs to be able to reach Tecton; multi-region monitoring reveals regional connectivity gaps early

Step 3: Monitor Tecton pipeline freshness with heartbeats

Tecton manages feature pipelines — batch jobs that run on a schedule and stream pipelines that continuously process events. When a pipeline falls behind, models are served stale features. This is the most insidious failure mode because everything appears to be running, but model quality degrades silently.

If you have a custom orchestrator (Airflow, Prefect, Dagster) triggering Tecton materializations, add a heartbeat ping after each successful run:

import requests
import tecton

def trigger_and_monitor_materialization(feature_view_name: str):
    workspace = tecton.get_workspace("production")
    feature_view = workspace.get_feature_view(feature_view_name)

    # Trigger batch materialization
    job = feature_view.run_materialization(
        start_time=yesterday(),
        end_time=today(),
        online=True,
        offline=True
    )

    # Wait for completion
    job.wait_for_completion()

    if job.state == "SUCCESS":
        # Ping Vigilmon heartbeat — only on success
        requests.get(
            "https://vigilmon.online/api/heartbeat/your-heartbeat-id",
            timeout=10
        )
    else:
        raise RuntimeError(f"Materialization job failed: {job.state}")

In Vigilmon:

  1. Go to Monitors → New Monitor → Heartbeat
  2. Name it tecton-batch-pipeline-{feature_view_name}
  3. Set the expected interval to your pipeline schedule (e.g., 1 hour for hourly features)
  4. Set the grace period to 15 minutes
  5. Save and copy the ping URL

Create one heartbeat per critical feature view. The heartbeat goes silent the moment a pipeline job fails or hangs — Vigilmon alerts you before the feature staleness has time to affect production model quality.

Step 4: Track feature freshness violations programmatically

Tecton's SDK exposes metadata about when features were last materialized. Build a separate freshness check service that Vigilmon can probe:

from fastapi import FastAPI, Response
from datetime import datetime, timedelta
import tecton

app = FastAPI()
workspace = tecton.get_workspace("production")

FRESHNESS_THRESHOLDS = {
    "user_transaction_features": timedelta(hours=1),
    "item_catalog_features": timedelta(hours=24),
    "real_time_user_activity": timedelta(minutes=15),
}

@app.get("/health/freshness")
async def freshness_check(response: Response):
    violations = []

    for feature_view_name, max_age in FRESHNESS_THRESHOLDS.items():
        fv = workspace.get_feature_view(feature_view_name)
        status = fv.get_online_store_metadata()

        last_materialized = status.last_online_store_update_time
        age = datetime.utcnow() - last_materialized

        if age > max_age:
            violations.append({
                "feature_view": feature_view_name,
                "last_updated": last_materialized.isoformat(),
                "age_minutes": int(age.total_seconds() / 60),
                "threshold_minutes": int(max_age.total_seconds() / 60)
            })

    if violations:
        response.status_code = 503
        return {"status": "stale", "violations": violations}

    return {"status": "fresh", "violations": []}

Add this as a third Vigilmon monitor:

  • URL: https://your-health-proxy.internal/health/freshness
  • Interval: 5 minutes
  • Expected status: 200

When this monitor goes down, you know features are stale even though the serving API is technically up.

Step 5: Configure alert routing

Route Tecton alerts to the right team:

  1. In Vigilmon, go to Alert Channels → Add Channel → Email
  2. Add your ML platform team's on-call email
  3. Assign to all Tecton monitors

For Slack:

  1. Go to Alert Channels → Add Channel → Webhook
  2. Paste your #ml-platform-oncall Slack webhook URL
  3. Assign to your monitors

A typical Vigilmon alert for a Tecton pipeline issue looks like:

{
  "monitor_name": "tecton-batch-pipeline-user-features",
  "status": "missing",
  "expected_every": "3600s",
  "last_seen_at": "2026-07-02T06:00:00Z",
  "missing_for_seconds": 7200
}

This tells your ML infrastructure team exactly which pipeline stopped reporting in and how long ago it last ran.

Step 6: Create an ML platform status page

Tecton serves multiple ML teams. A shared status page lets everyone see whether the feature platform is healthy without pinging the ML infra team directly:

  1. In Vigilmon, go to Status Pages → New Status Page
  2. Name it "ML Feature Platform — Tecton"
  3. Add monitors: workspace connectivity, feature serving, freshness check, per-pipeline heartbeats
  4. Group by: "Feature Serving", "Feature Pipelines", "Data Freshness"
  5. Publish

Share this URL in your ML platform documentation and your team's Slack channel description. Data scientists immediately know whether a model regression is a Tecton issue or something in their code.

Debugging guide when Tecton alerts fire

# 1. Check if the issue is network connectivity to Tecton
curl -s https://your-org.tecton.ai/health-check -H "Authorization: Bearer $TECTON_API_KEY"

# 2. Verify API key is still valid
tecton login --check

# 3. List feature services to isolate workspace-level issues
python -c "
import tecton
ws = tecton.get_workspace('production')
for svc in ws.list_feature_services():
    print(svc.name)
"

# 4. Check recent materialization job status
python -c "
import tecton
ws = tecton.get_workspace('production')
fv = ws.get_feature_view('user_transaction_features')
jobs = fv.get_materialization_status()
print(jobs)
"

# 5. Verify feature freshness directly
python -c "
import tecton
ws = tecton.get_workspace('production')
fv = ws.get_feature_view('user_transaction_features')
meta = fv.get_online_store_metadata()
print('Last updated:', meta.last_online_store_update_time)
"

Monitoring summary

| Monitor | Type | Interval | What it catches | |---------|------|----------|-----------------| | /health workspace probe | HTTP | 1 min | Tecton API unreachable, auth failure | | /health/serving end-to-end | HTTP | 2 min | Feature retrieval broken | | /health/freshness | HTTP | 5 min | Stale features in online store | | Per-pipeline heartbeat | Heartbeat | Per schedule | Pipeline job stopped running |

What's next

  • Incident history analysis — Vigilmon stores response time data so you can correlate Tecton API latency spikes with model performance degradation
  • Multiple workspace monitoring — create separate monitor groups for your dev, staging, and production Tecton workspaces
  • SSL certificate alerts — if your health proxy or any Tecton-adjacent service uses HTTPS, Vigilmon monitors cert expiry so a lapsed certificate doesn't cause a silent outage

Start free at vigilmon.online — no credit card, monitors are live in under two minutes.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles