tutorial

Monitoring Nuclio Serverless Functions with Vigilmon

Nuclio is a high-performance serverless framework for real-time AI/ML inference and event-driven workloads. Here's how to monitor function health, invocation latency, auto-scaling, event source connectivity, and CI/CD pipelines with Vigilmon.

Nuclio is an open-source, high-performance serverless functions framework built for real-time AI/ML inference, event-driven microservices, and data pipelines. Unlike general-purpose serverless platforms (AWS Lambda, OpenFaaS), Nuclio achieves significantly higher throughput by passing data directly in memory between event sources and function replicas, supporting native in-process parallelism, and tightly integrating with high-throughput data streams like Kafka, NATS, and Kinesis. The result: sub-millisecond latency at scale — but also a richer set of components to monitor. Vigilmon gives you end-to-end visibility across the Nuclio control plane, function health, invocation latency, and deployment pipelines.

What You'll Set Up

  • Nuclio Dashboard health monitor (port 8070, function management UI and API)
  • Nuclio Controller health check (the Kubernetes operator managing Function CRDs)
  • Function invocation success rate and error rate alerting
  • Function invocation latency (P50/P95/P99) monitoring
  • Function replica auto-scaling health check (stuck at 0 replicas alert)
  • Event source connectivity monitoring (Kafka, NATS, RabbitMQ)
  • Function pod crash rate alerting
  • Container image build success rate monitoring
  • GPU function availability (for ML inference functions)

Prerequisites

  • Nuclio deployed on Kubernetes (or Docker for development)
  • The Nuclio Dashboard accessible at port 8070
  • kubectl access to the cluster namespace where Nuclio functions are deployed
  • A free Vigilmon account

Why Nuclio Monitoring Needs Specific Attention

Nuclio's performance model creates unique monitoring challenges:

  1. Function invocations are short-lived — a crashed function pod may recover before you notice, but it silently drops the events it was processing
  2. Auto-scaling to zero is a feature — but functions stuck at 0 replicas when traffic arrives means dropped events, not graceful queuing
  3. Event source connectivity is critical — a disconnected Kafka consumer causes function starvation without any obvious error in the function itself
  4. The build pipeline is the deployment path — build failures block function updates and can leave production running stale code
  5. GPU functions need resource monitoring — OOM on a GPU function crashes all inference replicas simultaneously

Each of these failure modes needs a distinct monitor.

Step 1: Monitor the Nuclio Dashboard

The Nuclio Dashboard (port 8070) is the control plane web UI and REST API for function management. When it's down, you cannot deploy new functions, view function status, or trigger manual invocations.

Add a Vigilmon HTTP monitor for the Dashboard health endpoint:

  1. Log in to vigilmon.online and click Add Monitor.
  2. Set Type to HTTP / HTTPS.
  3. Enter the Dashboard URL: http://<nuclio-dashboard-ip>:8070/api/functions
  4. Set Check interval to 1 minute.
  5. Set Expected HTTP status to 200.
  6. Under Alert thresholds, set Failure threshold to 2 consecutive failures.
  7. Click Save.

If your Nuclio Dashboard is behind an ingress or load balancer, use that external URL instead. You can also check the root path:

# Verify Dashboard is responding
curl -s http://localhost:8070/api/functions \
  -H "Content-Type: application/json" | python3 -m json.tool

Alert rule: critical if the Dashboard is unreachable for more than 2 consecutive checks — function deployment and monitoring are completely blocked.

Step 2: Monitor the Nuclio Controller

The Nuclio Controller is a Kubernetes operator that watches Function CRDs and reconciles the actual function deployments, auto-scaling, and event source bindings. If it crashes, new functions won't deploy, updated functions won't roll out, and auto-scaling events stop being processed.

Check controller health:

# Check Nuclio controller deployment
kubectl get deployment nuclio -n nuclio \
  -o jsonpath='{.status.availableReplicas}'

# View controller logs for reconciliation errors
kubectl logs -n nuclio -l app=nuclio --tail=50 | grep -i "error\|fail"

Add a Vigilmon heartbeat for the controller:

#!/bin/bash
# Runs as a Kubernetes CronJob every 2 minutes

CONTROLLER_REPLICAS=$(kubectl get deployment nuclio -n nuclio \
  -o jsonpath='{.status.availableReplicas}' 2>/dev/null || echo "0")

if [ "$CONTROLLER_REPLICAS" -gt 0 ]; then
  curl -s "$VIGILMON_CONTROLLER_HEARTBEAT_URL"
fi

Create the Vigilmon monitor:

  1. Click Add Monitor → Heartbeat.
  2. Name it nuclio-controller-health.
  3. Set Expected interval to 5 minutes.
  4. Copy the heartbeat URL into your CronJob script.

Alert rule: critical if the controller heartbeat is missed — function deployments and auto-scaling are frozen.

Step 3: Monitor Function Invocation Success Rate

Nuclio functions expose Prometheus metrics at /metrics on each function pod. The most important is the invocation success/error rate.

Query invocation metrics from the Nuclio Dashboard API:

# Get function status including invocation stats via Dashboard API
curl -s "http://localhost:8070/api/functions?namespace=nuclio" \
  -H "Content-Type: application/json" | \
  python3 -c "
import sys, json
funcs = json.load(sys.stdin)
for f in funcs:
    name = f['metadata']['name']
    status = f.get('status', {})
    print(f\"{name}: {status.get('state', 'unknown')}\")"

For per-function error rate, query the function's Prometheus metrics endpoint:

# Get invocation error count from function metrics
FUNCTION_POD=$(kubectl get pods -n nuclio \
  -l "nuclio.io/function-name=my-function" \
  -o jsonpath='{.items[0].metadata.name}')

kubectl exec -n nuclio "$FUNCTION_POD" -- \
  curl -s localhost:8080/metrics | grep "nuclio_processor_handled_events_total"

Set up a monitoring script that calculates error rate and sends a heartbeat when within bounds:

#!/usr/bin/env python3
import requests, os, json

DASHBOARD = os.environ.get("NUCLIO_DASHBOARD_URL", "http://localhost:8070")
NAMESPACE = os.environ.get("NUCLIO_NAMESPACE", "nuclio")
ERROR_THRESHOLD = 0.01  # 1% error rate

response = requests.get(f"{DASHBOARD}/api/functions",
                        params={"namespace": NAMESPACE})
functions = response.json()

all_healthy = True
for func in functions:
    state = func.get("status", {}).get("state", "unknown")
    if state not in ("ready", "scaled_to_zero"):
        print(f"Function {func['metadata']['name']} state: {state}")
        all_healthy = False

if all_healthy:
    requests.get(os.environ["VIGILMON_FUNCTION_HEALTH_HEARTBEAT_URL"])

Alert rule: critical if any production function has an error rate >1% over a 5-minute window.

Step 4: Monitor Function Invocation Latency

Nuclio's value proposition is low latency. A latency regression on a real-time inference function can violate SLAs and indicate resource contention, cold-start issues, or upstream data source problems.

Expose function latency via the Nuclio metrics endpoint:

# Get latency histogram from function pod
kubectl exec -n nuclio "$FUNCTION_POD" -- \
  curl -s localhost:8080/metrics | grep -E "nuclio_processor_latency"

For P99 latency alerting, use a heartbeat pattern:

#!/bin/bash
# Check P99 latency against a baseline threshold

P99_LATENCY=$(kubectl exec -n nuclio "$FUNCTION_POD" -- \
  curl -s localhost:8080/metrics \
  | grep 'nuclio_processor_worker_latency_microseconds{quantile="0.99"}' \
  | awk '{print $2}')

# Alert if P99 > 50ms (50000 microseconds)
if [ "${P99_LATENCY%.*}" -lt 50000 ]; then
  curl -s "$VIGILMON_LATENCY_HEARTBEAT_URL"
fi

Create a Vigilmon heartbeat monitor named nuclio-p99-latency with a 2-minute expected interval. If the P99 exceeds your threshold, the heartbeat stops and Vigilmon alerts.

Alert rule: warning if P99 latency is >2× the established baseline; critical if P99 latency is >5× baseline or exceeds an absolute SLA threshold.

Step 5: Monitor Function Replica Auto-Scaling

Nuclio can scale function replicas to zero when idle (saving resources) and scale up when traffic arrives. Functions stuck at 0 replicas — failing to scale up when events arrive — cause silent event drops.

Check scaling status:

# Check current replica counts for all functions
kubectl get deployments -n nuclio \
  -l "nuclio.io/function-version=latest" \
  -o custom-columns="NAME:.metadata.name,READY:.status.readyReplicas,DESIRED:.spec.replicas"

# Check for functions with traffic but 0 ready replicas
kubectl get pods -n nuclio -l "nuclio.io/function-version=latest" \
  --field-selector=status.phase=Running | wc -l

Add a monitoring script for scale-up failures:

#!/usr/bin/env python3
import subprocess, json, os, requests

# Get all function deployments
result = subprocess.run(
    ["kubectl", "get", "deployments", "-n", "nuclio",
     "-l", "nuclio.io/function-version=latest", "-o", "json"],
    capture_output=True, text=True
)
deployments = json.loads(result.stdout)["items"]

stuck_at_zero = []
for d in deployments:
    name = d["metadata"]["name"]
    desired = d["spec"].get("replicas", 0)
    ready = d["status"].get("readyReplicas", 0)
    
    # A function stuck at 0 when it has an event source configured
    # is a potential problem (unless intentionally scaled to zero)
    if desired > 0 and ready == 0:
        stuck_at_zero.append(name)

if not stuck_at_zero:
    requests.get(os.environ["VIGILMON_SCALING_HEARTBEAT_URL"])
else:
    print(f"Functions stuck at 0 replicas: {stuck_at_zero}")

Alert rule: critical if a function with a configured event source has 0 ready replicas and traffic is arriving — events are being dropped.

Step 6: Monitor Event Source Connectivity

Nuclio functions often consume events from Kafka, NATS, or RabbitMQ. If the event source connection drops, the function receives no events and appears healthy (no errors), but silently starves.

Check event source connectivity per function:

# Check Nuclio function event sources via Dashboard API
curl -s "http://localhost:8070/api/functions/my-kafka-function?namespace=nuclio" \
  | python3 -c "
import sys, json
f = json.load(sys.stdin)
triggers = f.get('spec', {}).get('triggers', {})
for name, trigger in triggers.items():
    print(f'{name}: {trigger.get(\"kind\")} - {trigger.get(\"attributes\", {}).get(\"brokers\", [])}')"

For Kafka event source health, test broker connectivity directly:

# Test Kafka broker reachability from within the function namespace
kubectl run -it --rm kafka-test \
  --image=confluentinc/cp-kafka:latest \
  --restart=Never -n nuclio -- \
  kafka-broker-api-versions --bootstrap-server kafka:9092

Add a Vigilmon HTTP monitor for each event source's health endpoint:

  • Kafka: Monitor your Kafka broker's JMX metrics endpoint or a Kafka REST Proxy health URL
  • NATS: Monitor http://<nats-server>:8222/healthz
  • RabbitMQ: Monitor http://<rabbitmq>:15672/api/healthchecks/node

For NATS:

  1. Click Add Monitor → HTTP / HTTPS.
  2. Enter: http://<nats-server>:8222/healthz
  3. Set Expected status to 200.
  4. Set Check interval to 1 minute.

Alert rule: critical if any event source connected to a production Nuclio function becomes unreachable — functions will receive no events.

Step 7: Monitor Function Pod Crash Rate

Repeated function pod crashes indicate code errors, OOM kills, or misconfigured resource limits. Nuclio restarts crashed pods automatically, but a crash loop means events are being dropped or delayed during the restart window.

Track pod restart counts:

# Get restart counts for all Nuclio function pods
kubectl get pods -n nuclio \
  -l "nuclio.io/function-version=latest" \
  -o custom-columns="NAME:.metadata.name,\
FUNCTION:.metadata.labels.nuclio\.io/function-name,\
RESTARTS:.status.containerStatuses[0].restartCount"

Set up a Vigilmon heartbeat that alerts on high restart counts:

#!/bin/bash
# Alert if any function pod has restarted more than 3 times in the past 10 minutes

HIGH_RESTARTS=$(kubectl get pods -n nuclio \
  -l "nuclio.io/function-version=latest" \
  -o jsonpath='{.items[*].status.containerStatuses[0].restartCount}' \
  | tr ' ' '\n' | awk '$1 > 3' | wc -l)

if [ "$HIGH_RESTARTS" -eq 0 ]; then
  curl -s "$VIGILMON_CRASHRATE_HEARTBEAT_URL"
fi

Alert rule: warning if any function pod has restarted >3 times in 10 minutes; critical if >5 restarts — the function is likely in a crash loop.

Step 8: Monitor Function Build Success Rate

Nuclio builds function container images from source when you deploy or update a function. A build failure blocks function updates and leaves production running the previous version.

Check build status via the Dashboard API:

# Get function build status
curl -s "http://localhost:8070/api/functions?namespace=nuclio" \
  | python3 -c "
import sys, json
funcs = json.load(sys.stdin)
for f in funcs:
    state = f.get('status', {}).get('state', 'unknown')
    name = f['metadata']['name']
    if 'error' in state.lower() or 'build' in state.lower():
        print(f'{name}: {state}')"

Add a Vigilmon heartbeat that tracks recent build outcomes:

#!/bin/bash
# Check for functions currently in a failed build state

FAILED_BUILDS=$(curl -s "http://localhost:8070/api/functions?namespace=nuclio" \
  | python3 -c "
import sys, json
funcs = json.load(sys.stdin)
count = sum(1 for f in funcs
            if 'error' in f.get('status',{}).get('state','').lower())
print(count)")

if [ "$FAILED_BUILDS" -eq 0 ]; then
  curl -s "$VIGILMON_BUILD_HEARTBEAT_URL"
fi

Alert rule: warning if any function has been in a failed build state for more than 10 minutes; critical if a production function's build has been failing for more than 30 minutes.

Step 9: Monitor GPU Function Health (ML Inference)

If your Nuclio functions use GPUs for inference (common in real-time ML pipelines), GPU OOM or unavailability crashes all inference replicas simultaneously.

Check GPU availability on function pods:

# Check GPU allocation for Nuclio function pods
kubectl get pods -n nuclio -l "nuclio.io/function-version=latest" \
  -o json | python3 -c "
import sys, json
pods = json.load(sys.stdin)['items']
for pod in pods:
    for container in pod.get('spec', {}).get('containers', []):
        resources = container.get('resources', {})
        gpu = resources.get('requests', {}).get('nvidia.com/gpu', 0)
        if gpu:
            name = pod['metadata']['name']
            print(f'{name}: GPU requested={gpu}')"

Monitor GPU memory usage via nvidia-smi in a monitoring sidecar:

#!/bin/bash
# Check GPU memory usage from within a GPU-enabled pod

GPU_MEM_USED=$(nvidia-smi --query-gpu=memory.used --format=csv,noheader,nounits)
GPU_MEM_TOTAL=$(nvidia-smi --query-gpu=memory.total --format=csv,noheader,nounits)
GPU_PCT=$((GPU_MEM_USED * 100 / GPU_MEM_TOTAL))

if [ "$GPU_PCT" -lt 90 ]; then
  curl -s "$VIGILMON_GPU_HEARTBEAT_URL"
fi

Alert rules:

  • warning at GPU memory >80%
  • critical at GPU memory >90% or GPU device unavailable — all inference requests will fail

Summary: Nuclio Monitoring Checklist

| Component | Monitor Type | Alert Level | Interval | |---|---|---|---| | Nuclio Dashboard (port 8070) | HTTP | Critical | 1 min | | Nuclio Controller (K8s operator) | Heartbeat | Critical | 2 min | | Function state (all functions) | Heartbeat | Critical | 2 min | | Invocation error rate >1% | Heartbeat | Critical | 2 min | | P99 latency regression | Heartbeat | Warning/Critical | 2 min | | Function replicas stuck at 0 | Heartbeat | Critical | 2 min | | Kafka/NATS/RabbitMQ connectivity | HTTP | Critical | 1 min | | Pod crash rate (>3 restarts/10min) | Heartbeat | Warning/Critical | 5 min | | Build failure rate | Heartbeat | Warning | 5 min | | GPU memory utilization | Heartbeat | Warning/Critical | 1 min |

Nuclio's performance advantage — direct memory passing, native parallelism, tight event source integration — is real, but it also means more components that can fail silently. Event source disconnections, scale-to-zero failures, and build pipeline errors won't cause immediate errors; they'll just silently drop throughput. With Vigilmon watching every layer from the control plane to per-function GPU memory, you keep Nuclio's performance benefits while maintaining the operational confidence to run it in production.

Start monitoring your Nuclio deployment with a free Vigilmon account.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles