tutorial

How to Monitor WasmCloud with Vigilmon (Lattice Connectivity, Actor Health, NATS Messaging)

WasmCloud is a WebAssembly application platform that runs portable, secure actor components across distributed hosts. Actors are Wasm modules that communicat...

WasmCloud is a WebAssembly application platform that runs portable, secure actor components across distributed hosts. Actors are Wasm modules that communicate with the outside world through capability providers — pluggable components for HTTP, databases, key-value stores, and more. The glue holding everything together is a NATS-based messaging layer called the lattice. Hosts discover each other, actors invoke capability providers, and RPC calls flow through NATS subjects automatically.

When the lattice degrades, actors stop responding silently. A NATS outage doesn't throw a visible error at the HTTP edge — requests simply queue up, time out, or disappear depending on where in the call chain the failure occurred. A capability provider that loses its link to an actor looks healthy from the outside while every invocation fails internally. WasmCloud's distributed model shifts failures to places that conventional uptime monitors never look.

This tutorial shows you how to build a monitoring stack for WasmCloud using Vigilmon — covering NATS health, host availability, actor endpoint reachability, and Prometheus metrics.

Why WasmCloud needs external monitoring

Internal observability tools like wash and the built-in NATS JetStream dashboard are useful for debugging, but they can't tell you that your production actors are unreachable from the outside. The five failure modes that external monitoring catches and internal tooling misses:

  • NATS connectivity lost — The NATS server goes down or becomes partitioned. The WasmCloud lattice collapses entirely. No actor-to-actor communication, no capability provider invocations. The host processes are still running and ps shows nothing wrong.
  • Actor fails to instantiate — The Wasm component starts loading but crashes during initialization (bad module binary, missing import, memory allocation failure). The host reports the actor as scheduled but it never enters a healthy state.
  • Capability provider link broken — An actor's link to a database or HTTP capability provider is configured but silently dropped after a network event. The actor receives requests but every invocation returns an error it cannot surface externally.
  • Host process OOM — The WasmCloud host (wasmcloud_host) is killed by the kernel OOM killer because actor execution consumed too much memory. No graceful shutdown, no drain. Actors vanish mid-request.
  • wash / OCI registry unreachable — Deployment pipelines can't pull new actor images from the OCI registry. Running actors are unaffected but any deployment, restart, or scale-out operation fails silently until someone manually checks.

What you'll need

  • A running WasmCloud host (version 0.82 or later) — started with wash up or as a systemd service
  • A NATS server (typically started alongside WasmCloud via wash up)
  • The wash CLI installed and configured with credentials for your lattice
  • The nats CLI (nats-io/natscli) for lattice RPC tests
  • A free Vigilmon account — sign up takes under a minute, no credit card required

Step 1: Monitor NATS connectivity

NATS exposes an HTTP monitoring endpoint on port 8222 by default. Check it before setting up anything else — if NATS is unreachable, everything downstream is broken.

Test NATS health manually:

# Basic health check — returns HTTP 200 with JSON when NATS is up
curl -sf http://localhost:8222/healthz

# Detailed server info including connection counts and uptime
curl -s http://localhost:8222/varz | jq '{server_id: .server_id, uptime: .uptime, connections: .connections}'

# Check JetStream status (WasmCloud uses JetStream for durable messaging)
curl -s http://localhost:8222/jsz | jq '{enabled: .config.enabled, streams: .streams, consumers: .consumers}'

A healthy NATS server returns {"status":"ok"} from /healthz. If JetStream is disabled or shows zero streams, WasmCloud's durable message features will fail even if NATS itself responds.

Test lattice RPC directly using the NATS CLI to confirm the lattice subjects are reachable:

# Subscribe to the lattice event stream for your lattice ID (default: "default")
nats sub "wasmbus.evt.default.>" --count 5

# Ping all WasmCloud hosts on the lattice — you should see at least one reply
nats req "wasmbus.rpc.default.ping" "" --timeout 2s

# List capability providers on the lattice
wash get providers

If nats sub connects but receives nothing for 10+ seconds, the host processes are likely not joined to the lattice. If the connection itself times out, NATS is down.

Add a Vigilmon TCP monitor on your-server.example.com:4222 (the NATS client port) to catch connection-level failures, and an HTTP monitor on http://your-server.example.com:8222/healthz for application-level health. Both are needed — TCP confirms the port is open, HTTP confirms NATS is actually serving requests.

Step 2: Build a WasmCloud health proxy

WasmCloud doesn't expose a single consolidated health endpoint out of the box. The best approach is a lightweight HTTP proxy that shells out to wash and aggregates the results. Vigilmon then polls this proxy.

Install FastAPI and uvicorn:

pip install fastapi uvicorn

Save this as wasmcloud_health.py:

import subprocess
import json
from fastapi import FastAPI, Response

app = FastAPI()


def run_wash(args: list[str]) -> tuple[int, str]:
    """Run a wash command and return exit code and stdout."""
    result = subprocess.run(
        ["wash"] + args,
        capture_output=True,
        text=True,
        timeout=10,
    )
    return result.returncode, result.stdout


@app.get("/health")
def health_check(response: Response):
    issues = []

    # Check that at least one host is reachable on the lattice
    try:
        code, output = run_wash(["get", "hosts", "--output", "json"])
        if code != 0:
            issues.append("wash get hosts failed: non-zero exit code")
        else:
            try:
                hosts = json.loads(output)
                if not hosts:
                    issues.append("no WasmCloud hosts found on lattice")
                else:
                    for host in hosts:
                        actor_count = host.get("actors", 0)
                        if isinstance(actor_count, dict):
                            actor_count = len(actor_count)
                        host_id = host.get("id", "unknown")[:12]
                        if actor_count == 0:
                            issues.append(f"host {host_id} has zero actors")
            except json.JSONDecodeError:
                # wash may return table output without --output json support on older versions
                if "No hosts" in output or output.strip() == "":
                    issues.append("no WasmCloud hosts found on lattice")
    except subprocess.TimeoutExpired:
        issues.append("wash get hosts timed out — NATS or lattice unreachable")
    except FileNotFoundError:
        issues.append("wash CLI not found in PATH")

    # Check that capability providers are linked
    try:
        code, output = run_wash(["get", "links", "--output", "json"])
        if code != 0:
            issues.append("wash get links failed")
        else:
            try:
                links = json.loads(output)
                if not links:
                    issues.append("no capability provider links configured")
            except json.JSONDecodeError:
                pass  # older wash versions may not support JSON output for links
    except subprocess.TimeoutExpired:
        issues.append("wash get links timed out")

    if issues:
        response.status_code = 503
        return {"status": "degraded", "issues": issues}

    return {"status": "ok", "message": "WasmCloud lattice is healthy"}

Run the proxy:

uvicorn wasmcloud_health:app --host 0.0.0.0 --port 9100

Test it:

curl -s http://localhost:9100/health | jq .

When the lattice is healthy you'll see {"status":"ok","message":"WasmCloud lattice is healthy"}. When hosts are missing or links are broken, the endpoint returns HTTP 503 with a list of issues — which Vigilmon will detect immediately.

To run the proxy as a systemd service alongside WasmCloud:

# /etc/systemd/system/wasmcloud-health-proxy.service
[Unit]
Description=WasmCloud Health Proxy
After=network.target wasmcloud.service
Requires=wasmcloud.service

[Service]
ExecStart=/usr/local/bin/uvicorn wasmcloud_health:app --host 0.0.0.0 --port 9100
Restart=always
RestartSec=5
WorkingDirectory=/opt/wasmcloud
User=wasmcloud

[Install]
WantedBy=multi-user.target

systemctl daemon-reload
systemctl enable --now wasmcloud-health-proxy

Step 3: Monitor actor endpoints directly

HTTP-capability actors in WasmCloud expose real HTTP endpoints through the wasmcloud:httpserver capability provider. These are the same endpoints your users hit. Monitoring them directly from Vigilmon confirms the full request path — NATS connectivity, host health, actor execution, and capability provider link — in a single check.

If your actor exposes a /health route, add it to Vigilmon as an HTTP monitor directly. If it doesn't, add a minimal health route to your actor:

// In your Rust actor, add a /health handler
use wasmcloud_interface_httpserver::{HttpRequest, HttpResponse, HttpServer, HttpServerReceiver};

#[async_trait]
impl HttpServer for YourActor {
    async fn handle_request(&self, _ctx: &Context, req: &HttpRequest) -> RpcResult<HttpResponse> {
        if req.path == "/health" {
            return Ok(HttpResponse {
                status_code: 200,
                body: b"{\"status\":\"ok\"}".to_vec(),
                header: std::collections::HashMap::from([(
                    "content-type".to_string(),
                    vec!["application/json".to_string()],
                )]),
            });
        }
        // ... rest of your handler
    }
}

For actors you can't modify, use a synthetic check: confirm the actor's known endpoint returns the expected status code. If your actor serves POST /api/process, a simple GET / that returns 404 still confirms the actor is alive — a 404 from a running actor is different from a timeout from a dead one.

Vigilmon can verify both the status code and a substring in the response body. Set the expected body to "status":"ok" for health routes that return JSON.

Step 4: Monitor WasmCloud host metrics

The WasmCloud host binary exposes Prometheus metrics when started with the WASMCLOUD_OBS_METRICS_PORT environment variable set:

export WASMCLOUD_OBS_METRICS_PORT=9090
wash up

Or in a systemd unit file:

[Service]
Environment=WASMCLOUD_OBS_METRICS_PORT=9090
ExecStart=/usr/local/bin/wasmcloud_host

Key metrics to watch:

# Confirm metrics endpoint is live
curl -s http://localhost:9090/metrics | grep -E "^(wasmcloud_|nats_)"

# Actor instance counts per host
curl -s http://localhost:9090/metrics | grep actor_instance_count

# RPC errors — should be zero or low; spikes indicate lattice problems
curl -s http://localhost:9090/metrics | grep rpc_error_total

# Total invocations — a flatline here means actors are receiving no traffic
curl -s http://localhost:9090/metrics | grep invocation_total

Build a metrics health wrapper that reads these values and returns a simple pass/fail:

# Add to wasmcloud_health.py

import urllib.request

METRICS_URL = "http://localhost:9090/metrics"
RPC_ERROR_THRESHOLD = 10  # errors per scrape interval before alerting


def parse_metric(metrics_text: str, name: str) -> float | None:
    for line in metrics_text.splitlines():
        if line.startswith(name) and not line.startswith("#"):
            parts = line.split()
            if len(parts) >= 2:
                try:
                    return float(parts[1])
                except ValueError:
                    pass
    return None


@app.get("/metrics-health")
def metrics_health(response: Response):
    try:
        with urllib.request.urlopen(METRICS_URL, timeout=5) as resp:
            metrics_text = resp.read().decode()
    except Exception as e:
        response.status_code = 503
        return {"status": "degraded", "issues": [f"metrics endpoint unreachable: {e}"]}

    issues = []

    actor_count = parse_metric(metrics_text, "wasmcloud_actor_instance_count")
    if actor_count is not None and actor_count == 0:
        issues.append("actor_instance_count is 0 — no actors are running")

    rpc_errors = parse_metric(metrics_text, "wasmcloud_rpc_error_total")
    if rpc_errors is not None and rpc_errors > RPC_ERROR_THRESHOLD:
        issues.append(f"rpc_error_total is {rpc_errors:.0f} — lattice RPC failures detected")

    if issues:
        response.status_code = 503
        return {"status": "degraded", "issues": issues}

    return {
        "status": "ok",
        "actor_instance_count": actor_count,
        "rpc_error_total": rpc_errors,
    }

Restart the proxy and test:

curl -s http://localhost:9100/metrics-health | jq .

The /metrics-health endpoint is suitable for Vigilmon monitoring because it converts numeric Prometheus data into a binary healthy/degraded signal without requiring Vigilmon to understand Prometheus format.

Step 5: Add Vigilmon monitors

Log in to vigilmon.online and add the following monitors. All require your server to be publicly reachable, or you can use Vigilmon's private monitoring agent for internal addresses.

NATS health — HTTP monitor

  1. Go to Monitors → New Monitor
  2. Type: HTTP / HTTPS
  3. URL: http://your-server.example.com:8222/healthz
  4. Check interval: 30 seconds
  5. Expected status code: 200
  6. Expected body contains: ok
  7. Save

NATS port — TCP monitor

  1. Go to Monitors → New Monitor
  2. Type: TCP Port
  3. Host: your-server.example.com, Port: 4222
  4. Check interval: 30 seconds
  5. Save

WasmCloud lattice health proxy — HTTP monitor

  1. Go to Monitors → New Monitor
  2. Type: HTTP / HTTPS
  3. URL: http://your-server.example.com:9100/health
  4. Check interval: 30 seconds
  5. Expected status code: 200
  6. Expected body contains: "status":"ok"
  7. Save

Actor HTTP endpoint — HTTP monitor

  1. Go to Monitors → New Monitor
  2. Type: HTTP / HTTPS
  3. URL: http://your-server.example.com:8080/health (your actor's HTTP port)
  4. Check interval: 30 seconds
  5. Expected status code: 200
  6. Save

WasmCloud metrics health — HTTP monitor

  1. Go to Monitors → New Monitor
  2. Type: HTTP / HTTPS
  3. URL: http://your-server.example.com:9100/metrics-health
  4. Check interval: 60 seconds
  5. Expected status code: 200
  6. Save

For alert channels, go to Alert Channels → Add Channel and add email, Slack, or a webhook endpoint. Assign all five monitors to the same channel so any failure triggers a single alert.

Recommended alert thresholds

| Check | Signal | Action | |---|---|---| | NATS /healthz returns non-200 | Lattice is down | Page immediately — all actors are affected | | TCP port 4222 unreachable | NATS process dead or network partition | Page immediately | | Lattice health proxy returns 503 | Host missing or zero actors | Alert within 1 minute | | Actor endpoint timeout (>5s) | Actor not processing requests | Alert within 1 minute | | Metrics health returns 503 | RPC errors above threshold | Alert within 2 minutes | | NATS JetStream streams = 0 | Durable messaging broken | Alert within 5 minutes |

Use a 2-failure confirmation window in Vigilmon for the metrics health check to avoid false alarms from metric scrape timing. The NATS and actor checks should alert on the first failure — a NATS outage is never a transient blip worth waiting on.

For the lattice health proxy, set the timeout to 15 seconds in Vigilmon. The wash CLI makes NATS calls internally, and if the lattice is slow rather than dead, those calls may take several seconds to return a timeout of their own.

Conclusion

WasmCloud's lattice model is powerful but it creates a monitoring blind spot: the failure surface is distributed across NATS, hosts, actors, and capability provider links, none of which a simple port check will expose. The monitoring setup in this tutorial layers four signals — NATS health, lattice proxy status, actor endpoint reachability, and Prometheus metrics — to give you visibility into every layer where WasmCloud can fail silently.

The health proxy in Step 2 is the most valuable piece. Once it is running, a single Vigilmon HTTP check covers host discovery, actor scheduling, and capability provider links in one round trip. The other monitors add redundancy and catch failures that happen before the proxy's own checks run.

Get started at vigilmon.online — the free tier supports enough monitors to cover a full WasmCloud deployment, and setup takes less time than a single wash build cycle.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles