Monitoring Apache OpenWhisk with Vigilmon

Apache OpenWhisk distributes serverless function execution across Nginx, a Scala/Akka Controller, Apache Kafka, CouchDB, Invoker nodes, and Docker container pools. Each layer is a failure point: a Kafka consumer group falling behind means activations queue up silently; a cold start rate spike means your container pool is undersized; CouchDB disk exhaustion means all action metadata and activation logs stop persisting. Vigilmon gives you HTTP monitors for the always-on components and cron heartbeats for the internal metrics that OpenWhisk doesn't expose as health endpoints.

What You'll Set Up

Nginx entry point health monitoring (port 443/80)
OpenWhisk Controller liveness monitoring (port 8888)
Kafka broker availability and invoker consumer group lag
CouchDB health and disk usage monitoring (port 5984)
Invoker pod health and activation success rate
Docker container pool cold start rate heartbeat
Zookeeper availability monitoring
Activation queue depth alerts

Prerequisites

Apache OpenWhisk deployed (Docker Compose, Kubernetes, or bare-metal)
Access to the OpenWhisk API, CouchDB (port 5984), and Kafka
wsk CLI configured with your OpenWhisk credentials
A free Vigilmon account

Step 1: Monitor the Nginx Entry Point

Nginx is the public-facing entry point for all OpenWhisk API calls. TLS certificate expiry or an Nginx crash stops every user-facing invocation.

Add an HTTP monitor:

Log in to vigilmon.online and click Add Monitor.
Set Type to HTTP / HTTPS.
Enter: https://your-openwhisk-host/api/v1
Set Check interval to 1 minute.
Set Expected HTTP status to 200.
Enable Monitor SSL certificate and set alert threshold to 21 days.
Click Save.

Verify the endpoint:

curl -sk https://your-openwhisk-host/api/v1 | python3 -m json.tool
# Expected: {"support": {...}, "description": "OpenWhisk API"}

Step 2: Monitor the OpenWhisk Controller

The Controller (port 8888) handles authentication, action routing, and activation dispatch. Its /ping endpoint is the official liveness probe.

Add an HTTP monitor:

Add Monitor → HTTP / HTTPS.
URL: http://openwhisk-controller:8888/ping
Check interval: 1 minute.
Expected status: 200.
Keyword match: pong (the Controller responds with {"status":"up"} or similar).
Click Save.

Test manually:

curl -s http://openwhisk-controller:8888/ping
# Expected: {"status":"up"}

If you run multiple Controller instances (horizontal scale), add a monitor for each instance's /ping endpoint.

Step 3: Monitor Kafka Broker and Invoker Consumer Lag

OpenWhisk uses Kafka as the activation queue between the Controller and Invokers. If the Invoker consumer group falls behind, activations pile up and response times spike past the activation timeout.

Add HTTP monitors for each Kafka broker:

Add Monitor → HTTP / HTTPS.
If using Kafka with JMX-over-HTTP via Jolokia: URL: http://kafka-broker-1:8778/jolokia/read/kafka.server:type=BrokerTopicMetrics,name=MessagesInPerSec
Otherwise, use a sidecar health endpoint if available.

For consumer lag, deploy the kafka-consumer-lag-monitoring exporter or use kafka-consumer-groups.sh:

kafka-consumer-groups.sh \
  --bootstrap-server kafka:9092 \
  --describe \
  --group invokers

Add a cron heartbeat for Kafka lag:

#!/bin/bash
# check_openwhisk_kafka_lag.sh
HEARTBEAT_URL="https://vigilmon.online/heartbeat/YOUR_KAFKA_HEARTBEAT_ID"
BOOTSTRAP="kafka:9092"
GROUP="invokers"
LAG_THRESHOLD=1000  # activations

MAX_LAG=$(kafka-consumer-groups.sh \
  --bootstrap-server "$BOOTSTRAP" \
  --describe --group "$GROUP" 2>/dev/null | \
  awk 'NR>1 && $6 ~ /^[0-9]+$/ {print $6}' | \
  sort -rn | head -1)

if [ -z "$MAX_LAG" ]; then
  echo "Could not read consumer lag"
  exit 1
fi

if [ "$MAX_LAG" -gt "$LAG_THRESHOLD" ]; then
  echo "ALERT: Invoker consumer lag = $MAX_LAG (threshold: $LAG_THRESHOLD)"
  exit 1
fi

curl -s "$HEARTBEAT_URL" > /dev/null

Set the cron heartbeat interval to 2 minutes in Vigilmon.

Step 4: Monitor CouchDB Health and Disk Usage

CouchDB (port 5984) stores all OpenWhisk metadata: actions, rules, triggers, namespaces, and activation logs. CouchDB going down stops all new invocations (the Controller cannot authenticate requests without it).

Add an HTTP monitor for CouchDB:

Add Monitor → HTTP / HTTPS.
URL: http://couchdb:5984/_up
Check interval: 1 minute.
Expected status: 200.
Keyword match: ok (CouchDB responds with {"status":"ok"}).

Verify:

curl -s http://couchdb:5984/_up
# Expected: {"status":"ok"}

Monitor CouchDB disk usage:

curl -s http://admin:password@couchdb:5984/_node/_local/_stats | \
  python3 -c "
import sys, json
stats = json.load(sys.stdin)
# Check disk size of the whisk_local_activations db
"

# More directly, check disk per database:
curl -s http://admin:password@couchdb:5984/whisk_local_activations | \
  python3 -c "
import sys, json
d = json.load(sys.stdin)
sizes = d.get('sizes', {})
print(f'Active: {sizes.get(\"active\", 0) / 1024**3:.2f} GB')
print(f'File:   {sizes.get(\"file\", 0) / 1024**3:.2f} GB')
"

Add a cron heartbeat that checks disk usage across OpenWhisk CouchDB databases and only pings when usage is below 80% of available disk:

#!/bin/bash
HEARTBEAT_URL="https://vigilmon.online/heartbeat/YOUR_COUCH_HEARTBEAT_ID"
COUCH="http://admin:password@couchdb:5984"
DISK_THRESHOLD=80  # percent

# Get disk usage from the server node stats
DISK_INFO=$(df /var/lib/couchdb | awk 'NR==2 {print $5}' | tr -d '%')

if [ "$DISK_INFO" -gt "$DISK_THRESHOLD" ]; then
  echo "ALERT: CouchDB disk at ${DISK_INFO}%"
  exit 1
fi

curl -s "$HEARTBEAT_URL" > /dev/null

Step 5: Monitor Invoker Health and Activation Success Rate

Invokers pull activations from Kafka and execute them in Docker containers. An Invoker in offline state stops processing its Kafka partition, causing activations to queue indefinitely.

Check Invoker status via the OpenWhisk API:

# Using wsk CLI (requires admin credentials)
wsk -i system get
# Look for invoker health in the system properties

# Or check the Controller's internal state endpoint (if exposed):
curl -s http://openwhisk-controller:8888/invokers

Monitor activation success rate using the activations API:

#!/bin/bash
HEARTBEAT_URL="https://vigilmon.online/heartbeat/YOUR_INVOKER_HEARTBEAT_ID"
OW_HOST="https://your-openwhisk-host"
OW_AUTH="your_auth_key"
ERROR_THRESHOLD=5  # percent

# Get last 100 activations
ACTIVATIONS=$(wsk -i activation list --limit 100 --full 2>/dev/null)

TOTAL=$(echo "$ACTIVATIONS" | grep -c "activationId" || echo "0")
ERRORS=$(echo "$ACTIVATIONS" | grep -c '"statusCode": [^0]' || echo "0")

if [ "$TOTAL" -gt 0 ]; then
  ERROR_PCT=$(( ERRORS * 100 / TOTAL ))
  if [ "$ERROR_PCT" -gt "$ERROR_THRESHOLD" ]; then
    echo "ALERT: Activation error rate ${ERROR_PCT}% (${ERRORS}/${TOTAL})"
    exit 1
  fi
fi

curl -s "$HEARTBEAT_URL" > /dev/null

Step 6: Monitor Docker Container Pool and Cold Start Rate

OpenWhisk's Invoker maintains a warm container pool for fast function invocations. A high cold start rate indicates the pool is undersized relative to invocation concurrency.

Check warm container pool usage via Docker:

# List running OpenWhisk action containers
docker ps --filter "label=openwhisk.action" --format "{{.Names}} {{.Status}}"

# Count warm vs cold containers
WARM=$(docker ps --filter "label=openwhisk.action" -q | wc -l)
echo "Warm containers: $WARM"

Track cold starts from the activation logs:

# Count activations with initTime > 0 (cold starts) in last 10 minutes
wsk -i activation list --limit 100 --full 2>/dev/null | \
  python3 -c "
import sys, json
data = sys.stdin.read()
# Parse activation JSON objects
# An initTime annotation present and > 0 indicates a cold start
cold = sum(1 for line in data.split('\n') if 'initTime' in line and ':0,' not in line)
total = sum(1 for line in data.split('\n') if 'activationId' in line)
if total > 0:
    pct = cold / total * 100
    print(f'Cold start rate: {pct:.1f}% ({cold}/{total})')
    if pct > 30:
        print('ALERT: cold start rate > 30% — consider increasing container pool size')
"

Add a cron heartbeat with interval 5 minutes that alerts when cold start rate exceeds 30%.

Step 7: Monitor Zookeeper Health

OpenWhisk uses Zookeeper for Controller-Invoker coordination. Loss of Zookeeper quorum causes Controllers and Invokers to lose coordination, leading to duplicate or dropped activations.

Add HTTP monitors for Zookeeper using the mntr command over a simple TCP probe:

# Zookeeper health check via 4-letter command
echo mntr | nc zookeeper-1 2181 | grep zk_server_state
# Expected: zk_server_state leader OR zk_server_state follower

Expose Zookeeper health as an HTTP endpoint:

#!/bin/bash
# /usr/local/bin/zk_health_server.sh
while true; do
  ZK_STATE=$(echo mntr | nc -w2 zookeeper-1 2181 2>/dev/null | grep zk_server_state)
  if echo "$ZK_STATE" | grep -qE "leader|follower"; then
    echo -e "HTTP/1.1 200 OK\r\nContent-Type: text/plain\r\n\r\nOK: $ZK_STATE" | nc -l -p 9998 -q 1
  else
    echo -e "HTTP/1.1 503 Service Unavailable\r\nContent-Type: text/plain\r\n\r\nDEGRADED: $ZK_STATE" | nc -l -p 9998 -q 1
  fi
done

Add a Vigilmon HTTP monitor to http://zookeeper-host:9998 with expected status 200.

For a 3-node Zookeeper ensemble, add an HTTP monitor for each node.

Step 8: Monitor Activation Log Storage

The whisk_local_activations CouchDB database grows without bounds unless a purge job runs. Unbounded growth eventually exhausts disk and stops CouchDB.

Check activation record count and size:

curl -s http://admin:password@couchdb:5984/whisk_local_activations | \
  python3 -c "
import sys, json
db = json.load(sys.stdin)
print(f'Activation docs: {db[\"doc_count\"]:,}')
sizes = db.get('sizes', {})
print(f'Active size: {sizes.get(\"active\", 0) / 1024**2:.1f} MB')
if db['doc_count'] > 10_000_000:
    print('ALERT: activation records > 10M — check purge job')
"

Add a Vigilmon cron heartbeat that checks the activation record count and fails if the purge job hasn't run (indicated by count exceeding a threshold):

#!/bin/bash
HEARTBEAT_URL="https://vigilmon.online/heartbeat/YOUR_ACTIVATION_HEARTBEAT_ID"
COUCH="http://admin:password@couchdb:5984"
DOC_THRESHOLD=10000000

COUNT=$(curl -s "$COUCH/whisk_local_activations" | python3 -c "import sys,json; print(json.load(sys.stdin)['doc_count'])")

if [ "$COUNT" -gt "$DOC_THRESHOLD" ]; then
  echo "ALERT: $COUNT activation records (threshold: $DOC_THRESHOLD)"
  exit 1
fi

curl -s "$HEARTBEAT_URL" > /dev/null

Step 9: Configure Alerts in Vigilmon

Go to Alert Channels and add Slack, PagerDuty, or email.
Set thresholds per monitor:
- Nginx: alert after 1 failed check — user-facing API is fully down
- Controller /ping: alert after 1 failed check
- CouchDB /_up: alert after 1 failed check
- Zookeeper HTTP: alert after 2 failed checks
- Kafka lag heartbeat: alert when ping is 1× interval late
- Invoker success rate heartbeat: alert when ping is 1× interval late
- Cold start rate heartbeat: alert when ping is 2× interval late
- Activation storage heartbeat: alert when ping is 3× interval late

Step 10: Status Page for Operations

Status Pages → New Page.
Add monitor groups:
- API Layer: Nginx + Controller monitors
- Message Bus: Kafka broker + consumer lag heartbeat
- Persistent Storage: CouchDB health + disk + activation count heartbeats
- Execution Layer: Invoker health + cold start rate heartbeats
- Coordination: Zookeeper health monitors

Conclusion

Apache OpenWhisk's distributed architecture — Nginx, Akka Controller, Kafka activation bus, CouchDB metadata store, Docker-backed Invokers, and Zookeeper coordination — means a healthy deployment requires all layers to be monitored independently. A Kafka consumer falling behind doesn't trip CouchDB health checks; a cold start spike doesn't appear in Nginx logs. Vigilmon lets you cover each layer: direct HTTP monitors for the API-exposed components, and cron heartbeats that gate on the internal metrics OpenWhisk doesn't expose as health endpoints.

Start with Nginx, the Controller /ping, and CouchDB /_up — three monitors that cover the most impactful failure modes. Add Kafka lag and Invoker heartbeats once your alerting channels are configured.