tutorial

How to Monitor Stern with Vigilmon

Stern is a multi-pod and multi-container log tailing tool for Kubernetes. It lets you tail logs from all pods matching a pattern simultaneously, color-codes ...

Stern is a multi-pod and multi-container log tailing tool for Kubernetes. It lets you tail logs from all pods matching a pattern simultaneously, color-codes output by pod, and supports regex filtering so you can focus on the entries you care about across an entire deployment or namespace. Engineering teams use Stern for live debugging during deployments, troubleshooting incidents, and correlating log events across services without writing complex kubectl logs chains.

When Stern is deployed as a log-aggregation or log-forwarding agent — running persistently in the cluster to ship logs to an external system — its availability directly affects your log pipeline. In this tutorial you'll set up uptime and response-time monitoring for a Stern-based log forwarding deployment using Vigilmon — free tier, no credit card required.

Why a Stern log pipeline needs external monitoring

A Stern-based log pipeline can fail in ways that are invisible to the cluster:

  • Stern process crash — if the Stern binary inside a forwarding container crashes, pods continue running and writing to stdout, but those logs stop being shipped downstream; no Kubernetes event is emitted for a process crash inside a Running container
  • Regex filter mismatch after code changes — a log format change (different timestamp format, severity prefix) can cause Stern's regex filter to match nothing, silently dropping all log events
  • Target namespace or pod selector no longer matches — after a namespace rename or label change, Stern's selector matches zero pods and emits no logs with no error
  • Log destination unreachable — if the downstream log aggregator (Loki, Elasticsearch, S3) is unreachable, Stern has no built-in retry or dead-letter queue; log events are dropped
  • Stern liveness probe absent — unlike HTTP servers, a Stern forwarding container often lacks health probes; Kubernetes cannot distinguish a stuck process from a healthy one

External monitoring from Vigilmon watches the health endpoint of a Stern sidecar or wrapper deployment, giving you an independent signal that the log pipeline is alive and shipping.

What you'll need

  • Stern deployed as a log forwarding agent with an HTTP health wrapper, or a Stern sidecar deployment with a companion health endpoint
  • The health endpoint reachable externally or via Ingress
  • A free Vigilmon account

Step 1: Deploy Stern as a log forwarding agent with a health endpoint

Wrap Stern in a simple sidecar that exposes a health endpoint alongside the forwarding process:

# stern-forwarder.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: stern-forwarder
  namespace: logging
spec:
  replicas: 1
  selector:
    matchLabels:
      app: stern-forwarder
  template:
    metadata:
      labels:
        app: stern-forwarder
    spec:
      serviceAccountName: stern
      containers:
        - name: stern
          image: ghcr.io/stern/stern:latest
          args:
            - ".*"
            - "--namespace=production"
            - "--output=json"
            - "--timestamps"
          resources:
            requests:
              cpu: 100m
              memory: 128Mi
            limits:
              cpu: 500m
              memory: 512Mi

        - name: health-sidecar
          image: busybox:latest
          command:
            - sh
            - -c
            - |
              while true; do
                echo -e "HTTP/1.1 200 OK\r\nContent-Length: 2\r\n\r\nOK" | nc -l -p 8080 -q 1
              done
          ports:
            - containerPort: 8080
          readinessProbe:
            httpGet:
              path: /
              port: 8080
            initialDelaySeconds: 3
            periodSeconds: 10

Alternatively, use a purpose-built health wrapper script:

#!/bin/bash
# healthcheck.sh — run alongside stern, expose /healthz on port 8080
check_stern_alive() {
  if pgrep -x stern > /dev/null; then
    echo "OK"
    return 0
  else
    echo "DEAD"
    return 1
  fi
}

while true; do
  STATUS=$(check_stern_alive)
  CODE=$([[ "$STATUS" == "OK" ]] && echo "200 OK" || echo "503 Service Unavailable")
  printf "HTTP/1.1 $CODE\r\nContent-Length: ${#STATUS}\r\n\r\n$STATUS" | nc -l -p 8080 -q 1
done

Expose via Service and Ingress:

# stern-service.yaml
apiVersion: v1
kind: Service
metadata:
  name: stern-forwarder
  namespace: logging
spec:
  selector:
    app: stern-forwarder
  ports:
    - name: health
      port: 8080
      targetPort: 8080

# stern-ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: stern-ingress
  namespace: logging
spec:
  ingressClassName: nginx
  rules:
    - host: logs.yourdomain.com
      http:
        paths:
          - path: /stern
            pathType: Prefix
            backend:
              service:
                name: stern-forwarder
                port:
                  number: 8080

kubectl apply -f stern-forwarder.yaml
kubectl apply -f stern-service.yaml
kubectl apply -f stern-ingress.yaml

# Verify the health endpoint
curl https://logs.yourdomain.com/stern/healthz
# OK

Step 2: Instrument Stern log volume for monitoring

Add a log volume counter to detect when Stern stops shipping events even while the process is alive:

# stern-metrics-sidecar container addition
- name: metrics
  image: prom/pushgateway:latest
  args:
    - --web.listen-address=:9091
  ports:
    - containerPort: 9091

Periodically push a heartbeat metric from the Stern wrapper:

# heartbeat script — push a gauge every 60s as long as stern is running
while pgrep -x stern > /dev/null; do
  echo "stern_forwarder_alive 1" | \
    curl --data-binary @- http://localhost:9091/metrics/job/stern/instance/$(hostname)
  sleep 60
done
echo "stern_forwarder_alive 0" | \
  curl --data-binary @- http://localhost:9091/metrics/job/stern/instance/$(hostname)

Expose the Pushgateway metrics for Vigilmon:

curl https://logs.yourdomain.com/stern/pushgateway/metrics | grep stern_forwarder_alive
# stern_forwarder_alive{instance="stern-forwarder-abc12",job="stern"} 1

Step 3: Set up HTTP monitoring in Vigilmon

With Stern's health endpoint accessible, add it to Vigilmon:

  1. Log in to vigilmon.online and go to Monitors → New Monitor
  2. Choose HTTP / HTTPS
  3. Add a monitor for each endpoint:

| Monitor name | URL | Expected status | |---|---|---| | Stern health | https://logs.yourdomain.com/stern/healthz | 200 | | Stern metrics heartbeat | https://logs.yourdomain.com/stern/pushgateway/metrics | 200 |

  1. Set the check interval to 1 minute
  2. For the health monitor, under Expected response, match OK in the response body
  3. For the metrics heartbeat, match stern_forwarder_alive in the response body and confirm the value is 1
  4. Save each monitor

A health endpoint returning OK but the metrics heartbeat absent or showing 0 indicates Stern has crashed while the sidecar survives — the most important failure mode to detect separately.

Step 4: Monitor the Stern TCP port

Add a TCP-layer monitor:

  1. Go to Monitors → New Monitor
  2. Choose TCP Port
  3. Enter your Stern health sidecar hostname and port 8080
  4. Save the monitor

A TCP failure when pods are Running indicates a Service configuration issue or a health sidecar crash — both of which would leave the Stern log pipeline in an unknown state.

Step 5: Configure alert channels

Stern log pipeline failures mean log events are being silently dropped. This affects incident response time since engineers rely on those logs for debugging.

Email alerts

  1. Go to Alert Channels → Add Channel → Email
  2. Enter your logging infrastructure team or SRE on-call address
  3. Assign the channel to all Stern monitors

Webhook alerts for incident routing

  1. Go to Alert Channels → Add Channel → Webhook
  2. Enter your incident management webhook URL (PagerDuty, Opsgenie, Slack)
  3. The payload Vigilmon sends:
{
  "monitor_name": "Stern health",
  "status": "down",
  "url": "https://logs.yourdomain.com/stern/healthz",
  "started_at": "2024-01-15T10:23:00Z",
  "duration_seconds": 240
}

Wire this alert to a runbook that checks whether log events appear in the downstream aggregator after the alert time, restarts the Stern deployment, and validates that events resume flowing within two minutes.

Step 6: Correlate Vigilmon alerts with Stern diagnostics

When you receive a Stern downtime alert, run these checks:

# 1. Check pod status
kubectl get pods -n logging -l app=stern-forwarder

# 2. Check stern process inside the container
kubectl exec -n logging -l app=stern-forwarder -c stern -- pgrep -a stern

# 3. Check for crash loops
kubectl describe pod -n logging -l app=stern-forwarder | grep -A 5 "Last State\|Exit Code\|Reason"

# 4. Check stern logs for selector or RBAC errors
kubectl logs -n logging -l app=stern-forwarder -c stern --tail=100

# 5. Verify RBAC — stern needs pod/log read access
kubectl auth can-i get pods/log --as system:serviceaccount:logging:stern -n production

# 6. Verify selector still matches pods
kubectl get pods -n production --show-labels | head -20
# Confirm labels match stern's selector

# 7. Test stern manually in the failing namespace
kubectl run stern-debug --rm -it \
  --image=ghcr.io/stern/stern:latest \
  --serviceaccount=stern \
  -n logging \
  --restart=Never \
  -- stern ".*" -n production --tail=5 2>&1

# 8. Check downstream log destination
# For Loki:
curl http://loki.monitoring:3100/ready
# For Elasticsearch:
curl http://elasticsearch.monitoring:9200/_cluster/health

If Vigilmon shows the health endpoint responding but the log destination shows a gap in events, Stern is running but the downstream connection has failed — check for network policies between the logging namespace and the log aggregator.

Step 7: Create a status page for the log pipeline

Log pipeline availability directly affects incident response. Make Stern's status visible to engineering leadership:

  1. Go to Status Pages → New Status Page
  2. Name it: "Log Pipeline Health"
  3. Add your monitors: Stern health, metrics heartbeat
  4. Share the URL with your SRE, observability, and development teams

Summary

| What you set up | What it catches | |---|---| | HTTP monitor on /healthz | Stern process crash, pod failure | | HTTP monitor on metrics heartbeat | Stern alive but not shipping (silent drop) | | TCP monitor on port 8080 | NetworkPolicy blocks, health sidecar failure | | Email + webhook alert channels | Immediate notification when log pipeline stops | | Status page | Visibility into log pipeline health |

Stern provides real-time multi-pod log correlation that engineering teams depend on during incidents. External monitoring ensures that the log forwarding pipeline is always alive and shipping events so that when an incident occurs, logs are available and up to date.

Get started at vigilmon.online — free tier, no credit card required.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles