openelb_health.py

Prerequisites

• OpenELB 0.5.x or later installed in a Kubernetes cluster
• kubectl with cluster-admin access
• A free account at vigilmon.online

Part 1: Understanding what to monitor in OpenELB

OpenELB has three main failure surfaces:

1. Speaker pods — one speaker runs per node; if a speaker crashes, that node stops advertising routes
2. BGP sessions — if the BGP session to an upstream router drops, traffic can no longer reach services on that node
3. EIP pools — if the pool is exhausted, new LoadBalancer services stay in Pending with no external IP assigned

Monitoring only pod readiness misses BGP session state. All three need coverage.

Part 2: Build an OpenELB HTTP health endpoint

Create a lightweight health sidecar that queries the Kubernetes API for OpenELB status.

Create openelb_health.py

import os
import subprocess
import json
from fastapi import FastAPI, Response

app = FastAPI()

NAMESPACE = os.environ.get("OPENELB_NAMESPACE", "openelb-system")
SPEAKER_LABEL = os.environ.get("OPENELB_SPEAKER_LABEL", "app=openelb-speaker")


def kubectl_json(*args):
    result = subprocess.run(
        ["kubectl", "-n", NAMESPACE, *args, "-o", "json"],
        capture_output=True,
        text=True,
        timeout=15,
    )
    if result.returncode != 0:
        raise RuntimeError(result.stderr.strip()[:300])
    return json.loads(result.stdout)


@app.get("/health")
def health(response: Response):
    checks = {}

    # Speaker pod readiness
    try:
        pods = kubectl_json("get", "pods", "-l", SPEAKER_LABEL)
        items = pods.get("items", [])
        if not items:
            checks["speaker_pods"] = "error: no speaker pods found"
        else:
            not_ready = [
                p["metadata"]["name"]
                for p in items
                if not all(
                    c.get("status") == "True"
                    for c in p.get("status", {}).get("conditions", [])
                    if c.get("type") == "Ready"
                )
            ]
            if not_ready:
                checks["speaker_pods"] = f"degraded: not ready: {', '.join(not_ready)}"
            else:
                checks["speaker_pods"] = f"ok ({len(items)} ready)"
    except Exception as e:
        checks["speaker_pods"] = f"error: {e}"

    # EIP pool availability
    try:
        result = subprocess.run(
            ["kubectl", "get", "eip", "-A", "-o", "json"],
            capture_output=True, text=True, timeout=15,
        )
        if result.returncode == 0:
            eips = json.loads(result.stdout).get("items", [])
            pools = []
            for eip in eips:
                spec = eip.get("spec", {})
                status = eip.get("status", {})
                total = len(spec.get("address", "").split("-")) if "-" in spec.get("address", "") else 1
                used = status.get("usage", 0)
                pools.append({"name": eip["metadata"]["name"], "used": used})
            checks["eip_pools"] = pools if pools else "ok (no pools found)"
        else:
            checks["eip_pools"] = "warning: could not query EIP resources"
    except Exception as e:
        checks["eip_pools"] = f"error: {e}"

    speaker_ok = "ok" in str(checks.get("speaker_pods", ""))
    healthy = speaker_ok

    response.status_code = 200 if healthy else 503
    return {"status": "ok" if healthy else "degraded", "checks": checks}

Run the health service

pip install fastapi uvicorn
uvicorn openelb_health:app --host 0.0.0.0 --port 8095

Add a Kubernetes Deployment

# openelb-health-deploy.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: openelb-health
  namespace: openelb-system
spec:
  replicas: 1
  selector:
    matchLabels:
      app: openelb-health
  template:
    metadata:
      labels:
        app: openelb-health
    spec:
      serviceAccountName: openelb-health-sa
      containers:
        - name: health
          image: python:3.12-slim
          command:
            - sh
            - -c
            - |
              pip install -q fastapi uvicorn &&
              uvicorn openelb_health:app --host 0.0.0.0 --port 8095
          ports:
            - containerPort: 8095
          env:
            - name: OPENELB_NAMESPACE
              value: openelb-system
---
apiVersion: v1
kind: Service
metadata:
  name: openelb-health
  namespace: openelb-system
spec:
  selector:
    app: openelb-health
  ports:
    - port: 8095
      targetPort: 8095
  type: ClusterIP
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: openelb-health-sa
  namespace: openelb-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: openelb-health-reader
rules:
  - apiGroups: [""]
    resources: ["pods"]
    verbs: ["get", "list"]
  - apiGroups: ["network.kubesphere.io"]
    resources: ["eips"]
    verbs: ["get", "list"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: openelb-health-reader
subjects:
  - kind: ServiceAccount
    name: openelb-health-sa
    namespace: openelb-system
roleRef:
  kind: ClusterRole
  name: openelb-health-reader
  apiGroup: rbac.authorization.k8s.io

kubectl apply -f openelb-health-deploy.yaml

Verify the endpoint

kubectl port-forward svc/openelb-health 8095:8095 -n openelb-system
curl http://localhost:8095/health

Expected response when all speakers are ready:

{
  "status": "ok",
  "checks": {
    "speaker_pods": "ok (3 ready)",
    "eip_pools": [{"name": "eip-pool-1", "used": 4}]
  }
}

Part 3: Set up HTTP monitoring in Vigilmon

Monitor OpenELB speaker health

1. Log in to vigilmon.online and click Add Monitor.
2. Choose HTTP(S) monitor.
3. Enter the endpoint URL reachable from your monitoring host: http://your-cluster-ingress:8095/health
4. Set interval to 1 minute.
5. Add a keyword check: must contain "status":"ok".
6. Add your alert channel (Slack, PagerDuty, or email).
7. Click Save.

The keyword check fires an alert when speakers are degraded even if the endpoint returns HTTP 200 with a "degraded" body.

Monitor individual LoadBalancer service endpoints

Add a separate Vigilmon monitor for each critical service that uses a LoadBalancer IP:

| Monitor name | URL | Notes | |---|---|---| | API Gateway — LB | http://203.0.113.10:80 | Primary ingress | | gRPC service — LB | http://203.0.113.11:9090 | Internal gRPC | | Database proxy — LB | tcp://203.0.113.12:5432 | TCP check |

These service-level monitors detect BGP failures that the speaker pod readiness check may not catch immediately — if BGP withdraws routes, the service IP stops responding before the pod becomes NotReady.

Part 4: Monitor BGP session state

OpenELB uses GoBGP internally. Extract BGP session status for deeper monitoring.

# bgp_health.py
import subprocess
import json
from fastapi import FastAPI, Response

app = FastAPI()


@app.get("/health/bgp")
def bgp_health(response: Response):
    try:
        result = subprocess.run(
            ["kubectl", "exec", "-n", "openelb-system",
             "deploy/openelb-manager", "--",
             "gobgp", "neighbor", "-j"],
            capture_output=True, text=True, timeout=15,
        )
        if result.returncode != 0:
            response.status_code = 503
            return {"status": "error", "detail": result.stderr.strip()[:200]}

        neighbors = json.loads(result.stdout)
        down = [
            n["conf"]["neighborAddress"]
            for n in neighbors
            if n.get("state", {}).get("sessionState") != "ESTABLISHED"
        ]

        if down:
            response.status_code = 503
            return {
                "status": "degraded",
                "bgp_sessions_down": down,
                "total_neighbors": len(neighbors),
            }

        return {
            "status": "ok",
            "bgp_sessions_established": len(neighbors),
        }

    except Exception as e:
        response.status_code = 503
        return {"status": "error", "detail": str(e)}

Add a Vigilmon monitor pointing at /health/bgp with keyword "status":"ok" and 2-minute interval.

Part 5: Docker Compose for local testing

# docker-compose.yml
version: "3.8"

services:
  openelb-health-mock:
    image: python:3.12-slim
    command: >
      sh -c "pip install -q fastapi uvicorn &&
             uvicorn openelb_health:app --host 0.0.0.0 --port 8095"
    ports:
      - "8095:8095"
    volumes:
      - ./openelb_health.py:/app/openelb_health.py
    working_dir: /app
    environment:
      OPENELB_NAMESPACE: openelb-system
      KUBECONFIG: /root/.kube/config
    volumes:
      - ~/.kube:/root/.kube:ro
      - ./openelb_health.py:/app/openelb_health.py

Part 6: Webhook alerts for networking team escalation

OpenELB failures affect every service using a LoadBalancer IP — configure Vigilmon webhooks to page the networking team immediately.

// webhook-receiver.ts
import express from 'express';

const app = express();
app.use(express.json());

app.post('/webhook/vigilmon', async (req, res) => {
  const { monitor_name, status, url, checked_at } = req.body;

  if (status === 'down') {
    console.error('[NETWORK] OpenELB health check DOWN', {
      monitor: monitor_name,
      url,
      at: checked_at,
    });

    await pageNetworkingTeam({
      title: `OpenELB DOWN: ${monitor_name}`,
      severity: 'critical',
      impact: 'LoadBalancer services may be unreachable cluster-wide',
      since: checked_at,
      runbook: 'https://wiki.example.com/runbooks/openelb-recovery',
    });

  } else if (status === 'up') {
    console.info('[NETWORK] OpenELB recovered', { monitor: monitor_name });
    await resolveNetworkAlert(monitor_name);
  }

  res.sendStatus(204);
});

Vigilmon sends this payload on status change:

{
  "monitor_id": "mon_openelb",
  "monitor_name": "OpenELB — Speaker health",
  "status": "down",
  "url": "http://cluster.internal:8095/health",
  "checked_at": "2026-07-02T11:00:00Z",
  "response_code": 503,
  "response_time_ms": 1420
}

Part 7: SSL monitoring for OpenELB management endpoints

If OpenELB's webhook or manager endpoint is served over HTTPS:

1. In Vigilmon, click Add Monitor.
2. Choose SSL monitor.
3. Enter openelb-manager.openelb-system.svc.cluster.local (or your external domain).
4. Set alert threshold to 14 days before expiry.
5. Add your alert channel.

An expired certificate on the OpenELB webhook blocks Kubernetes from calling the admission webhook, causing all new LoadBalancer service creation to fail.

Summary

Your OpenELB deployment now has layered monitoring:

1. Speaker pod readiness endpoint — confirms all per-node speakers are running, polled every 60 seconds by Vigilmon.
2. Service-level monitors — catch BGP route withdrawal before pod readiness reflects the failure.
3. BGP session monitor — detects upstream router disconnections before traffic is impacted.
4. Webhook escalation — pages the networking team on first failure with runbook link.
5. EIP pool visibility — surfaces pool exhaustion before new services get stuck in Pending.
6. SSL monitors — alerts before certificate expiry breaks OpenELB's admission webhook.

Vigilmon handles check scheduling, multi-region polling, and alert routing. When OpenELB's BGP sessions drop or speakers crash, your networking team knows within 60 seconds — not after users start filing tickets.

Monitor your OpenELB bare-metal load balancer free at vigilmon.online

#openelb #kubernetes #baremetal #bgp #loadbalancer #monitoring