How to Monitor KubeArmor Runtime Security with Vigilmon

KubeArmor is a Kubernetes runtime security engine that enforces security policies at the system call level using eBPF and Linux Security Modules. It intercepts container syscalls, blocks unauthorized file access, restricts network connections, and prevents privilege escalation — all without modifying your application code.

But security enforcement infrastructure has a unique monitoring challenge: when KubeArmor goes down, it doesn't just leave a service unavailable — it may leave your cluster in an unprotected state. Depending on your audit and allow/deny policy configuration, a crashed KubeArmor DaemonSet pod might allow containers to execute syscalls that should be blocked, or it might fail-closed and break your workloads entirely.

Either way, you need to know immediately.

This tutorial covers setting up external uptime monitoring for KubeArmor using Vigilmon — so you get alerted the moment your runtime security enforcement has a gap.

Why KubeArmor needs external monitoring

KubeArmor runs as a DaemonSet on every node in your Kubernetes cluster. Kubernetes monitors those pods with liveness probes and will restart crashed containers. But there are gaps:

• Restart loops — a KubeArmor pod that crashes and restarts 10 times per hour is technically "running" most of the time, but it's providing broken security enforcement during those restart windows; Kubernetes won't alert you
• Node-level failures — if a specific node's KubeArmor pod fails to start (incompatible kernel, missing eBPF support), workloads on that node are unprotected; kubectl get pods -A won't make this obvious unless you know to look
• KubeArmor Relay server down — KubeArmor uses a relay server to aggregate and export telemetry; if that's unavailable, your security audit logs stop flowing without any obvious error in Kubernetes
• KubeArmor Controller issues — the KubeArmor controller (if deployed) processes KubeArmorPolicy resources; if it's unhealthy, new policies won't be applied
• Cluster admission webhook failures — if KubeArmor's mutating webhook goes down, pod admission can be blocked (if fail-closed) or security annotations may not be applied (if fail-open)

An external monitoring tool like Vigilmon watches these components from outside Kubernetes, giving you an independent signal that doesn't depend on the cluster's own health.

What you'll need

• A Kubernetes cluster with KubeArmor installed
• kubectl access to the cluster
• A free Vigilmon account

Step 1: Install KubeArmor and verify it's running

If you haven't installed KubeArmor yet:

# Install KubeArmor using helm
helm repo add kubearmor https://kubearmor.github.io/charts
helm repo update kubearmor
helm install kubearmor kubearmor/kubearmor -n kubearmor --create-namespace

# Or use the KubeArmor CLI
curl -sfL https://raw.githubusercontent.com/kubearmor/KubeArmor/main/contribution/installation/install.sh | sudo bash
karmor install

Verify all components are healthy:

# Check DaemonSet status
kubectl get daemonset -n kubearmor
# NAME         DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE
# kubearmor    3         3         3       3            3

# Check all KubeArmor pods
kubectl get pods -n kubearmor -o wide
# All should show Running/Ready

# Check KubeArmor controller
kubectl get deployment -n kubearmor

# Check KubeArmor relay
kubectl get svc -n kubearmor

Step 2: Create a health-check proxy endpoint

KubeArmor exposes a gRPC API (port 32767 by default) and a health endpoint, but it's not directly accessible for simple HTTP monitoring. The cleanest approach is to deploy a small monitoring sidecar that checks the KubeArmor DaemonSet status and exposes an HTTP health endpoint.

Option A: Deploy a KubeArmor health proxy

# kubearmor-health-proxy.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: kubearmor-health-proxy
  namespace: kubearmor
spec:
  replicas: 1
  selector:
    matchLabels:
      app: kubearmor-health-proxy
  template:
    metadata:
      labels:
        app: kubearmor-health-proxy
    spec:
      serviceAccountName: kubearmor-health-sa
      containers:
        - name: health-proxy
          image: python:3.11-alpine
          command: ["/bin/sh", "-c"]
          args:
            - |
              cat > /health.py << 'EOF'
              import subprocess, json, os
              from http.server import HTTPServer, BaseHTTPRequestHandler
              from urllib.request import urlopen

              class Health(BaseHTTPRequestHandler):
                  def do_GET(self):
                      if self.path != '/health':
                          self.send_response(404); self.end_headers(); return
                      try:
                          # Check KubeArmor DaemonSet via kubectl
                          result = subprocess.run(
                              ['kubectl', 'get', 'daemonset', 'kubearmor',
                               '-n', 'kubearmor', '-o', 'json'],
                              capture_output=True, text=True, timeout=10
                          )
                          ds = json.loads(result.stdout)
                          desired = ds['status']['desiredNumberScheduled']
                          ready = ds['status']['numberReady']
                          healthy = ready == desired and desired > 0
                          status_code = 200 if healthy else 503
                          body = json.dumps({
                              "status": "ok" if healthy else "degraded",
                              "desired": desired,
                              "ready": ready
                          }).encode()
                      except Exception as e:
                          status_code = 503
                          body = json.dumps({"status": "error", "error": str(e)}).encode()
                      self.send_response(status_code)
                      self.send_header('Content-Type', 'application/json')
                      self.end_headers()
                      self.wfile.write(body)
                  def log_message(self, *args): pass
              HTTPServer(('0.0.0.0', 8080), Health).serve_forever()
              EOF
              pip install -q requests && python /health.py
          ports:
            - containerPort: 8080
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: kubearmor-health-sa
  namespace: kubearmor
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: kubearmor-health-reader
rules:
  - apiGroups: ["apps"]
    resources: ["daemonsets", "deployments"]
    verbs: ["get", "list"]
  - apiGroups: [""]
    resources: ["pods"]
    verbs: ["get", "list"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: kubearmor-health-reader
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: kubearmor-health-reader
subjects:
  - kind: ServiceAccount
    name: kubearmor-health-sa
    namespace: kubearmor
---
apiVersion: v1
kind: Service
metadata:
  name: kubearmor-health
  namespace: kubearmor
spec:
  type: LoadBalancer   # or NodePort if no LoadBalancer support
  selector:
    app: kubearmor-health-proxy
  ports:
    - port: 80
      targetPort: 8080

kubectl apply -f kubearmor-health-proxy.yaml

# Get the external IP
kubectl get svc kubearmor-health -n kubearmor
# NAME                TYPE           CLUSTER-IP     EXTERNAL-IP    PORT(S)
# kubearmor-health    LoadBalancer   10.96.42.200   203.0.113.75   80:31234/TCP

Test the health endpoint:

curl http://203.0.113.75/health
# {"status":"ok","desired":3,"ready":3}

Step 3: Set up HTTP monitoring in Vigilmon

1. Log in to vigilmon.online and go to Monitors → New Monitor
2. Choose HTTP / HTTPS
3. URL: http://203.0.113.75/health
4. Check interval: 1 minute
5. Expected response:
   • Status code: 200
   • Response body contains: "status":"ok"
6. Name the monitor: "KubeArmor DaemonSet Health"
7. Save the monitor

When any KubeArmor DaemonSet pod goes missing (crash, node failure, failed rollout), the ready count will drop below desired, the endpoint returns 503, and Vigilmon alerts you within 60 seconds.

Step 4: Monitor the KubeArmor gRPC API port

KubeArmor's gRPC API listens on port 32767 (configurable). While Vigilmon can't speak gRPC, it can verify the TCP port is accepting connections — which confirms the core KubeArmor process is running.

To expose the gRPC port externally:

kubectl port-forward svc/kubearmor -n kubearmor 32767:32767 &
# For persistent access, create a NodePort service

Or add a TCP NodePort service:

apiVersion: v1
kind: Service
metadata:
  name: kubearmor-grpc-external
  namespace: kubearmor
spec:
  type: NodePort
  selector:
    app: kubearmor
  ports:
    - port: 32767
      targetPort: 32767
      nodePort: 32767
      protocol: TCP

Then in Vigilmon:

1. Go to Monitors → New Monitor
2. Choose TCP Port
3. Host: your node IP or LoadBalancer hostname, Port: 32767
4. Save the monitor

Step 5: Monitor the KubeArmor Relay

The KubeArmor relay server aggregates telemetry from all DaemonSet pods. If it goes down, your security audit logs stop flowing — a dangerous blind spot.

# Check the relay service
kubectl get svc kubearmor-relay -n kubearmor

Add a TCP monitor for the relay's gRPC port (default: 32768):

1. In Vigilmon, add a TCP monitor for your-cluster-ip:32768
2. Name it: "KubeArmor Relay gRPC"

Step 6: Monitor the KubeArmor Controller

The KubeArmor Controller (if deployed) watches KubeArmorPolicy resources and applies them to the DaemonSet. If the controller is unhealthy, policy changes are silently ignored.

Add the health proxy to report controller status alongside the DaemonSet. Or create a separate HTTP monitor if the controller exposes its own metrics endpoint:

kubectl get deployment kubearmor-controller -n kubearmor
kubectl get svc -n kubearmor | grep controller

If the controller exposes a /healthz endpoint on a port, add it as a Vigilmon HTTP monitor.

Step 7: Configure alert channels

Security infrastructure failures are high-priority incidents — alerts should reach your on-call security team immediately.

Email alerts

1. Go to Alert Channels → Add Channel → Email
2. Add your security operations team email
3. Assign to all KubeArmor monitors

PagerDuty or OpsGenie for on-call

1. Go to Alert Channels → Add Channel → Webhook
2. Paste your PagerDuty Events API or OpsGenie webhook URL
3. Assign to your monitors
4. Set priority to Critical if your monitoring tool supports it

Vigilmon alert payload when KubeArmor degrades:

{
  "monitor_name": "KubeArmor DaemonSet Health",
  "status": "down",
  "url": "http://203.0.113.75/health",
  "started_at": "2026-07-02T11:45:00Z",
  "duration_seconds": 90
}

Step 8: Diagnose KubeArmor failures

When Vigilmon fires a KubeArmor alert, use this runbook:

# 1. Check overall DaemonSet status
kubectl get daemonset kubearmor -n kubearmor
kubectl describe daemonset kubearmor -n kubearmor

# 2. Find unhealthy pods
kubectl get pods -n kubearmor -o wide
kubectl get pods -n kubearmor | grep -v Running

# 3. Inspect a failing pod
kubectl describe pod <failing-pod> -n kubearmor
kubectl logs <failing-pod> -n kubearmor --previous  # previous container if crashed

# 4. Check node kernel compatibility
# KubeArmor requires kernel >= 4.15 with eBPF support
kubectl get node <node-name> -o jsonpath='{.status.nodeInfo.kernelVersion}'

# 5. Check BPF filesystem availability (required for eBPF)
kubectl exec -n kubearmor <kubearmor-pod> -- ls /sys/fs/bpf/

# 6. Check KubeArmor relay health
kubectl get pods -n kubearmor | grep relay
kubectl logs -n kubearmor -l app=kubearmor-relay

# 7. Check controller health
kubectl get deployment kubearmor-controller -n kubearmor
kubectl logs -n kubearmor -l app=kubearmor-controller

Common KubeArmor failure patterns

| Alert pattern | Probable cause | Remediation | |---------------|----------------|-------------| | DaemonSet ready < desired | Pod(s) crash-looping or failed | Check pod logs, node kernel version, eBPF support | | All pods down on one node | Node kernel doesn't support eBPF | Upgrade kernel or exclude node from KubeArmor | | Relay down, DaemonSet up | Relay process crash | Restart relay pod: kubectl rollout restart deploy/kubearmor-relay -n kubearmor | | Controller down | Webhook timeout or controller crash | Restart controller; check webhook certificates | | Health proxy 503, cluster fine | Health proxy RBAC issue | Verify ClusterRoleBinding is intact |

Step 9: Create a security infrastructure status page

A status page for your security tooling gives your security and compliance teams visibility without requiring kubectl access.

1. In Vigilmon, go to Status Pages → New Status Page
2. Name: "Cluster Security Infrastructure"
3. Add monitors:
   • KubeArmor DaemonSet Health
   • KubeArmor Relay
   • KubeArmor Controller (if applicable)
4. Publish and share with your security and compliance teams

Recommended monitoring setup for KubeArmor

| Monitor | Type | What it detects | |---------|------|-----------------| | http://cluster-ip/health | HTTP | DaemonSet healthy (all nodes enforcing) | | cluster-ip:32767 | TCP | Core KubeArmor gRPC process reachable | | cluster-ip:32768 | TCP | Relay gRPC reachable (audit logs flowing) | | Controller /healthz | HTTP | Controller processing policies |

What's next

With external monitoring covering KubeArmor's components, you'll know within a minute if runtime security enforcement has a gap anywhere in your cluster. Integrate these alerts into your incident response process:

• Compliance reporting: log each KubeArmor downtime incident with duration and affected nodes for your audit trail
• Policy drift detection: combine Vigilmon uptime data with KubeArmor's policy violation logs — a spike in violations after a restart might indicate an incomplete policy re-application
• Failsafe policies: review your KubeArmorPolicy action settings; Block mode prevents unauthorized syscalls but if KubeArmor is down, you need to understand your cluster's default posture

Set up your first KubeArmor monitor in under 2 minutes at vigilmon.online — free tier, no credit card required.