tutorial

Monitoring Beyla with Vigilmon: eBPF Probe Health, Service Discovery, RED Metrics & Kernel Compatibility Checks

How to monitor Beyla with Vigilmon — eBPF probe health checks, instrumented service discovery validation, RED metrics accuracy monitoring, kernel compatibility alerts, and Vigilmon integration.

Beyla is Grafana Labs' eBPF-based automatic instrumentation tool that injects observability into your applications without requiring code changes — it intercepts HTTP, gRPC, and database calls at the kernel level by attaching eBPF probes to running processes. When Beyla loses a probe attachment because the target process restarts under a new PID, that service disappears from your RED metrics without any error. When the kernel version on a node is below Beyla's minimum requirement, probes silently fail to load and you have a blind spot in your observability coverage. When Beyla's OTLP exporter can't reach Grafana Cloud or an Alloy instance, spans queue up and are eventually dropped. Vigilmon gives you external endpoint checks and health validation for Beyla's service, so you catch probe failures and exporter outages before your RED metrics go stale.

What You'll Build

  • An HTTP monitor on Beyla's internal health endpoint to detect probe manager failures
  • Service availability checks on endpoints Beyla is actively instrumenting
  • Prometheus metric alerting for probe attachment failures and export errors
  • Kernel compatibility validation checks in your deployment pipeline
  • SSL monitoring for Beyla's OTLP export endpoints
  • An alerting runbook mapping Beyla failure modes to Vigilmon monitor states

Prerequisites

  • Beyla 1.x running as a DaemonSet (Kubernetes) or systemd service (Linux hosts)
  • Linux kernel 5.8 or newer on instrumented nodes (BTF enabled, required for Beyla)
  • Prometheus configured to scrape Beyla's /metrics endpoint
  • At least one application process being auto-instrumented by Beyla
  • A free account at vigilmon.online

Step 1: Understand Beyla's Health Surface

Beyla exposes an HTTP server for metrics and internal health status. The default port is 9090 for metrics, though this is configurable via BEYLA_PROMETHEUS_PORT.

# Check Beyla probe status
curl http://localhost:9090/metrics | grep beyla_

# Key health metrics
beyla_process_cpu_seconds_total          # Beyla's own CPU usage
beyla_internal_instrumented_processes    # Number of currently attached probes
beyla_ebpf_tracer_errors_total           # eBPF errors (probe load failures, etc.)

For Kubernetes DaemonSet deployments, check pod readiness across all nodes:

kubectl -n beyla get pods -o wide
# Every node should have a Running/Ready Beyla pod

# Check Beyla logs for probe attachment errors
kubectl -n beyla logs -l app=beyla --tail=50 | grep -i "error\|warn\|probe"

Beyla attaches eBPF probes to processes it discovers via its selector configuration:

# Beyla process selector (in values.yaml or BEYLA_EXECUTABLE_NAME env var)
discovery:
  services:
    - name: myapp
      namespace: production
      exe_path: /usr/local/bin/myapp

When a target process restarts, Beyla must re-attach probes. During the re-attachment window, telemetry data for that service is missing.

Step 2: Monitor Services Instrumented by Beyla

The most reliable way to validate that Beyla is instrumenting a service is to monitor that service's HTTP health endpoint directly. If the service is healthy but absent from your traces, Beyla's probe has detached:

  1. Log in to Vigilmon → Add Monitor → HTTP.
  2. URL: https://myapp.example.com/health (the application Beyla is instrumenting).
  3. Check interval: 60 seconds.
  4. Response timeout: 10 seconds.
  5. Expected status: 200.
  6. Label: myapp (Beyla-instrumented).
  7. Click Save.

Cross-reference Vigilmon uptime data against your Grafana traces dashboard: if Vigilmon shows the service is up but your trace count drops to zero, Beyla's probe has detached from that process.

Step 3: Monitor the Beyla Metrics Endpoint

Beyla's Prometheus metrics endpoint reports internal health including instrumented process count and eBPF error rates. Monitoring this endpoint confirms Beyla itself is alive:

  1. Add Monitor → HTTP.
  2. URL: http://beyla-node.example.com:9090/metrics (adjust to your exposed metrics endpoint).
  3. Check interval: 2 minutes.
  4. Response timeout: 15 seconds.
  5. Expected status: 200.
  6. Keyword: beyla_internal_instrumented_processes.
  7. Label: Beyla metrics endpoint.
  8. Click Save.

The keyword beyla_internal_instrumented_processes confirms both that metrics are being served and that Beyla has at least loaded its process discovery logic. If this keyword disappears, Beyla's internal discovery has stopped entirely.

Step 4: Configure Prometheus Alerting for Probe Health

Beyla emits metrics that reveal whether probes are attached and whether the eBPF subsystem is healthy. Set up Prometheus rules to alert on degraded instrumentation:

# prometheus-beyla-alerts.yaml
groups:
  - name: beyla_probe_health
    rules:
      - alert: BeylaNoInstrumentedProcesses
        expr: beyla_internal_instrumented_processes == 0
        for: 3m
        labels:
          severity: critical
        annotations:
          summary: "Beyla has no instrumented processes"
          description: "Beyla on {{ $labels.instance }} is attached to 0 processes — service discovery may have failed."

      - alert: BeylaEBPFErrors
        expr: rate(beyla_ebpf_tracer_errors_total[5m]) > 0.1
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Beyla eBPF tracer errors elevated"
          description: "Beyla is experiencing eBPF probe errors on {{ $labels.instance }} — check kernel version compatibility."

      - alert: BeylaExporterDropped
        expr: rate(beyla_otel_trace_export_error_total[5m]) > 0
        for: 2m
        labels:
          severity: warning
        annotations:
          summary: "Beyla OTLP exporter dropping spans"
          description: "Beyla cannot export traces to {{ $labels.target }} — check OTLP endpoint availability."

Apply and verify:

kubectl apply -f prometheus-beyla-alerts.yaml
# Check rule evaluation
curl http://prometheus:9090/api/v1/rules | jq '.data.groups[].rules[] | select(.name | startswith("Beyla"))'

Step 5: Kernel Compatibility Validation

Beyla requires Linux kernel 5.8 or newer with BTF (BPF Type Format) enabled. A node running an older kernel will silently fail to load probes. Add a kernel check to your node provisioning pipeline:

# Check kernel version on each node
uname -r
# Minimum: 5.8; recommended: 5.15+ for full HTTP/2 and gRPC support

# Verify BTF is available
ls /sys/kernel/btf/vmlinux
# Must exist — absence means BTF is disabled in kernel config

# Check eBPF capabilities
capsh --print | grep sys_admin
# Beyla DaemonSet needs CAP_SYS_ADMIN or CAP_BPF

For Kubernetes clusters, monitor the node kernel version via a DaemonSet that reports the version as a Prometheus metric:

# kernel-version-exporter snippet
command: ["sh", "-c", "while true; do echo kernel_version_major $(uname -r | cut -d. -f1) > /metrics; sleep 60; done"]

Create a Prometheus alert for nodes below the minimum:

kernel_version_major < 5

This alert fires before you deploy Beyla to a new node pool, preventing silent instrumentation failures.

Step 6: Monitor the OTLP Export Endpoint

Beyla exports traces and metrics to an OTLP endpoint — either Grafana Cloud, a local Alloy instance, or a Tempo/Mimir backend. Monitor the export endpoint directly from Vigilmon:

  1. Add Monitor → HTTP.
  2. URL: https://otlp.example.com/v1/traces (your OTLP endpoint).
  3. Check interval: 2 minutes.
  4. Response timeout: 10 seconds.
  5. Expected status: 405 (confirms the OTLP HTTP endpoint is live; POST is required for data ingestion).
  6. Label: OTLP endpoint (Beyla export target).
  7. Click Save.

For Grafana Cloud OTLP endpoints, the URL format is https://otlp-gateway-<zone>.grafana.net/otlp/v1/traces.

Step 7: SSL Certificate Monitoring

Beyla's OTLP exporter communicates over TLS to Grafana Cloud or a local backend. Certificate issues on the backend break all trace export:

  1. Add Monitor → SSL Certificate.
  2. Domain: otlp.example.com (your OTLP backend hostname).
  3. Alert when expiry is within: 30 days.
  4. Alert again: 14 days, 7 days, 3 days.
  5. Click Save.

If Beyla is configured to verify the server certificate (BEYLA_OTEL_INSECURE_SKIP_VERIFY=false, which is the default), an expired or invalid certificate on the OTLP endpoint causes all export to fail silently.

Step 8: Configure Alerting for Probe Failures

In Vigilmon under Settings → Notifications, configure alert channels and map them to the correct response runbook:

| Monitor | Trigger | Immediate action | |---|---|---| | Instrumented service health | Non-200 from monitored app | Verify service health independently; if service is healthy, check Beyla probe attachment | | Beyla metrics endpoint | Non-200 or keyword missing | Beyla process has crashed; restart DaemonSet pod; check systemd status | | OTLP endpoint | Non-405 or connection refused | Beyla cannot export; check Alloy/Grafana Cloud availability and credentials | | SSL certificate | < 30 days to expiry | Renew certificate; Beyla OTLP export will fail on TLS validation error |

Common Beyla Failure Modes and What Vigilmon Catches

| Scenario | Vigilmon signal | |---|---| | Target process restarts with new PID | No Vigilmon signal (service still responds); trace count drops in Grafana — Beyla re-attaches within seconds | | Beyla DaemonSet pod crashes | Metrics endpoint monitor fires; all instrumented services lose trace data | | Kernel BTF missing on new node | Beyla fails to load probes silently; no Vigilmon signal until metrics show 0 processes | | OTLP backend unreachable | OTLP endpoint monitor fires; Beyla queues spans then drops on buffer overflow | | OTLP TLS certificate expired | SSL monitor fires 30 days before; export fails entirely on expiry | | Beyla runs out of eBPF map space | eBPF error rate rises; Prometheus alert fires | | Service scaled to 0 replicas | Vigilmon fires on service health check; Beyla probe count drops naturally | | Container runtime OCI hook failed | Beyla cannot auto-discover new container; no Vigilmon signal — check DaemonSet logs | | Network policy blocks OTLP port | OTLP endpoint monitor fires from Vigilmon's external vantage; internal Beyla errors rise | | Resource limits throttle Beyla CPU | Export latency increases; check beyla_otel_trace_export_duration_seconds_bucket |

Beyla's zero-code instrumentation is its greatest strength and its biggest operational risk: when a probe detaches or the export pipeline breaks, your applications keep running while your observability coverage silently degrades. Vigilmon gives you external validation — health checks on every instrumented service, availability monitoring on your OTLP backend, and SSL certificate tracking — so you know whether Beyla's data is reaching your dashboards or quietly disappearing. When Beyla can't tell you something is wrong, Vigilmon can.

Start monitoring your Beyla deployments in under 5 minutes — register free at vigilmon.online.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles