Odigos is an automatic distributed tracing platform that uses eBPF and OpenTelemetry to instrument services without code changes. Point it at a Kubernetes cluster and it discovers running workloads, injects instrumentation, sets up collector pipelines, and ships traces to your backend — all automatically. The appeal is obvious: zero-touch tracing for every service.
The risk is equally obvious: when Odigos's instrumentor crashes, services silently stop producing traces. When a collector pipeline backs up, spans are dropped without error. When trace export to the backend fails intermittently, you see partial traces and assume your services are broken, not your telemetry pipeline. Vigilmon adds external health monitoring that catches these failures before they corrupt your tracing data or create false debugging signals.
Why Odigos Needs External Monitoring
Odigos runs several components whose failures aren't visible from the instrumented services:
- Odigos Instrumentor: The control-plane component that detects workloads and manages instrumentation lifecycle. If it crashes, new deployments receive no auto-instrumentation.
- OpenTelemetry Collector pipelines: Each destination has a collector pipeline with receivers, processors, and exporters. Pipeline saturation silently drops spans.
- Trace export failures: If the backend (Jaeger, Tempo, Datadog) is unreachable or rate-limiting, the OTel exporter backs off and drops excess spans without surfacing the failure clearly.
- Source auto-detection: Odigos uses language detection to decide how to instrument each workload. If detection fails for a new service type, that service produces no traces and the gap is invisible.
External monitoring with Vigilmon provides the independent signal that confirms Odigos is working, not just running.
Step 1: Monitor Odigos Instrumentor Health
Odigos Instrumentor runs as a Deployment in the odigos-system namespace. Check its readiness probe and expose a health endpoint:
# Verify instrumentor pod is running
kubectl get pods -n odigos-system -l app=odigos-instrumentor
# Check instrumentor leader election and control loop health
kubectl logs -n odigos-system -l app=odigos-instrumentor --tail=50 | grep -E "error|leader|reconcile"
Build a Kubernetes-aware health check that validates the instrumentor Deployment's ready replica count:
# odigos_health.py
from flask import Flask, jsonify
import subprocess
import json
import requests
import os
app = Flask(__name__)
ODIGOS_NAMESPACE = os.environ.get('ODIGOS_NAMESPACE', 'odigos-system')
ODIGOS_UI_URL = os.environ.get('ODIGOS_UI_URL', 'http://odigos-ui:3000')
@app.route('/health/odigos/instrumentor')
def instrumentor_health():
try:
result = subprocess.run(
[
'kubectl', 'get', 'deployment', 'odigos-instrumentor',
'-n', ODIGOS_NAMESPACE,
'-o', 'json',
],
capture_output=True, text=True, timeout=10,
)
if result.returncode != 0:
return jsonify(status='down', error=result.stderr.strip()), 503
deployment = json.loads(result.stdout)
status = deployment.get('status', {})
desired = deployment.get('spec', {}).get('replicas', 1)
ready = status.get('readyReplicas', 0)
if ready < desired:
return jsonify(
status='down',
reason='instrumentor_not_ready',
desired=desired,
ready=ready,
), 503
return jsonify(status='ok', desired=desired, ready=ready), 200
except subprocess.TimeoutExpired:
return jsonify(status='down', error='kubectl_timeout'), 503
except Exception as e:
return jsonify(status='down', error=str(e)), 503
if __name__ == '__main__':
app.run(host='0.0.0.0', port=int(os.environ.get('PORT', '5012')))
Step 2: Monitor Collector Pipeline Status and Throughput
Each Odigos destination has an OpenTelemetry Collector pipeline. Use the OTel Collector's built-in zpages and metrics endpoints to validate pipeline health:
# Check collector health (zpages endpoint)
curl -s http://odigos-gateway-collector:55679/debug/servicez
# Check if collector is receiving spans
curl -s http://odigos-gateway-collector:8888/metrics | grep otelcol_receiver_accepted_spans
Add a pipeline throughput check to the health sidecar:
COLLECTOR_METRICS_URL = os.environ.get(
'COLLECTOR_METRICS_URL',
'http://odigos-gateway-collector:8888/metrics'
)
DROPPED_SPANS_THRESHOLD = int(os.environ.get('DROPPED_SPANS_THRESHOLD', '100'))
@app.route('/health/odigos/collector')
def collector_health():
try:
resp = requests.get(COLLECTOR_METRICS_URL, timeout=10)
resp.raise_for_status()
accepted_spans = None
dropped_spans = 0
export_errors = 0
for line in resp.text.splitlines():
if line.startswith('#'):
continue
if 'otelcol_receiver_accepted_spans' in line:
try:
accepted_spans = float(line.split()[-1])
except ValueError:
pass
if 'otelcol_processor_dropped_spans' in line:
try:
dropped_spans += float(line.split()[-1])
except ValueError:
pass
if 'otelcol_exporter_send_failed_spans' in line:
try:
export_errors += float(line.split()[-1])
except ValueError:
pass
if dropped_spans > DROPPED_SPANS_THRESHOLD:
return jsonify(
status='degraded',
reason='spans_being_dropped',
dropped_spans=dropped_spans,
threshold=DROPPED_SPANS_THRESHOLD,
), 503
if export_errors > 0:
return jsonify(
status='degraded',
reason='export_failures_detected',
export_errors=export_errors,
), 503
return jsonify(
status='ok',
accepted_spans=accepted_spans,
dropped_spans=dropped_spans,
export_errors=export_errors,
), 200
except Exception as e:
return jsonify(status='down', error=str(e)), 503
Step 3: Monitor Trace Export Success Rate
Trace export failures to the backend (Jaeger, Grafana Tempo, Datadog, etc.) produce incomplete traces without any obvious user-facing error. Monitor the exporter success rate directly:
@app.route('/health/odigos/trace-export')
def trace_export_health():
try:
resp = requests.get(COLLECTOR_METRICS_URL, timeout=10)
resp.raise_for_status()
sent_spans = 0.0
failed_spans = 0.0
for line in resp.text.splitlines():
if line.startswith('#'):
continue
if 'otelcol_exporter_sent_spans' in line:
try:
sent_spans += float(line.split()[-1])
except ValueError:
pass
if 'otelcol_exporter_send_failed_spans' in line:
try:
failed_spans += float(line.split()[-1])
except ValueError:
pass
total = sent_spans + failed_spans
if total == 0:
# No spans processed — check if this is expected (idle cluster)
return jsonify(
status='ok',
note='no_spans_processed_yet',
sent_spans=0,
failed_spans=0,
), 200
failure_rate = failed_spans / total
max_failure_rate = float(os.environ.get('MAX_EXPORT_FAILURE_RATE', '0.05'))
if failure_rate > max_failure_rate:
return jsonify(
status='degraded',
reason='high_export_failure_rate',
failure_rate=round(failure_rate, 4),
threshold=max_failure_rate,
sent_spans=sent_spans,
failed_spans=failed_spans,
), 503
return jsonify(
status='ok',
failure_rate=round(failure_rate, 4),
sent_spans=sent_spans,
failed_spans=failed_spans,
), 200
except Exception as e:
return jsonify(status='down', error=str(e)), 503
Step 4: Monitor Source Auto-Detection Health
Odigos auto-detects the programming language of each workload to choose the right instrumentation. If auto-detection fails for a new service, tracing silently stops working for that service. Validate Odigos is detecting and instrumenting sources:
@app.route('/health/odigos/source-detection')
def source_detection_health():
try:
# Query Odigos CRD for InstrumentedApplication resources
result = subprocess.run(
[
'kubectl', 'get', 'instrumentedapplications',
'--all-namespaces', '-o', 'json',
],
capture_output=True, text=True, timeout=10,
)
if result.returncode != 0:
return jsonify(status='down', error=result.stderr.strip()), 503
data = json.loads(result.stdout)
items = data.get('items', [])
if not items:
return jsonify(
status='degraded',
reason='no_instrumented_applications_found',
note='odigos may not have detected any workloads',
), 503
# Check for items with failed detection
failed = [
{
'name': item.get('metadata', {}).get('name'),
'namespace': item.get('metadata', {}).get('namespace'),
}
for item in items
if not item.get('spec', {}).get('languages')
]
if failed:
return jsonify(
status='degraded',
reason='language_detection_failed_for_some_sources',
failed_count=len(failed),
total=len(items),
failed_sources=failed[:10], # cap list length
), 503
return jsonify(
status='ok',
instrumented_app_count=len(items),
), 200
except Exception as e:
return jsonify(status='down', error=str(e)), 503
Step 5: Configure Vigilmon HTTP Monitors for Odigos
Monitor 1: Odigos Instrumentor Health
- Log in to vigilmon.online and go to Monitors → New Monitor
- Choose HTTP / HTTPS
- URL:
https://odigos-health.your-domain.com/health/odigos/instrumentor - Interval: 2 minutes
- Expected: Status code
200, body contains"status":"ok" - Alert: Slack + PagerDuty (P1 — instrumentor down means new deployments get no tracing)
- Save
Monitor 2: Collector Pipeline Health
- URL:
https://odigos-health.your-domain.com/health/odigos/collector - Expected:
200, body contains"status":"ok" - Interval: 2 minutes
- Alert: Slack + PagerDuty (span dropping is a data loss event)
Monitor 3: Trace Export Success Rate
- URL:
https://odigos-health.your-domain.com/health/odigos/trace-export - Expected:
200, body contains"status":"ok" - Interval: 5 minutes
- Alert: Slack + PagerDuty (export failures mean incomplete traces in the backend)
Monitor 4: Source Auto-Detection
- URL:
https://odigos-health.your-domain.com/health/odigos/source-detection - Expected:
200, body contains"status":"ok" - Interval: 10 minutes
- Alert: Slack (P2 — detection gaps don't cause immediate production impact)
Step 6: Heartbeat Monitor for Odigos UI Availability
Validate that the Odigos UI and API are accessible for operators:
#!/bin/bash
# odigos_ui_heartbeat.sh
STATUS=$(curl -sf -o /dev/null -w "%{http_code}" http://odigos-ui:3000/api/health 2>/dev/null)
if [ "$STATUS" = "200" ]; then
curl -sf "$VIGILMON_HEARTBEAT_URL" > /dev/null
fi
*/5 * * * * /opt/odigos/odigos_ui_heartbeat.sh
In Vigilmon:
- Monitors → New Monitor → Heartbeat
- Name:
odigos-ui-alive - Expected interval: 10 minutes, Grace: 15 minutes
- Save and configure
VIGILMON_HEARTBEAT_URL
Alert Routing Summary
| Monitor | Signal | Priority | |---|---|---| | Instrumentor health | Odigos control plane liveness | P1 | | Collector pipeline | Span drop and pipeline saturation | P1 | | Trace export rate | Export failures to backend | P1 | | Source auto-detection | Language detection coverage | P2 | | UI heartbeat | Odigos operator interface availability | P3 |
Summary
Odigos eliminates instrumentation boilerplate — but introduces a new class of invisible failure where traces silently stop flowing without any obvious error. External monitoring with Vigilmon ensures you know about Odigos failures before they corrupt debugging sessions or create false alarms:
| What Vigilmon Monitors | Why It Matters | |---|---| | Instrumentor Deployment | Catches control-plane failures before new deployments go uninstrumented | | Collector pipeline health | Detects span drops and pipeline saturation in real time | | Trace export success rate | Flags backend export failures that produce partial or missing traces | | Source auto-detection | Validates all workloads are being detected and instrumented | | UI availability | Confirms operators can access Odigos for configuration changes |
Get started free at vigilmon.online — your first Odigos monitor is running in under two minutes.