tutorial

How to Monitor Lumigo with Vigilmon (HTTP + TCP + Alerts)

Lumigo is a distributed tracing and monitoring platform built for serverless and microservices architectures. It auto-instruments AWS Lambda functions, conta...

Lumigo is a distributed tracing and monitoring platform built for serverless and microservices architectures. It auto-instruments AWS Lambda functions, containerized services, and Node.js, Python, and Java applications to capture distributed traces, errors, and performance metrics — giving teams end-to-end visibility across their entire stack. When Lumigo's tracer endpoint goes down or becomes unreachable, your Lambda functions keep executing and returning 200s, but every span is silently discarded. You lose distributed tracing across your entire serverless stack with no error surfaced in CloudWatch, no exception thrown, and no obvious indication anything is wrong until someone notices a blank trace dashboard hours later.

External monitoring via Vigilmon closes that gap. By probing Lumigo's ingestion and dashboard endpoints from outside your infrastructure, you get an alert the moment Lumigo becomes unreachable — before your team starts debugging missing traces in production. Vigilmon's free tier is enough to cover the core monitors described in this tutorial.

Why Lumigo needs external monitoring

  • Ingestion endpoint unreachable — silent data loss across all services. Lumigo tracers send spans via HTTP POST to Lumigo's ingestion API. If that endpoint is down or returning errors, the tracer fails silently: your Lambda functions complete normally, CloudWatch shows no errors, and you only discover the problem when someone opens the Lumigo dashboard and sees no traces. Depending on your retention window, hours of distributed tracing data may be unrecoverable.

  • Lumigo dashboard unavailable — blind operations during incidents. If app.lumigo.io is unreachable, your engineering team loses access to traces exactly when they need them most — during an active incident. Without the ability to view spans, correlate errors, and trace request paths, incident response slows dramatically and root cause analysis becomes guesswork.

  • Token expiry or rotation breaks tracer authentication. The LUMIGO_TRACER_TOKEN environment variable must match a valid, active token in Lumigo's backend. When tokens are rotated — during a security audit, a team member offboarding, or a Lumigo account change — Lambda functions with stale tokens silently stop sending traces. The function still runs, the token validation failure is swallowed by the tracer library, and no CloudWatch alarm fires.

  • Tracer library version incompatibility after a Lambda layer update. Lumigo distributes tracers as Lambda layers. When a layer is updated automatically or pinned to a new version, the tracer may encounter an incompatibility with the runtime or an upstream SDK it depends on. Spans stop being emitted, but the function itself continues executing. Without an external check on whether traces are flowing, this regression can go undetected across dozens of functions simultaneously.

  • Regional failover misconfiguration drops spans during AWS regional events. Lumigo's ingestion infrastructure is region-aware. During an AWS regional disruption, if the Lumigo tracer is not configured to route through a failover region or if DNS resolution for the ingestion endpoint fails, all spans from Lambda functions in the affected region are dropped. The failure is confined to observability infrastructure, so application-level monitors and CloudWatch alarms remain green throughout.

What you'll need

  • A Lumigo account with at least one instrumented Lambda function or containerized service sending spans, or a self-hosted Lumigo deployment (enterprise)
  • A way to probe Lumigo's HTTP and TCP endpoints — curl or any HTTP client works for manual validation; Vigilmon handles automated probing
  • A free Vigilmon account — no credit card required, monitors start in under a minute

Step 1: Identify Lumigo's endpoints to monitor

Lumigo SaaS (app.lumigo.io)

Most teams use Lumigo's hosted SaaS platform. There are two distinct endpoints to monitor:

The Lumigo dashboard — this is the web interface your team uses to view traces and errors:

https://app.lumigo.io

The span ingestion endpoint — this is where Lumigo tracers send spans via HTTP POST. Lumigo's tracer libraries target their ingestion API, which resolves to endpoints under lumigo.io. The primary ingestion host used by Lumigo tracers is:

https://ga4gh.lumigo.io

You can verify the ingestion endpoint is reachable with a simple curl probe:

curl -v -o /dev/null -w "%{http_code}" \
  -X POST https://ga4gh.lumigo.io/api/spans \
  -H "Content-Type: application/json" \
  -H "Authorization: LumigoToken your-token-here" \
  -d '[]'

A response in the 2xx or 4xx range confirms the endpoint is reachable (a 4xx with an empty payload is expected and still means the ingestion service is up). A connection timeout or 5xx is the failure condition you want to alert on.

To confirm that your Lambda functions can reach the ingestion endpoint at runtime, add a simple connectivity check as a one-time test in your function code:

import urllib.request

def test_lumigo_connectivity():
    try:
        req = urllib.request.Request(
            "https://ga4gh.lumigo.io/api/spans",
            method="POST",
            data=b"[]",
            headers={"Content-Type": "application/json"}
        )
        with urllib.request.urlopen(req, timeout=5) as response:
            print(f"Lumigo ingestion reachable: {response.status}")
    except Exception as e:
        print(f"Lumigo ingestion UNREACHABLE: {e}")

This can be invoked during a Lambda cold start or warm-up phase to validate network path connectivity before relying on tracer output.

Self-hosted / On-premise Lumigo (enterprise)

For enterprise deployments running Lumigo's backend in a Kubernetes cluster, first identify the relevant services:

# List all Lumigo-related services in the cluster
kubectl get services -n lumigo --output wide

# Check the health endpoint of the Lumigo backend
kubectl get pods -n lumigo -l app=lumigo-backend

# Port-forward to test the ingestion service locally
kubectl port-forward svc/lumigo-ingestion 8080:80 -n lumigo

# Then probe the local health endpoint
curl http://localhost:8080/health

For self-hosted deployments, monitor both the externally exposed ingestion endpoint (via ingress) and the internal cluster health endpoint if you have internal Vigilmon agents or network access.

Step 2: Set up HTTP monitoring in Vigilmon

Monitor the Lumigo dashboard endpoint

  1. Log in to Vigilmon and navigate to Monitors in the sidebar.
  2. Click Add Monitor and select HTTP(S) as the monitor type.
  3. Set the URL to https://app.lumigo.io and the check interval to 60 seconds (or 30 seconds if you want faster detection).
  4. Under Expected response, set the HTTP status to 200 and optionally add a keyword check for a string that appears on the Lumigo login page (such as Lumigo) to confirm content is being served, not just an empty response.
  5. Set the timeout to 10 seconds — Lumigo's dashboard should respond well within that window under normal conditions.

Monitor the trace ingestion endpoint

  1. Add a second HTTP(S) monitor in Vigilmon.
  2. Set the URL to https://ga4gh.lumigo.io/api/spans.
  3. Set the method to POST and add the header Content-Type: application/json with a minimal body of [].
  4. Set expected status to accept 2xx or 4xx responses as "up" — you are checking reachability, not authentication. Configure Vigilmon to alert only on 5xx responses or connection timeouts.
  5. Set the check interval to 60 seconds and timeout to 10 seconds.

Why external monitoring catches what CloudWatch misses

AWS CloudWatch tracks what happens inside your Lambda functions — invocation count, errors, duration, and throttles. It has no visibility into whether the observability infrastructure those functions depend on is functioning. When Lumigo's ingestion endpoint goes down:

  • Lambda invocations continue to succeed — Errors metric stays at zero
  • The Lumigo tracer catches the network failure internally and discards the span buffer — no exception propagates to your handler
  • CloudWatch Logs show no anomaly — the tracer's internal failure is not logged at ERROR level by default
  • Your X-Ray traces (if used alongside Lumigo) are unaffected — the failure is invisible at the AWS layer

Vigilmon's external HTTP probe checks the ingestion endpoint directly, independent of your application code, Lambda runtime, and AWS account. It is the only check that reliably detects ingestion endpoint failures before your team notices empty dashboards.

Step 3: Monitor Lumigo's TCP connectivity

A TCP monitor adds a lower-level check: it verifies that port 443 on Lumigo's ingestion host is accepting connections, independent of HTTP-level behavior. This catches scenarios where TLS termination or the TCP stack fails before the HTTP layer responds.

  1. In Vigilmon, click Add Monitor and select TCP as the monitor type.
  2. Set the host to ga4gh.lumigo.io and port to 443.
  3. Set the check interval to 60 seconds and timeout to 5 seconds.
  4. Save the monitor. Vigilmon will now alert if a TCP connection to the ingestion endpoint cannot be established — which indicates a network-level failure that would prevent all Lumigo tracers from sending spans regardless of their HTTP retry logic.

For the Lumigo dashboard, add a second TCP monitor:

  • Host: app.lumigo.io
  • Port: 443
  • Interval: 60 seconds
  • Timeout: 5 seconds

TCP monitors are lightweight and fast to respond, making them a reliable first signal for connectivity failures before the HTTP-level probe confirms the nature of the failure.

Step 4: Configure alert channels

Email alerts

  1. In Vigilmon, go to Alert Channels and click Add Channel.
  2. Select Email and enter the address for your on-call engineer or team distribution list.
  3. Assign this channel to each of the Lumigo monitors you created in Steps 2 and 3.
  4. Set the alert to trigger after 1 failed check for the ingestion endpoint (data loss starts immediately) and 2 failed checks for the dashboard endpoint (reduces noise from transient blips).

Webhook alerts

  1. Add a second alert channel of type Webhook.
  2. Enter your incident management webhook URL (PagerDuty, Opsgenie, Slack incoming webhook, or a custom endpoint).
  3. Vigilmon will POST a JSON payload to your endpoint on each alert trigger and resolution.

Example alert payload:

{
  "monitor_name": "Lumigo Span Ingestion",
  "monitor_type": "HTTP",
  "status": "DOWN",
  "url": "https://ga4gh.lumigo.io/api/spans",
  "http_status_code": null,
  "error": "Connection timed out after 10000ms",
  "checked_at": "2026-07-11T14:23:01Z",
  "downtime_started_at": "2026-07-11T14:22:00Z",
  "alert_channel": "PagerDuty"
}

Diagnostic runbook when an alert fires

1. Confirm the ingestion endpoint is down from your local machine:

curl -v --max-time 10 \
  -X POST https://ga4gh.lumigo.io/api/spans \
  -H "Content-Type: application/json" \
  -H "Authorization: LumigoToken your-token-here" \
  -d '[]'

If this times out or returns a 5xx, the issue is on Lumigo's side. Check Lumigo's status page at https://status.lumigo.io.

2. Verify Lambda functions still have the correct tracer token:

# List all Lambda functions and check the LUMIGO_TRACER_TOKEN env var
aws lambda list-functions --query 'Functions[*].FunctionName' --output text | \
  tr '\t' '\n' | \
  xargs -I{} aws lambda get-function-configuration --function-name {} \
    --query 'Environment.Variables.LUMIGO_TRACER_TOKEN' --output text

Any function returning None or an unexpected token value is not sending traces.

3. Check Lumigo tracer logs in CloudWatch:

# Search recent log events for Lumigo tracer output in a specific function
aws logs filter-log-events \
  --log-group-name /aws/lambda/your-function-name \
  --filter-pattern "lumigo" \
  --start-time $(date -d '1 hour ago' +%s)000 \
  --query 'events[*].message' \
  --output text

Look for messages containing lumigo, LUMIGO_DEBUG, or any connection error output from the tracer.

4. Enable Lumigo debug logging on a specific function to capture tracer output:

aws lambda update-function-configuration \
  --function-name your-function-name \
  --environment "Variables={LUMIGO_TRACER_TOKEN=your-token,LUMIGO_DEBUG=true}"

Invoke the function once and inspect CloudWatch Logs for the tracer's internal connection attempts and error messages.

5. Validate the Lumigo token is active in the Lumigo console:

Navigate to https://app.lumigo.io > Settings > Tracing > Token. Confirm the token shown matches the value in your Lambda environment variables. If the token has been rotated, update all Lambda functions with the new value and redeploy.

Step 5: Create a public status page

A Vigilmon status page lets your team — and optionally your customers — see the current status of Lumigo monitoring at a glance.

  1. In Vigilmon, navigate to Status Pages in the sidebar and click Create Status Page.
  2. Give the page a name such as Observability Infrastructure or Lumigo Monitoring.
  3. Add all four monitors you created: Lumigo Dashboard (HTTP), Lumigo Ingestion Endpoint (HTTP), Lumigo Dashboard (TCP), and Lumigo Ingestion Endpoint (TCP).
  4. Set the page visibility: Public if you want to share it with your team via a bookmarked URL, or Private if it is for internal use only.
  5. Copy the generated status page URL and add it to your team's incident response runbook and internal wiki so engineers can check Lumigo's status immediately when traces go missing.

Putting it all together

| Monitor | Type | What it catches | |---|---|---| | https://app.lumigo.io | HTTP | Lumigo dashboard unavailable; team cannot view traces during incidents | | https://ga4gh.lumigo.io/api/spans | HTTP | Span ingestion endpoint down; all Lumigo tracers silently dropping data | | ga4gh.lumigo.io:443 | TCP | Network-level failure preventing tracer connections before HTTP layer responds | | app.lumigo.io:443 | TCP | TLS or network failure blocking dashboard access at the connection level |

When an alert fires, work through these steps in order:

  1. Check https://status.lumigo.io for an active Lumigo incident
  2. Run the curl probe against https://ga4gh.lumigo.io/api/spans from your local machine to confirm the failure is external
  3. If the endpoint is reachable from your machine but tracers are not sending, check that LUMIGO_TRACER_TOKEN is set and valid in all Lambda functions
  4. Enable LUMIGO_DEBUG=true on one function and invoke it to capture tracer output in CloudWatch Logs
  5. If a Lambda layer update preceded the alert, check the Lumigo changelog for the updated layer version and roll back if a regression is documented

What's next

  • Synthetic heartbeat from Lambda — deploy a minimal Lambda function that sends a test span to Lumigo on a schedule (every 5 minutes) and alerts via SNS if the span does not appear in the Lumigo dashboard within a defined window, giving you an end-to-end tracer health check
  • Monitor Lumigo's dependencies — Lumigo's SaaS platform runs on AWS; add HTTP monitors for the AWS service health endpoints (https://health.aws.amazon.com/health/status) relevant to the regions your Lambda functions run in, so you can correlate Lumigo outages with upstream AWS events
  • Alert on trace ingestion gaps — use Lumigo's own alerting (or a scheduled CloudWatch query) to detect when the trace count for a high-traffic function drops to zero, creating an application-level complement to Vigilmon's infrastructure-level checks

Get started free at vigilmon.online — no credit card, monitors start running in under a minute.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles