Monitoring Pixie with Vigilmon

Pixie is an open-source Kubernetes observability platform developed by New Relic and donated to the CNCF. Unlike traditional APM agents that require code instrumentation, Pixie uses eBPF kernel probes to automatically capture HTTP, gRPC, DNS, Postgres, Redis, and Kafka traffic with no changes to application code. Pixie's architecture runs a Proxemics Execution Module (PEM) as a DaemonSet on every node, which streams telemetry up to the Kelvin aggregation node and then to the Vizier service for querying via Pixie's PxL scripting language. When PEMs go unhealthy, node-level telemetry stops. When Kelvin degrades, cluster-wide queries return incomplete results. Vigilmon gives you the external monitoring layer to catch Pixie infrastructure failures before your engineers discover them through missing observability data.

What You'll Set Up

• HTTP monitor for the Pixie Cloud or Vizier API endpoint
• Vizier pod health check for in-cluster components
• Kelvin node agent connectivity monitoring
• PEM health status checks
• Data retention and streaming health
• SSL certificate monitoring for Pixie's cloud proxy

Prerequisites

• Pixie deployed on a Kubernetes cluster (version 1.24+) via Helm or the Pixie CLI
• The vizier-cloud-connector service is running and connected to Pixie Cloud (or your self-hosted Pixie Cloud)
• The Pixie API key and cluster ID available for authenticated checks
• A free Vigilmon account

Step 1: Monitor the Pixie Cloud Proxy Endpoint

Pixie's in-cluster Vizier connects to Pixie Cloud (or your self-hosted Pixie Cloud) over a persistent gRPC stream. The cloud endpoint is what your px CLI and Pixie UI communicate with. If the cloud endpoint is unreachable, the Vizier loses its connection and you can no longer run PxL queries — even if the in-cluster eBPF probes are still capturing data.

For the Pixie-hosted cloud:

GET https://withpixie.ai/api/healthz

A healthy response returns HTTP 200. Configure Vigilmon:

1. Log in to vigilmon.online and click Add Monitor.
2. Set Type to HTTP / HTTPS.
3. Enter URL: https://withpixie.ai/api/healthz.
4. Set Expected HTTP status to 200.
5. Set Check interval to 2 minutes.
6. Click Save.

For self-hosted Pixie Cloud:

1. Use your self-hosted Pixie Cloud domain: https://pixie-cloud.example.com/api/healthz.
2. Add the SSL certificate monitor for your domain too (Step 6 below).

This check validates that the Pixie Cloud control plane is reachable, which is the prerequisite for all in-cluster PxL query execution.

Step 2: Monitor the Vizier Cloud Connector

The vizier-cloud-connector deployment inside your cluster maintains the persistent connection between your Vizier and Pixie Cloud. When this connector is down, queries hang indefinitely and the Pixie UI shows the cluster as disconnected.

Expose the cloud-connector's health via a Kubernetes Service and ingress, then monitor it:

# patch-vizier-healthz-ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: vizier-cloud-connector-healthz
  namespace: pl
spec:
  rules:
  - host: vizier-health.example.com
    http:
      paths:
      - path: /healthz
        pathType: Prefix
        backend:
          service:
            name: vizier-cloud-connector
            port:
              number: 50800

Once exposed:

1. Add Monitor → HTTP / HTTPS.
2. URL: https://vizier-health.example.com/healthz.
3. Set Expected HTTP status to 200.
4. Set Check interval to 2 minutes.
5. Click Save.

Alternatively, if you have a Vigilmon agent inside the cluster, monitor http://vizier-cloud-connector.pl.svc:50800/healthz directly without an ingress.

Step 3: Monitor PEM Health via the Vizier API

Pixie's PEM (Proxemics Execution Module) DaemonSet runs one pod per node. PEMs are responsible for all eBPF data capture. A node where the PEM is CrashLoopBackOff produces no HTTP, gRPC, or DNS telemetry for workloads on that node — silently.

Use the Pixie API to query PEM status:

# Get Pixie API key
export PIXIE_API_KEY=<your-api-key>

# Query PEM status via Pixie's REST API
curl -H "Authorization: Bearer ${PIXIE_API_KEY}" \
  "https://withpixie.ai/api/cluster/<cluster-id>/status"

A healthy cluster returns:

{
  "status": "CS_HEALTHY",
  "numNodes": 5,
  "numInstrumentedNodes": 5
}

Configure Vigilmon to poll the cluster status endpoint:

1. Add Monitor → HTTP / HTTPS.
2. URL: https://withpixie.ai/api/cluster/<cluster-id>/status.
3. In Custom headers, add Authorization: Bearer <your-api-key>.
4. Set Expected body contains to "status":"CS_HEALTHY".
5. Set Check interval to 5 minutes.
6. Click Save.

When numInstrumentedNodes is less than numNodes, one or more PEMs are down. This often happens after node autoscaling events where new nodes join the cluster before the DaemonSet's PEM pod has fully initialized the eBPF probes.

Step 4: Monitor HTTP/gRPC Auto-Instrumentation Health

Pixie's HTTP and gRPC auto-instrumentation is what makes it useful — but it can fail silently on certain kernel versions, CPU architectures, or when eBPF verifier limits are hit. You can validate that Pixie is capturing HTTP traffic by running a PxL script against a known endpoint and checking for data.

For a lightweight Vigilmon-compatible check, expose a Pixie query endpoint via the px-operator or your own API gateway, then check it returns non-empty data:

GET https://pixie-query.example.com/api/check-http-capture

This endpoint (implemented as a simple wrapper around the Pixie Go client) runs:

# PxL query to check for recent HTTP data
import px
df = px.DataFrame(table='http_events', start_time='-5m')
df = df[['time_', 'req_path', 'resp_status']]
px.display(df.head(5))

If the response is empty (no HTTP events in the last 5 minutes), eBPF HTTP capture is not working. Configure Vigilmon:

1. Add Monitor → HTTP / HTTPS.
2. URL: https://pixie-query.example.com/api/check-http-capture.
3. Set Expected HTTP status to 200.
4. Set Expected body contains to req_path (confirms non-empty results).
5. Set Check interval to 10 minutes.
6. Click Save.

This check is only useful if your cluster has consistent HTTP traffic (which any production cluster does). A zero-result response is an early warning that eBPF probes have detached after a kernel update or node restart.

Step 5: Monitor Data Retention and Streaming Pipeline

Pixie stores recent telemetry in-cluster in PEM memory (configurable, default 2 GiB per PEM). When PEM memory fills up, old data is evicted. You can monitor the Pixie data pipeline's health by checking the kelvin aggregation node — if Kelvin is unhealthy, cluster-wide aggregations stop working.

Expose Kelvin's health endpoint:

# Add to your Kubernetes manifests or Helm values
apiVersion: v1
kind: Service
metadata:
  name: kelvin-healthz
  namespace: pl
spec:
  selector:
    name: kelvin
  ports:
  - port: 59300
    name: http

Then monitor:

1. Add Monitor → TCP Port.
2. Host: kelvin.pl.svc (via internal Vigilmon agent) or the ingress hostname.
3. Port: 59300.
4. Set Check interval to 3 minutes.
5. Click Save.

If you can expose the Kelvin HTTP health probe via ingress, use an HTTP check on /healthz instead of TCP — HTTP checks provide richer failure information when Kelvin is running but its internal state is degraded.

Step 6: SSL Certificate Monitoring

Pixie's cloud proxy and any self-hosted Pixie Cloud deployment use TLS for all communication. Certificate expiry breaks the Vizier-to-cloud connection and all PxL queries:

1. Add Monitor → SSL Certificate.
2. URL: https://withpixie.ai (or your self-hosted Pixie Cloud domain).
3. Set Alert when certificate expires in less than 30 days.
4. Click Save.

For self-hosted Pixie Cloud with custom domains, add SSL checks for each subdomain used by the cloud API and the cloud proxy.

Step 7: Configure Alert Channels

1. Go to Alert Channels in Vigilmon and configure your destination: Slack, PagerDuty, email, or webhook.
2. For the Pixie Cloud health and Vizier cloud-connector checks: Consecutive failures before alert = 1 — these are critical infrastructure-level failures.
3. For PEM health and HTTP capture checks: use 2 consecutive failures — brief gaps during node autoscaler events are normal.
4. Add a Maintenance window for Pixie upgrades (DaemonSet rolling restarts can take several minutes on large clusters).

Consider creating a Status page in Vigilmon for your platform engineering team — surfacing Pixie cluster health alongside your other observability infrastructure makes it clear when eBPF data capture is degraded vs. the application itself.

Going Further

• Kernel compatibility checks: Pixie requires kernel 4.14+ and certain kernel features. After node OS updates, the PEM may fail to re-attach eBPF probes. Add a post-update check that re-validates the cluster health endpoint and confirms numInstrumentedNodes == numNodes.
• Multi-cluster fleet monitoring: If you run Pixie across multiple clusters, add a cluster status check per cluster using the cluster-specific ID in the API URL. A single Vigilmon status page can aggregate health across your entire fleet.
• gRPC capture validation: Beyond HTTP events, Pixie captures gRPC calls in the grpc_events table. Add a separate PxL query check that validates gRPC data is flowing — gRPC over HTTP/2 uses a different eBPF probe path that can fail independently of HTTP/1.1 capture.
• Memory pressure monitoring: When PEM memory fills, Pixie evicts old data silently. Monitor the Kubernetes kubectl top pod -n pl memory usage for PEM pods via a custom exporter, or set Kubernetes resource alerts on the kelvin and PEM Deployments via your in-cluster monitoring stack — and use Vigilmon as the external validation layer.

With Vigilmon monitoring Pixie Cloud health, the Vizier cloud-connector, PEM instrumented-node counts, and HTTP capture validity, you have an independent external layer that tells you when your eBPF observability platform is itself unhealthy — the observer's observer that Pixie's own telemetry can't provide.