How to Monitor Telepresence with Vigilmon

Telepresence (from Ambassador Labs) lets developers run a service locally while it intercepts live Kubernetes traffic — dramatically speeding up the inner dev loop. But Telepresence depends on a cluster-side Traffic Manager and a set of webhook injectors. When those break, every developer's intercept session silently fails, and debugging the root cause inside Kubernetes takes valuable time.

External monitoring with Vigilmon gives you a clear signal: is the Traffic Manager healthy, are intercept connections alive, and is the webhook injection path working? This tutorial walks you through setting that up.

Why Telepresence needs external monitoring

Telepresence has a multi-component architecture:

| Component | Role | |---|---| | Traffic Manager | In-cluster deployment that routes intercepted traffic | | Traffic Agent | Sidecar injected into target pods | | Webhook | Mutating admission webhook that injects the Traffic Agent | | Ambassador Cloud | Optional cloud dashboard for intercept management |

Each component is a potential point of failure:

• Traffic Manager crash-loops — all intercept sessions die with a cryptic grpc: the client connection is closing error
• Webhook certificate expired — new pod deployments fail because the mutating webhook returns a TLS handshake error
• Traffic Agent injection fails — existing services lose the sidecar; new intercepts can't be started
• Cluster connectivity degraded — high latency on the local↔cluster tunnel makes development painful without an obvious cause
• RBAC change breaks the Traffic Manager — the Traffic Manager service account loses permissions; it can no longer patch pods

None of these show up as obvious cluster-level alerts. External monitoring is the fastest way to detect them.

What you'll need

• Kubernetes cluster with Telepresence 2.x installed (telepresence helm install)
• The Traffic Manager running in the ambassador namespace (default)
• A free Vigilmon account

Step 1: Verify the Traffic Manager is reachable

Telepresence's Traffic Manager exposes a gRPC API and an optional HTTP health endpoint. Check which ports are exposed:

kubectl get svc -n ambassador
# NAME              TYPE        CLUSTER-IP      PORT(S)
# traffic-manager   ClusterIP   10.96.55.12     8081/TCP,8082/TCP,8083/TCP

Port 8081 is the gRPC API. Port 8083 (when present) is the REST/health endpoint. Port-forward to verify:

kubectl port-forward -n ambassador svc/traffic-manager 8083:8083
curl -s http://localhost:8083/healthz
# {"status":"ok","version":"2.x.y"}

To expose the health endpoint externally (for Vigilmon to reach it), add an Ingress:

# traffic-manager-health-ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: traffic-manager-health
  namespace: ambassador
  annotations:
    nginx.ingress.kubernetes.io/use-regex: "true"
spec:
  ingressClassName: nginx
  rules:
    - host: telepresence-health.internal.example.com
      http:
        paths:
          - path: /healthz
            pathType: Exact
            backend:
              service:
                name: traffic-manager
                port:
                  number: 8083

Apply:

kubectl apply -f traffic-manager-health-ingress.yaml

Keep this endpoint on an internal domain or behind IP allowlisting — it should not be publicly accessible.

Step 2: Monitor the Traffic Manager health

In Vigilmon:

1. Click Add Monitor → HTTP(S)
2. URL: https://telepresence-health.internal.example.com/healthz
3. Interval: 1 minute
4. Expected status: 200
5. String assertion: response contains "status":"ok"
6. Timeout: 10 seconds
7. Name: Telepresence – Traffic Manager Health
8. Save

Step 3: Monitor webhook injection health

The mutating admission webhook is what injects the Traffic Agent sidecar into pods. If its TLS certificate expires, pod deployments will fail cluster-wide (for any namespace where Telepresence injection is enabled).

Check the webhook's current certificate expiry:

kubectl get mutatingwebhookconfiguration agent-injector -o jsonpath='{.webhooks[0].clientConfig.caBundle}' \
  | base64 -d | openssl x509 -noout -enddate
# notAfter=Dec 31 23:59:59 2026 GMT

In Vigilmon, monitor the webhook endpoint:

# Port-forward the webhook service
kubectl port-forward -n ambassador svc/agent-injector 8443:443

Add an HTTPS monitor in Vigilmon:

1. URL: https://agent-injector.ambassador.svc.cluster.local:443 (if Vigilmon agent is in-cluster) or use the Ingress path
2. Enable TLS certificate expiry monitoring: alert when cert expires within 14 days
3. Name: Telepresence – Webhook TLS

Step 4: Monitor intercept connection latency

High tunnel latency makes local development painful. You can instrument this with a simple probe that measures round-trip time to an in-cluster service:

# intercept-latency-probe.yaml
apiVersion: v1
kind: Pod
metadata:
  name: latency-probe
  namespace: ambassador
spec:
  containers:
    - name: probe
      image: curlimages/curl:8.5.0
      command:
        - /bin/sh
        - -c
        - |
          while true; do
            START=$(date +%s%3N)
            curl -fsS --max-time 5 http://traffic-manager:8083/healthz -o /dev/null
            END=$(date +%s%3N)
            LATENCY=$((END - START))
            echo "latency_ms=$LATENCY"
            sleep 10
          done
  restartPolicy: Always

For Vigilmon's purposes, add an HTTP monitor for the Traffic Manager with a response time threshold:

1. Edit the Traffic Manager Health monitor
2. Under Performance, set Alert if response time exceeds: 500ms
3. This fires if the Traffic Manager is alive but responding slowly — which correlates with tunnel degradation

Step 5: Monitor cluster connectivity from the developer's perspective

Telepresence works by creating a VPN-like tunnel. Add a dedicated connectivity check endpoint to your cluster:

# connectivity-probe.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: connectivity-probe
  namespace: ambassador
spec:
  replicas: 1
  selector:
    matchLabels:
      app: connectivity-probe
  template:
    metadata:
      labels:
        app: connectivity-probe
    spec:
      containers:
        - name: echo
          image: hashicorp/http-echo:0.2.3
          args:
            - "-text=cluster-ok"
            - "-listen=:8080"
          ports:
            - containerPort: 8080
---
apiVersion: v1
kind: Service
metadata:
  name: connectivity-probe
  namespace: ambassador
spec:
  selector:
    app: connectivity-probe
  ports:
    - port: 80
      targetPort: 8080

Expose via Ingress with host connectivity-probe.internal.example.com, then in Vigilmon:

1. Add Monitor → HTTP
2. URL: https://connectivity-probe.internal.example.com/
3. String assertion: response body contains cluster-ok
4. Name: Telepresence – Cluster Connectivity Probe

Step 6: Set up alerts

In Vigilmon → Alert Channels:

1. Add a Slack webhook for your platform team channel
2. Add PagerDuty for after-hours escalation
3. Set Alert after: 2 consecutive failures
4. Set Recovery notifications: enabled

Assign alert channels to all monitors tagged with telepresence.

A meaningful alert message:

:satellite: *Telepresence Traffic Manager is DOWN*
Monitor: {{ monitor.name }}
Status: {{ check.status_code }}
Region: {{ check.region }}
Impact: All developer intercept sessions may be broken.
Runbook: https://wiki.example.com/runbooks/telepresence

Step 7: Smoke-test your monitoring setup

Trigger a failure deliberately to confirm alerts fire:

# Scale down Traffic Manager (restore immediately after)
kubectl scale deploy traffic-manager -n ambassador --replicas=0

# Wait 2 minutes — Vigilmon should fire an alert
# Then restore:
kubectl scale deploy traffic-manager -n ambassador --replicas=1

Verify you received the Slack/PagerDuty alert and that Vigilmon shows the monitor as recovered once the deployment comes back.

Monitor checklist

| Monitor | What it detects | |---|---| | Traffic Manager /healthz | Traffic Manager process health | | Traffic Manager response time | Tunnel latency degradation | | Webhook TLS certificate | Expiring cert before pod injection breaks | | Cluster connectivity probe | End-to-end in-cluster reachability |

Summary

Telepresence is a productivity multiplier for Kubernetes teams, but its multi-component architecture creates several silent failure modes. Vigilmon's HTTP and TLS monitors give you continuous external visibility into Traffic Manager health, webhook certificate validity, and cluster connectivity — so you catch broken intercepts before every developer on your team loses their local dev environment.

Get started for free at vigilmon.online.