tutorial

Monitoring Dapr with Vigilmon: Sidecar Health, State Store Connectivity, Pub/Sub Broker Health & Service Invocation Latency

How to monitor Dapr with Vigilmon — sidecar health API checks, state store and pub/sub broker connectivity, actor runtime metrics, and service invocation latency tracking.

Dapr (Distributed Application Runtime) is the CNCF project that gives microservices a standard API for state management, pub/sub messaging, service invocation, secrets, and actors — all through a sidecar proxy injected alongside each application pod. When the Dapr sidecar crashes or becomes unresponsive, your application loses access to every Dapr building block simultaneously: state reads fail, pub/sub messages stop flowing, and service-to-service calls fall back to direct HTTP or error out entirely. When the Dapr control plane (the operator, placement service, or sentry) degrades, new sidecar injections stop working and actor placement breaks across the cluster. Vigilmon gives you external visibility into the health of your Dapr-powered applications, control plane components, and the backing infrastructure that Dapr depends on.

What You'll Build

  • HTTP monitors on Dapr sidecar health endpoints to detect sidecar failures per service
  • Control plane availability checks for the Dapr operator, sentry, and placement service
  • State store and pub/sub broker connectivity monitoring via Dapr's component health API
  • Service invocation latency tracking using Vigilmon HTTP monitors on app endpoints
  • Actor runtime health monitoring for Dapr actor-based services
  • Alerting configuration that maps Dapr failure modes to actionable runbook steps

Prerequisites

  • A Kubernetes cluster with Dapr 1.x installed (via Dapr CLI or Helm)
  • At least one application pod with the Dapr sidecar injected (dapr.io/enabled: "true" annotation)
  • Dapr control plane components running in the dapr-system namespace
  • Applications exposed via Kubernetes services or Ingress
  • A free account at vigilmon.online

Step 1: Understand Dapr's Health Endpoints

Every Dapr sidecar exposes a health API on port 3500 (the default Dapr HTTP port). The primary health check endpoint is:

# Check sidecar health from within the cluster
kubectl exec -n production myapp-pod -c myapp -- \
  curl http://localhost:3500/v1.0/healthz
# Returns HTTP 204 when healthy (no body)

The /v1.0/healthz endpoint returns:

  • 204 No Content — sidecar is healthy and all configured components are ready
  • 500 Internal Server Error — sidecar is running but one or more components failed to initialize

For outbound health checks (from Vigilmon's external perspective), you need the sidecar port exposed through a Kubernetes Service or via port-forwarding for testing:

# Temporary port-forward for testing
kubectl -n production port-forward pod/myapp-pod 3500:3500 &
curl -v http://localhost:3500/v1.0/healthz
# Expect: HTTP/1.1 204 No Content

The Dapr control plane components have their own health endpoints:

# Dapr operator
kubectl -n dapr-system port-forward svc/dapr-operator 6500:6500 &
curl http://localhost:6500/healthz

# Dapr sentry (mTLS certificate authority)
kubectl -n dapr-system port-forward svc/dapr-sentry 8080:8080 &
curl http://localhost:8080/healthz

# Dapr placement (actor placement)
kubectl -n dapr-system port-forward svc/dapr-placement-server 8080:8080 &
curl http://localhost:8080/healthz

Step 2: Monitor Application Endpoints That Depend on Dapr

The most practical external monitoring target is your application's own health endpoint — which should check Dapr component connectivity as part of its health response. If your app exposes a /health or /ready endpoint that queries Dapr state store or pub/sub connectivity, monitoring it gives you a combined view of app + Dapr health:

// Example Go health handler that checks Dapr state store
func healthHandler(w http.ResponseWriter, r *http.Request) {
    client, _ := dapr.NewClient()
    _, err := client.GetState(r.Context(), "statestore", "health-probe", nil)
    if err != nil {
        http.Error(w, "dapr state store unreachable", http.StatusServiceUnavailable)
        return
    }
    w.WriteHeader(http.StatusOK)
    json.NewEncoder(w).Encode(map[string]string{"status": "healthy"})
}
  1. Log in to Vigilmon → Add Monitor → HTTP.
  2. URL: https://myapp.example.com/health.
  3. Check interval: 60 seconds.
  4. Response timeout: 10 seconds.
  5. Expected status: 200.
  6. Keyword: healthy (or your app's health response field).
  7. Label: myapp (Dapr-integrated health).
  8. Click Save.

This catches Dapr state store disconnects, pod restarts that kill the sidecar before the app, and configuration changes that break component initialization.

Step 3: Monitor the Dapr Component Health API

Dapr exposes a component-level health endpoint that checks whether each configured component (state stores, pub/sub brokers, bindings) is reachable and initialized. When exposed externally, this endpoint reveals which specific building block is failing:

# Check all component health via the sidecar
curl http://localhost:3500/v1.0/healthz/outbound
# Returns: 200 OK when all outbound components are healthy
# Returns: 500 when any component is failing

# Check Dapr metadata (shows which components are loaded)
curl http://localhost:3500/v1.0/metadata

If you expose the Dapr sidecar port via a dedicated internal service for monitoring:

# NodePort service to expose Dapr sidecar health for monitoring
apiVersion: v1
kind: Service
metadata:
  name: myapp-dapr-health
  namespace: production
spec:
  selector:
    app: myapp
  ports:
    - name: dapr-http
      port: 3500
      targetPort: 3500
  type: ClusterIP

For each critical service, add a Vigilmon monitor targeting the sidecar health endpoint via your ingress or internal gateway.

Step 4: Monitor State Store Connectivity

Dapr state stores (Redis, PostgreSQL, Azure Cosmos DB, etc.) are often the most critical dependency in a Dapr application. Monitor the backing store directly in addition to the Dapr component health endpoint:

For Redis (common Dapr default state store):

# Check Redis health
redis-cli -h redis.production.svc.cluster.local ping
# Returns: PONG

If Redis is exposed via an external address:

  1. Add Monitor → TCP.
  2. Host: redis.example.com.
  3. Port: 6379.
  4. Check interval: 2 minutes.
  5. Label: Redis (Dapr state store).
  6. Click Save.

For PostgreSQL state store:

  1. Add Monitor → TCP.
  2. Host: postgres.example.com.
  3. Port: 5432.
  4. Check interval: 2 minutes.
  5. Label: PostgreSQL (Dapr state store).
  6. Click Save.

A state store TCP monitor that fires before your app's health endpoint indicates the problem is in the backing infrastructure, not in Dapr's component layer — this distinction saves significant debugging time.

Step 5: Monitor Pub/Sub Broker Health

Dapr pub/sub relies on a message broker (Redis Streams, Kafka, RabbitMQ, Azure Service Bus, etc.). When the broker is unreachable, Dapr's pub/sub API returns errors silently unless your application handles and surfaces them. Monitor the broker directly:

For RabbitMQ:

  1. Add Monitor → HTTP.
  2. URL: https://rabbitmq.example.com/api/healthchecks/node.
  3. Check interval: 2 minutes.
  4. Response timeout: 10 seconds.
  5. Expected status: 200.
  6. Keyword: ok.
  7. Label: RabbitMQ (Dapr pub/sub broker).
  8. Click Save.

For Kafka (if you expose a REST proxy or health endpoint):

# Kafka REST Proxy health
curl https://kafka-rest.example.com/
# Returns broker metadata JSON
  1. Add Monitor → HTTP.
  2. URL: https://kafka-rest.example.com/topics.
  3. Check interval: 2 minutes.
  4. Expected status: 200.
  5. Label: Kafka (Dapr pub/sub broker).
  6. Click Save.

Pub/sub silent failures: Dapr's pub/sub component does not surface broker failures as HTTP errors on the /v1.0/publish endpoint in all versions. Always monitor the broker independently — a healthy-looking app can be silently dropping messages into a broken broker.

Step 6: Monitor Service Invocation Latency

Dapr's service invocation building block routes requests through the sidecar, adding a small latency overhead compared to direct HTTP calls. Monitoring the latency of Dapr-invoked endpoints lets you detect sidecar performance degradation before it becomes a user-facing issue:

# Test service invocation through Dapr sidecar
curl -w "@curl-format.txt" -s \
  http://localhost:3500/v1.0/invoke/orderservice/method/orders/status

In Vigilmon:

  1. Add Monitor → HTTP.
  2. URL: https://myapp.example.com/api/orders/status (the externally visible endpoint backed by Dapr service invocation).
  3. Check interval: 60 seconds.
  4. Response timeout: 5 seconds (Dapr invocation should be fast; a 5-second timeout catches sidecar latency spikes).
  5. Expected status: 200.
  6. Label: Order service (Dapr invocation latency).
  7. Click Save.

Set the response timeout conservatively (2-5 seconds). When Dapr sidecar overhead causes requests to exceed your SLA, Vigilmon's timeout-triggered alert fires before customers notice.

Step 7: Monitor Actor Runtime Health

Dapr's actor runtime requires the placement service to be healthy for actor activation and load balancing. Actor-based services expose an actor health endpoint that Dapr calls periodically:

# Dapr calls this endpoint to verify actor health
# Your app must respond to:
GET /healthz
# And for actors:
GET /dapr/config

Monitor the actor health endpoint:

  1. Add Monitor → HTTP.
  2. URL: https://actorservice.example.com/healthz.
  3. Check interval: 60 seconds.
  4. Expected status: 200.
  5. Label: Actor service (Dapr runtime health).
  6. Click Save.

Also monitor the Dapr placement service via TCP since actor placement failure prevents any actor from being activated across the entire cluster:

  1. Add Monitor → TCP.
  2. Host: dapr-placement-server.dapr-system.svc.cluster.local (or your external address).
  3. Port: 50005 (gRPC placement port).
  4. Check interval: 2 minutes.
  5. Label: Dapr placement (actor runtime).
  6. Click Save.

Step 8: Configure Alerting

In Vigilmon under Settings → Notifications, map each monitor to its response runbook:

| Monitor | Trigger | Immediate action | |---|---|---| | App health endpoint | Non-200 | Check app logs and Dapr sidecar: kubectl logs myapp-pod -c daprd | | State store TCP | Connection refused | Check Redis/Postgres; Dapr state operations will fail across all services using this store | | Pub/sub broker | Non-200 or connection refused | Message publishing silently failing; check broker logs; inspect Dapr component status | | Service invocation endpoint | Timeout | Dapr sidecar performance issue; check kubectl top pod for sidecar resource pressure | | Actor service healthz | Non-200 | Actor runtime degraded; check placement service and sidecar logs | | Dapr placement TCP | Connection refused | Actor placement broken cluster-wide; no new actors can be activated |

Common Dapr Failure Modes and What Vigilmon Catches

| Scenario | Vigilmon signal | |---|---| | Dapr sidecar OOM killed | App health endpoint fires; sidecar restart causes brief service disruption | | State store Redis evicts all keys | App health check fails if it probes state; TCP monitor stays green (Redis is up) | | Pub/sub broker network partition | Broker monitor fires; app appears healthy but messages silently dropped | | Dapr sentry certificate rotation failure | mTLS between sidecars fails; service invocation latency spikes then times out | | Placement service pod crash | Actor monitor fires; TCP placement monitor fires; actor invocations fail | | Component misconfiguration after Dapr upgrade | App health endpoint fires (component init fails); sidecar /healthz returns 500 | | Node pool replacement evicts Dapr control plane | Multiple monitors fire simultaneously | | State store connection pool exhaustion | App health degrades intermittently; TCP monitor stays green; increase pool size | | Dapr operator down (no new sidecar injections) | Not caught by external monitoring; only new pod deployments affected |

Dapr's sidecar architecture makes every building block a potential failure point — state stores, pub/sub brokers, the placement service, and the sidecar itself all need independent visibility. Vigilmon gives you external eyes on the health of each layer: your application's Dapr-integrated health check, the backing infrastructure that Dapr components depend on, service invocation latency that reveals sidecar overhead, and actor runtime health that depends on the placement service being reachable. When Dapr's internal observability doesn't surface a problem, Vigilmon's external checks will.

Start monitoring your Dapr applications in under 5 minutes — register free at vigilmon.online.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles