tutorial

How to Monitor Buf Schema Registry and gRPC Services with Vigilmon

Buf is the standard toolchain for Protocol Buffers — it lints and formats your `.proto` files, generates type-safe client and server code, enforces backward ...

Buf is the standard toolchain for Protocol Buffers — it lints and formats your .proto files, generates type-safe client and server code, enforces backward compatibility rules, and hosts a schema registry (Buf Schema Registry, BSR) that your teams use like a package manager for APIs. When your gRPC services and Buf-managed schemas are the backbone of your internal platform, their availability directly determines whether product teams can ship.

In this tutorial you'll set up monitoring for your Buf-backed gRPC services and BSR integrations using Vigilmon — free tier, no credit card required.

Why Buf services need external monitoring

Buf's tooling tells you whether your schemas are valid and breaking-change-free. It doesn't tell you whether the runtime services implementing those schemas are actually serving traffic:

  • gRPC services don't expose HTTP health by default — a gRPC service is healthy from the client's perspective only if it responds to RPCs; a misconfigured load balancer or broken TLS handshake can make it unreachable without any error in your CI pipeline
  • Buf Schema Registry connectivity — if your internal developer tooling depends on BSR for schema resolution, a BSR connectivity failure silently breaks buf generate across all teams
  • Transcoding proxy failures — teams often use Envoy or grpc-gateway to expose gRPC services over HTTP/JSON for browser clients; the transcoding layer can fail while the underlying gRPC service is healthy
  • Reflection endpoint availability — gRPC Server Reflection is often used by developer tooling (grpcurl, Postman, BloomRPC); its failure disrupts debugging without affecting production RPCs
  • Certificate rotation — gRPC with mTLS or TLS is sensitive to certificate expiry; a lapsed cert causes connection failures that can cascade across many dependent services

Vigilmon monitors your gRPC services through their HTTP/JSON transcoding layers, health check endpoints, and TCP ports — giving you continuous external visibility from multiple geographic regions.

What you'll need

  • A gRPC service with a public or VPN-accessible endpoint
  • At least one of: HTTP/JSON transcoding via grpc-gateway or Envoy, a gRPC health check endpoint exposed over HTTP, or a TCP-accessible port
  • A free Vigilmon account (sign up takes 30 seconds)

Step 1: Expose a monitorable health endpoint

gRPC services typically implement the gRPC Health Checking Protocol. To make this monitorable by Vigilmon (which uses HTTP probes), expose it over HTTP via one of these approaches:

Option A: grpc-gateway HTTP bridge

If you're already using grpc-gateway for HTTP/JSON transcoding, add a dedicated health route:

// health.go
package main

import (
    "encoding/json"
    "net/http"
    "time"
)

func healthHandler(w http.ResponseWriter, r *http.Request) {
    w.Header().Set("Content-Type", "application/json")
    json.NewEncoder(w).Encode(map[string]interface{}{
        "status": "ok",
        "service": "user-service",
        "ts": time.Now().UTC().Format(time.RFC3339),
    })
}

// In your HTTP mux setup
mux.HandleFunc("/healthz", healthHandler)

Option B: Envoy admin endpoint

If you're running Envoy as a sidecar or gateway, enable the admin interface for health checks:

# envoy.yaml (excerpt)
admin:
  address:
    socket_address:
      address: 0.0.0.0
      port_value: 9901

static_resources:
  listeners:
  - name: health_listener
    address:
      socket_address:
        address: 0.0.0.0
        port_value: 8080
    filter_chains:
    - filters:
      - name: envoy.filters.network.http_connection_manager
        typed_config:
          route_config:
            virtual_hosts:
            - routes:
              - match: { prefix: "/healthz" }
                direct_response:
                  status: 200
                  body:
                    inline_string: '{"status":"ok"}'

Option C: Standalone HTTP sidecar

Add a minimal HTTP health server alongside your gRPC server:

// Runs alongside gRPC on a separate port
go func() {
    http.HandleFunc("/healthz", func(w http.ResponseWriter, r *http.Request) {
        // Optionally check internal state here
        w.WriteHeader(http.StatusOK)
        w.Write([]byte(`{"status":"ok"}`))
    })
    http.ListenAndServe(":8080", nil)
}()

Verify the endpoint responds:

curl https://user-service.yourcompany.com/healthz
# {"status":"ok","service":"user-service","ts":"2024-01-15T10:23:00Z"}

Step 2: Set up HTTP monitoring in Vigilmon

  1. Log in to vigilmon.online and go to Monitors → New Monitor
  2. Choose HTTP / HTTPS
  3. Set the URL to your health endpoint (e.g., https://user-service.yourcompany.com/healthz)
  4. Set the check interval to 1 minute
  5. Under Expected response:
    • Status code: 200
    • (Optional) Response body contains: "status":"ok"
  6. Save the monitor

Monitor your gRPC-gateway transcoding endpoints

If your services expose REST APIs via grpc-gateway, monitor those too — they're the layer your browser clients and third-party integrations hit:

# Verify your REST transcoding endpoint
curl https://api.yourcompany.com/v1/users
# {"users": [...]}

Add a Vigilmon monitor for each public transcoding endpoint:

| Service | gRPC port | Health/REST endpoint | Vigilmon monitor | |---------|-----------|----------------------|------------------| | user-service | 50051 | https://api.yourcompany.com/v1/users | grpc/user-service | | order-service | 50052 | https://api.yourcompany.com/v1/orders | grpc/order-service | | catalog-service | 50053 | https://api.yourcompany.com/v1/catalog | grpc/catalog-service |

Step 3: TCP monitoring for raw gRPC ports

HTTP health endpoints catch application-level failures. TCP monitoring catches port-level failures — a gRPC server that fails to bind, a load balancer with no healthy backends, or a firewall rule that blocks traffic before the TLS handshake completes.

Add a TCP monitor for each gRPC service port:

  1. In Vigilmon, go to Monitors → New Monitor
  2. Choose TCP Port
  3. Enter the host and port (e.g., user-service.yourcompany.com and 443 for TLS, or 50051 for plaintext gRPC)
  4. Save the monitor

TCP monitoring is particularly valuable for gRPC because HTTP/2 multiplexing means a single bad TCP connection can affect many concurrent RPCs simultaneously.

Step 4: Monitor Buf Schema Registry connectivity

If your team uses BSR for schema distribution, an outage — or a self-hosted BSR instance going down — silently breaks buf generate, buf push, and any tooling that fetches schemas at runtime.

Test your BSR connectivity first:

# For Buf Cloud
curl -H "Authorization: Bearer $BUF_TOKEN" \
  https://buf.build/api/v1/registry/repositories

# For self-hosted BSR
curl -H "Authorization: Bearer $BUF_TOKEN" \
  https://buf.yourcompany.com/api/v1/registry/repositories

If your self-hosted BSR exposes an HTTP health endpoint, add it to Vigilmon:

# Verify self-hosted BSR health
curl https://buf.yourcompany.com/healthz
# {"status":"ok"}

Add a Vigilmon HTTP monitor for https://buf.yourcompany.com/healthz.

For Buf Cloud users, you can monitor your own API access by adding a monitor that checks your registry's primary endpoint — if it returns auth errors rather than 200, your schema distribution is broken even if Buf Cloud itself is up.

Step 5: Heartbeat monitoring for Buf CI pipelines

Buf CLI runs in your CI pipelines to lint schemas, check for breaking changes, and generate code. These jobs should run on every merge. Use heartbeat monitors to verify they're running:

  1. In Vigilmon, go to Monitors → New Monitor
  2. Choose Heartbeat
  3. Set the expected interval to match your pipeline cadence (e.g., daily for scheduled lint jobs)
  4. Copy the heartbeat URL

Add a heartbeat ping to your CI pipeline:

# .github/workflows/buf.yml
name: Buf CI

on:
  push:
    branches: [main]
  schedule:
    - cron: '0 2 * * *'  # Daily at 2am

jobs:
  buf:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Buf lint
        uses: bufbuild/buf-action@v1
        with:
          lint: true

      - name: Buf breaking change detection
        uses: bufbuild/buf-action@v1
        with:
          breaking: true
          breaking_against: 'https://buf.build/yourorg/yourrepo.git#branch=main'

      - name: Ping Vigilmon heartbeat
        run: curl -fsS --retry 3 "https://hb.vigilmon.online/your-heartbeat-id" > /dev/null

If your nightly lint job fails or your CI pipeline stops running (repository archived, workflow disabled, runner quota exhausted), Vigilmon alerts you.

Step 6: Configure alert channels

When a gRPC service goes down, downstream clients see connection errors or cascading timeouts — it spreads fast. Your team needs immediate notification.

Slack webhook alerts

  1. In Vigilmon, go to Alert Channels → Add Channel → Webhook
  2. Enter your Slack incoming webhook URL
  3. Assign the channel to your gRPC service monitors

Vigilmon's downtime payload for integration with your internal tooling:

{
  "monitor_name": "grpc/user-service",
  "status": "down",
  "url": "https://user-service.yourcompany.com/healthz",
  "started_at": "2024-01-15T10:23:00Z",
  "duration_seconds": 45
}

Diagnosing gRPC/Buf failures when alerts fire

# Test the gRPC service directly with grpcurl
grpcurl -plaintext user-service.yourcompany.com:50051 list

# Check health via gRPC Health Protocol
grpcurl -plaintext user-service.yourcompany.com:50051 \
  grpc.health.v1.Health/Check

# Test the HTTP health endpoint
curl -v https://user-service.yourcompany.com/healthz

# Check TLS certificate
openssl s_client -connect user-service.yourcompany.com:443 \
  -servername user-service.yourcompany.com </dev/null 2>&1 | \
  grep -E "subject|issuer|notAfter"

# Test TCP connectivity
nc -zv user-service.yourcompany.com 50051

Step 7: SSL/TLS certificate monitoring

gRPC with TLS is particularly sensitive to certificate issues — an expired cert causes every RPC to fail immediately, and the error is often cryptic on the client side ("transport: authentication handshake failed").

Vigilmon monitors your TLS certificate expiry automatically for HTTPS monitors. Alerts at:

  • 30 days before expiry — time to rotate before impact
  • 7 days before expiry — urgent
  • On expiry — monitor starts reporting down

For services with mTLS, add an HTTPS monitor targeting the public-facing endpoint. Certificate issues in the chain show up as HTTP failures.

Putting it all together

| Monitor | Type | What it catches | |---------|------|-----------------| | https://user-service.yourcompany.com/healthz | HTTP | App failures, routing issues | | user-service.yourcompany.com:50051 | TCP | gRPC port binding failures | | https://api.yourcompany.com/v1/users | HTTP | Transcoding proxy failures | | https://buf.yourcompany.com/healthz | HTTP | Self-hosted BSR outage | | Buf CI heartbeat | Heartbeat | Schema lint pipeline failures |

What's next

  • Per-service status pages — create Vigilmon status pages organized by your Buf module structure so platform consumers can check service health independently
  • SLO tracking — use Vigilmon's uptime history to calculate availability SLOs for each gRPC service and share them with internal consumers
  • Multi-region gRPC monitoring — Vigilmon probes from multiple regions, which catches gRPC load balancing failures that only affect traffic from specific geographic areas

Get started free at vigilmon.online — no credit card, monitors start running in under a minute.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles