tutorial

Monitoring Azure Event Grid with Vigilmon: Subscriber Health Checks, Delivery Failure Alerts & Event Handler Heartbeats

Practical guide to uptime monitoring for Azure Event Grid — HTTP health endpoints for event handlers, heartbeat monitors for Azure Functions subscribers, delivery failure probes, and external alerting with Vigilmon.

Azure Event Grid routes events from dozens of Azure services to your handlers in milliseconds. But when a subscriber endpoint goes unhealthy, Event Grid retries for up to 24 hours before giving up and dropping events — and you won't see a red alert, just a slowly growing delivery failure count in Azure Monitor. Your alarms live inside the same Azure tenant as the problem. Vigilmon adds an external, independent monitoring layer: HTTP monitors on your event handler services, heartbeat monitors for Functions-based subscribers, and delivery failure probes that alert before you cross the retry cliff.

This tutorial shows you how to wire Event Grid subscribers into Vigilmon for reliable uptime monitoring.

What You'll Build

  • A health endpoint on your event handler service that validates Event Grid connectivity
  • A Vigilmon HTTP monitor with subscription confirmation and delivery health checks
  • A heartbeat pattern for Azure Functions Event Grid triggers
  • A delivery failure probe using Azure Monitor Metrics API
  • An alerting strategy for subscriber outages and event routing failures

Prerequisites

  • Azure subscription with at least one Event Grid topic or system topic and a subscriber
  • Azure CLI or Bicep/ARM configured locally
  • A free account at vigilmon.online

Step 1: Add a Health Endpoint to Your Event Handler Service

Event Grid HTTP subscribers must validate a subscription handshake and then handle event payloads. Add a /health endpoint separate from your event handler.

C# (ASP.NET Core)

// Endpoints/HealthEndpoint.cs
using Azure;
using Azure.Messaging.EventGrid;

public static class HealthEndpoint
{
    public static void MapHealthEndpoint(this WebApplication app)
    {
        app.MapGet("/health", async (IConfiguration config) =>
        {
            var checks = new Dictionary<string, object>();
            var ok = true;

            try
            {
                // Probe the Event Grid topic endpoint to confirm access
                var topicEndpoint = new Uri(config["EventGridTopicEndpoint"]);
                var credential = new AzureKeyCredential(config["EventGridTopicKey"]);
                var client = new EventGridPublisherClient(topicEndpoint, credential);

                // Dry-run: we don't actually publish, just verify the client can be constructed
                // For a real connectivity check, publish a low-cost "health" event type
                checks["eventGrid"] = "ok";
                checks["topicEndpoint"] = topicEndpoint.Host;
            }
            catch (Exception ex)
            {
                checks["eventGrid"] = $"error: {ex.Message}";
                ok = false;
            }

            return ok
                ? Results.Ok(new { status = "ok", service = "eventgrid-handler", checks })
                : Results.Json(new { status = "degraded", service = "eventgrid-handler", checks },
                    statusCode: 503);
        });
    }
}

Python (FastAPI)

# routers/health.py
import os
from fastapi import APIRouter
from azure.eventgrid import EventGridPublisherClient
from azure.core.credentials import AzureKeyCredential
from azure.core.exceptions import HttpResponseError

router = APIRouter()

@router.get("/health")
async def health():
    checks = {}
    ok = True

    try:
        client = EventGridPublisherClient(
            endpoint=os.environ["EVENTGRID_TOPIC_ENDPOINT"],
            credential=AzureKeyCredential(os.environ["EVENTGRID_TOPIC_KEY"]),
        )
        # Verify credentials resolve without error
        checks["eventGrid"] = "ok"
        checks["topicEndpoint"] = os.environ["EVENTGRID_TOPIC_ENDPOINT"].split("/")[2]
    except Exception as e:
        checks["eventGrid"] = f"error: {e}"
        ok = False

    return {
        "status": "ok" if ok else "degraded",
        "service": "eventgrid-handler",
        "checks": checks,
    }

Node.js (Express)

// routes/health.js
import { EventGridPublisherClient, AzureKeyCredential } from "@azure/eventgrid";

export async function healthHandler(req, res) {
  const checks = {};
  let ok = true;

  try {
    const client = new EventGridPublisherClient(
      process.env.EVENTGRID_TOPIC_ENDPOINT,
      "EventGrid",
      new AzureKeyCredential(process.env.EVENTGRID_TOPIC_KEY)
    );
    checks.eventGrid = "ok";
    checks.topicEndpoint = new URL(process.env.EVENTGRID_TOPIC_ENDPOINT).hostname;
  } catch (err) {
    checks.eventGrid = `error: ${err.message}`;
    ok = false;
  }

  res.status(ok ? 200 : 503).json({
    status: ok ? "ok" : "degraded",
    service: "eventgrid-handler",
    checks,
  });
}

Step 2: Handle Subscription Validation in Your Handler

Event Grid sends a SubscriptionValidationEvent when you create a webhook subscription. Your handler must respond to it — if it doesn't, the subscription stays unconfirmed and no events are delivered.

// routes/events.js
export async function eventsHandler(req, res) {
  const events = req.body;

  for (const event of events) {
    if (event.eventType === "Microsoft.EventGrid.SubscriptionValidationEvent") {
      // Respond with the validation code to confirm the subscription
      const validationCode = event.data.validationCode;
      return res.json({ validationResponse: validationCode });
    }

    await processEvent(event);
  }

  res.sendStatus(200);
}

Add a check for subscription confirmation status in your health endpoint:

// Enhanced health with subscription check
checks.subscriptionHandler = req.app.locals.subscriptionConfirmed ? "confirmed" : "pending";
if (!req.app.locals.subscriptionConfirmed) ok = false;

Set app.locals.subscriptionConfirmed = true after your handler successfully responds to the first SubscriptionValidationEvent.

Step 3: Configure the Vigilmon HTTP Monitor

  1. Log in to Vigilmon → Add Monitor → HTTP.
  2. URL: your event handler's health endpoint (e.g. https://handlers.example.com/health).
  3. Check interval: 60 seconds.
  4. Response timeout: 10 seconds.
  5. Expected status: 200.
  6. JSON assertion: path status, expected value ok.

Tip: If your handler is behind Azure Application Gateway, ensure the /health route is allowed through WAF rules. Event Grid also sends validation requests from specific IP ranges — add your Vigilmon check IP separately if your WAF blocks unknown sources.

Step 4: Heartbeat Monitoring for Azure Functions Event Grid Triggers

Azure Functions with Event Grid triggers are serverless — no persistent process to probe. Use Vigilmon's Heartbeat monitor and ping it after processing each batch.

// Functions/EventHandler.cs
[FunctionName("HandleOrderEvents")]
public async Task Run(
    [EventGridTrigger] EventGridEvent eventGridEvent,
    ILogger log)
{
    log.LogInformation($"Processing event: {eventGridEvent.EventType}");

    switch (eventGridEvent.EventType)
    {
        case "Order.Created":
            await HandleOrderCreated(eventGridEvent.Data.ToObjectFromJson<OrderCreatedData>());
            break;
        case "Order.Cancelled":
            await HandleOrderCancelled(eventGridEvent.Data.ToObjectFromJson<OrderCancelledData>());
            break;
    }

    // Ping Vigilmon heartbeat after successful processing
    using var http = new HttpClient();
    await http.PostAsync(
        Environment.GetEnvironmentVariable("VIGILMON_HEARTBEAT_URL"),
        null
    );

    log.LogInformation("Event processed — heartbeat sent");
}

In Vigilmon, create a Heartbeat monitor:

  • Set the grace period to 2–3× the maximum expected gap between events for your topic.
  • For topics with sporadic traffic, set a longer grace period (e.g., 30 minutes) and generate synthetic test events on a schedule.

Store the heartbeat URL as a Function App setting:

az functionapp config appsettings set \
  --name my-function-app \
  --resource-group my-rg \
  --settings VIGILMON_HEARTBEAT_URL="https://vigilmon.online/api/heartbeat/<your-id>"

Step 5: Delivery Failure Probe

Event Grid exposes delivery failure metrics via Azure Monitor. A lightweight probe Lambda (or Azure Function) can read these and withhold a Vigilmon heartbeat when failures accumulate.

# delivery-probe/handler.py
import os
from datetime import datetime, timedelta, timezone
from azure.monitor.query import MetricsQueryClient
from azure.identity import DefaultAzureCredential
import urllib.request

credential = DefaultAzureCredential()
metrics_client = MetricsQueryClient(credential)

def handler(event, context):
    resource_id = os.environ["EVENTGRID_TOPIC_RESOURCE_ID"]
    end = datetime.now(timezone.utc)
    start = end - timedelta(minutes=5)

    result = metrics_client.query_resource(
        resource_id,
        metric_names=["DeliveryFailedCount"],
        timespan=(start, end),
        granularity=timedelta(minutes=5),
    )

    total_failures = 0
    for metric in result.metrics:
        for ts in metric.timeseries:
            for dp in ts.data:
                total_failures += dp.total or 0

    if total_failures == 0:
        req = urllib.request.Request(
            os.environ["VIGILMON_HEARTBEAT_URL"],
            method="POST"
        )
        urllib.request.urlopen(req)
        print("No delivery failures — heartbeat sent")
    else:
        print(f"Event Grid delivery failures: {int(total_failures)} — heartbeat withheld")

Schedule this as an Azure Function on a timer trigger every 5 minutes. Set the Vigilmon heartbeat grace period to 10 minutes.

Step 6: Dead-Letter Monitoring

When Event Grid exhausts retries, it moves events to a dead-letter storage blob (if configured). Monitor the dead-letter blob for new files as a secondary signal.

# dead-letter-probe/handler.py
import os
from datetime import datetime, timedelta, timezone
from azure.storage.blob import BlobServiceClient
import urllib.request

def handler(event, context):
    client = BlobServiceClient.from_connection_string(os.environ["DEADLETTER_STORAGE_CONNECTION"])
    container = client.get_container_client(os.environ["DEADLETTER_CONTAINER"])

    # Check for blobs written in the last 10 minutes
    cutoff = datetime.now(timezone.utc) - timedelta(minutes=10)
    recent_blobs = [
        b for b in container.list_blobs()
        if b.last_modified and b.last_modified > cutoff
    ]

    if not recent_blobs:
        # No dead-lettered events — send heartbeat
        req = urllib.request.Request(os.environ["VIGILMON_HEARTBEAT_URL"], method="POST")
        urllib.request.urlopen(req)
        print("No dead-lettered events — heartbeat sent")
    else:
        print(f"{len(recent_blobs)} dead-lettered events in last 10 min — heartbeat withheld")

Step 7: Alerting Strategy

| Alert type | Recommended channel | |---|---| | Handler service 503 | Email + PagerDuty | | Functions heartbeat missed | Slack + PagerDuty | | Delivery failure probe triggered | Slack + email | | Dead-letter probe triggered | PagerDuty (events lost) | | System topic source degraded | Status page |

Set alert escalation: notify immediately on first failure, re-notify every 5 minutes if still down. Event Grid's 24-hour retry window sounds generous, but exponential backoff means most retries happen in the first hour.

What Vigilmon Catches That Azure Monitor Misses

| Scenario | Azure Monitor | Vigilmon | |---|---|---| | Handler service crashes (HTTP 503) | Needs ALB health probe + alert rule | HTTP monitor catches immediately | | Functions trigger stalls | Needs custom metric + alert | Heartbeat grace period fires alert | | Subscription not confirmed | No built-in alert | Health endpoint checks confirmation status | | Dead-letter blob accumulates | Needs Storage metric + alert | Dead-letter probe + heartbeat | | Azure Monitor itself is degraded | Self-monitoring blind spot | Vigilmon is external — unaffected | | Cross-region topic failover | Needs geo-redundant alert config | One monitor per topic endpoint, same alert channel |

Event Grid makes reactive architectures feel effortless — until an event handler goes down and 24 hours of retries quietly eat your event backlog. External monitoring from Vigilmon gives you the independent signal you need — before Event Grid gives up and dead-letters your events.

Start monitoring your Event Grid handlers in under 5 minutes — register free at vigilmon.online.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles