How to Monitor Semantic Kernel with Vigilmon

Semantic Kernel is Microsoft's open-source LLM SDK for .NET, Python, and Java that enables enterprise AI application development. Its plugin-based architecture composes AI capabilities from multiple sources; memory and vector store abstractions handle semantic search; planner modules decompose complex goals into subtasks autonomously; and it integrates natively with Azure OpenAI, OpenAI, Hugging Face, and local models.

Enterprise AI applications built on Semantic Kernel are often customer-facing copilots, internal productivity tools, or automated pipelines — all of which have uptime expectations. When the LLM backend is degraded, a plugin fails silently, or the vector store becomes unavailable, users experience broken or degraded AI features. Monitoring with Vigilmon gives you external visibility into the health of every component in your Semantic Kernel stack.

Why Monitor Semantic Kernel

Semantic Kernel applications have more failure points than a simple API wrapper. Each component can degrade independently:

1. LLM provider unavailability — Azure OpenAI, OpenAI, or Hugging Face endpoints return errors, making all AI features fail
2. Plugin execution failures — custom plugins (HTTP, code execution, database) fail silently and cause planner loops to stall
3. Vector store degradation — Chroma, Pinecone, Azure AI Search, or Qdrant unavailability breaks semantic search and memory retrieval
4. Planner timeout — autonomous planners can loop indefinitely on complex goals when the LLM produces unexpected output
5. Memory retrieval failures — embedding generation or vector search failures cause agents to lose conversational context
6. Token budget exhaustion — long agentic loops consume token budgets faster than expected, causing truncated responses
7. Latency accumulation — multi-step SK pipelines compound LLM latency across multiple chained calls

Without monitoring, all of these failures look the same to users: the AI feature stopped working.

Key Metrics to Monitor

| Metric | Why it matters | |--------|---------------| | LLM provider availability | Root-cause all AI feature failures | | Application health endpoint | Composite health across all SK components | | Vector store availability | Semantic search and memory retrieval depend on it | | Plugin health (per plugin) | Individual plugin failures degrade specific AI capabilities | | Planner step count | Anomalously high step counts indicate runaway planners | | End-to-end response time | Multi-step pipelines accumulate latency — track the full chain | | Embedding service health | Required for vector store operations |

Setup Guide

1. Add a Health Endpoint to Your Semantic Kernel Application

The most important monitor is a composite health endpoint that validates every component SK depends on. In .NET with ASP.NET Core:

// HealthChecks/SemanticKernelHealthCheck.cs
using Microsoft.Extensions.Diagnostics.HealthChecks;
using Microsoft.SemanticKernel;

public class SemanticKernelHealthCheck : IHealthCheck
{
    private readonly Kernel _kernel;
    private readonly ILogger<SemanticKernelHealthCheck> _logger;

    public SemanticKernelHealthCheck(Kernel kernel, ILogger<SemanticKernelHealthCheck> logger)
    {
        _kernel = kernel;
        _logger = logger;
    }

    public async Task<HealthCheckResult> CheckHealthAsync(
        HealthCheckContext context,
        CancellationToken cancellationToken = default)
    {
        try
        {
            // Minimal LLM probe — uses the cheapest/fastest model configured
            var result = await _kernel.InvokePromptAsync(
                "Reply with exactly: ok",
                new KernelArguments(),
                cancellationToken: cancellationToken
            );

            var response = result.GetValue<string>()?.Trim().ToLower() ?? "";
            if (!response.Contains("ok"))
            {
                return HealthCheckResult.Degraded(
                    $"LLM probe returned unexpected response: {response}"
                );
            }

            return HealthCheckResult.Healthy("Semantic Kernel LLM probe passed");
        }
        catch (Exception ex)
        {
            _logger.LogError(ex, "Semantic Kernel health check failed");
            return HealthCheckResult.Unhealthy(
                "Semantic Kernel LLM probe failed",
                ex
            );
        }
    }
}

Register the health check and map the endpoint:

// Program.cs
builder.Services.AddHealthChecks()
    .AddCheck<SemanticKernelHealthCheck>("semantic_kernel", 
        failureStatus: HealthStatus.Unhealthy,
        timeout: TimeSpan.FromSeconds(15))
    .AddUrlGroup(
        new Uri("https://yourvectorstore.com/health"),
        "vector_store",
        failureStatus: HealthStatus.Degraded
    );

app.MapHealthChecks("/health", new HealthCheckOptions
{
    ResponseWriter = UIResponseWriter.WriteHealthCheckUIResponse,
});

In Vigilmon:

1. Monitors → New Monitor → HTTP
2. URL: https://yourapp.com/health
3. Keyword check: "status":"Healthy"
4. Interval: 2 minutes
5. Response timeout: 20000ms (SK health checks include an LLM round-trip)
6. Save

2. Monitor Your LLM Provider Directly

Semantic Kernel's LLM provider availability is the root cause of most AI feature failures. Monitor it directly, independent of your application:

For Azure OpenAI:

1. Type: HTTP
2. Method: GET
3. URL: https://your-resource.openai.azure.com/openai/models?api-version=2024-02-01
4. Custom headers:
   • api-key: your-dedicated-monitoring-key
5. Keyword check: gpt-4
6. Interval: 5 minutes
7. Save

For OpenAI:

1. URL: https://api.openai.com/v1/models
2. Custom headers:
   • Authorization: Bearer sk-your-dedicated-monitoring-key
3. Keyword check: gpt-4
4. Interval: 5 minutes
5. Save

Security note: Create dedicated API keys for monitoring only. Never reuse production application keys.

3. Monitor the Vector Store

Semantic Kernel's memory and semantic search features depend on the vector store. Add a health monitor for each vector store you use:

For Chroma:

# health/chroma_probe.py
import httpx

def check_chroma_health(host: str = "http://localhost:8000") -> dict:
    try:
        response = httpx.get(f"{host}/api/v1/heartbeat", timeout=5)
        response.raise_for_status()
        return {"ok": True, "status": response.json()}
    except Exception as e:
        return {"ok": False, "error": str(e)}

For hosted vector stores, monitor their status pages or health endpoints directly in Vigilmon:

• Pinecone: https://status.pinecone.io — keyword check: All Systems Operational
• Qdrant Cloud: https://qdrant.to/cloud-status — keyword check: Operational
• Azure AI Search: Check the Azure status page for your region

4. Python SDK: Health Endpoint with Composite Checks

For Python Semantic Kernel applications:

# health/semantic_kernel_health.py
import asyncio
import time
import os
from semantic_kernel import Kernel
from semantic_kernel.connectors.ai.open_ai import OpenAIChatCompletion
from semantic_kernel.connectors.memory.chroma import ChromaMemoryStore

async def check_sk_health() -> dict:
    results = {}

    # LLM probe
    try:
        kernel = Kernel()
        kernel.add_service(
            OpenAIChatCompletion(
                ai_model_id="gpt-4o-mini",
                api_key=os.environ["OPENAI_MONITORING_API_KEY"],
            )
        )

        start = time.time()
        result = await kernel.invoke_prompt("Reply with exactly: ok")
        latency_ms = int((time.time() - start) * 1000)

        response = str(result).strip().lower()
        results["llm"] = {
            "ok": "ok" in response,
            "latency_ms": latency_ms,
        }
    except Exception as e:
        results["llm"] = {"ok": False, "error": str(e)}

    # Vector store probe
    try:
        store = ChromaMemoryStore(host=os.environ.get("CHROMA_HOST", "localhost"))
        collections = await store.get_collections_async()
        results["vector_store"] = {"ok": True, "collections": len(collections)}
    except Exception as e:
        results["vector_store"] = {"ok": False, "error": str(e)}

    all_ok = all(v.get("ok", False) for v in results.values())
    return {"ok": all_ok, "components": results}

Expose via FastAPI:

from fastapi import FastAPI
from fastapi.responses import JSONResponse
from health.semantic_kernel_health import check_sk_health

app = FastAPI()

@app.get("/health/sk")
async def sk_health():
    result = await check_sk_health()
    if not result["ok"]:
        return JSONResponse(content=result, status_code=503)
    return result

Monitor in Vigilmon:

1. URL: https://yourapp.com/health/sk
2. Keyword check: "ok":true
3. Interval: 5 minutes
4. Response time warning: 10000ms (SK health checks include LLM round-trips)
5. Response time critical: 20000ms
6. Save

5. Heartbeat Monitoring for Agentic Pipelines

Semantic Kernel's planner runs autonomous multi-step pipelines. If a background agent loop stops executing — due to a planner bug, token exhaustion, or unhandled exception — your application silently stops processing work.

Use heartbeat monitoring to detect silent pipeline failures:

# agents/document_processor.py
import asyncio
import httpx
import os
from semantic_kernel import Kernel
from semantic_kernel.planners.function_calling_stepwise_planner import (
    FunctionCallingStepwisePlanner,
)

async def run_document_processing_pipeline():
    kernel = build_kernel()  # your kernel setup
    planner = FunctionCallingStepwisePlanner(kernel)

    try:
        # Process pending documents
        docs = fetch_pending_documents()
        for doc in docs:
            result = await planner.invoke(
                kernel,
                f"Process and summarize the following document: {doc.content}"
            )
            save_result(doc.id, result)

        # Ping heartbeat on successful completion
        heartbeat_url = os.environ.get("VIGILMON_DOC_PIPELINE_HEARTBEAT")
        if heartbeat_url:
            await asyncio.to_thread(
                httpx.get, heartbeat_url, timeout=5
            )

    except Exception as e:
        print(f"[sk] Document pipeline failed: {e}")
        # No heartbeat ping → Vigilmon alerts on missed interval
        raise

# Run on a schedule (e.g., every hour via your task scheduler)

In Vigilmon:

1. New Monitor → Heartbeat
2. Expected interval: match your pipeline schedule
3. Grace period: 15 minutes
4. Copy the ping URL into VIGILMON_DOC_PIPELINE_HEARTBEAT

Alerting

Configure alert channels for your Semantic Kernel monitors:

1. Open monitor → Alerts → Add Channel
2. Recommended setup:
   • Slack #ai-ops: immediate notification on LLM provider failure or SK health check 503
   • Email: on-call engineer for sustained (2+ consecutive) failures
   • PagerDuty: if SK powers a customer-facing copilot with uptime SLA

Recommended thresholds:

| Monitor | Alert condition | Escalation | |---------|----------------|------------| | LLM provider (Azure OpenAI) | 1 failure | 2 consecutive → email | | SK composite health endpoint | 1 failure | Immediate → Slack | | Vector store health | 1 failure | 2 consecutive → email | | Planner pipeline heartbeat | 1 missed interval | Immediate → Slack + PagerDuty | | Response time warning | >10000ms | >20000ms → page |

Alert routing by component

• LLM provider down: all AI features are broken — page immediately, set status page to degraded
• Vector store down: memory and semantic search broken, but completion features still work — alert team, investigate
• Planner heartbeat missed: autonomous pipeline stopped — investigate logs for planner exceptions or token exhaustion
• SK health 503 but LLM provider healthy: application-level issue — check plugin registrations and memory configuration

Conclusion

Semantic Kernel's composable, multi-component architecture is its strength for building sophisticated AI applications — and also the source of its monitoring complexity. An LLM provider outage, a broken plugin, a degraded vector store, or a stalled planner loop all produce the same user-visible symptom: the AI feature stopped working.

Vigilmon gives you the external observability to distinguish between these failure modes quickly. The LLM provider monitor and the composite health endpoint cover the most common causes of AI feature failure. Heartbeat monitors for autonomous planner pipelines catch the silent failures that are hardest to detect from inside the application.

Start with the LLM provider monitor and the composite health endpoint (10 minutes to set up), then add heartbeat monitoring for any Semantic Kernel pipelines running on a schedule.

Further Reading

• Monitoring AutoGen with Vigilmon
• Monitoring CrewAI with Vigilmon
• Monitoring Azure App Service with Vigilmon