tutorial

How to Monitor k8sgpt with Vigilmon

k8sgpt is an open-source CLI and operator that uses large language models to analyze your Kubernetes cluster, diagnose resource failures, and surface human-r...

k8sgpt is an open-source CLI and operator that uses large language models to analyze your Kubernetes cluster, diagnose resource failures, and surface human-readable explanations for why pods are crashing, PVCs are stuck, or nodes are degraded. It's increasingly deployed as an in-cluster operator with a REST API — and when that API goes down, automated cluster diagnostics go silent exactly when you need them most.

This tutorial shows you how to use Vigilmon to monitor k8sgpt's operator API, its backend AI provider connectivity, and the health of the workloads it depends on — so you always know whether your AI-powered cluster diagnostics are actually running.

Why k8sgpt needs monitoring

k8sgpt the CLI is a one-shot tool, but the k8sgpt Operator runs continuously as a deployment inside your cluster. It polls for degraded Kubernetes resources, sends them to an AI backend (OpenAI, Anthropic, Ollama, Azure OpenAI, etc.), and surfaces Results as Kubernetes custom resources. Failures are subtle:

  • Operator pod crash — the controller silently stops producing Results; existing stale Results remain in the cluster giving a false sense of activity
  • AI backend unreachable — the operator is running but all analysis calls time out; no new Results are produced for any newly failing resources
  • k8sgpt API server down — if you've deployed the REST API server (k8sgpt serve), tools and dashboards that query it get no data
  • Result staleness — Results exist but haven't been updated in hours; the cluster may have new failures that haven't been analyzed
  • RBAC misconfiguration after an upgrade — the operator loses list/watch permissions on core resources; analysis stops with no visible error in logs unless you're watching

External monitoring catches the difference between "k8sgpt is running" and "k8sgpt is actually analyzing."

What you'll need

  • A Kubernetes cluster with the k8sgpt Operator installed (via Helm or the k8sgpt-operator manifests)
  • Optionally: k8sgpt serve running as a deployment with an exposed REST API
  • A free Vigilmon account

Step 1: Understand k8sgpt's monitoring surface

k8sgpt has two deployment models:

1. CLI only — run manually or via a CronJob; no persistent API surface to monitor externally. Use Vigilmon heartbeat monitors for this pattern (see Step 5).

2. k8sgpt Operator — runs continuously as a deployment. The operator itself exposes a /healthz probe. If you deploy the REST API server, it exposes an HTTP API you can monitor directly.

Check what you have running:

# Check for the operator
kubectl get deployment -n k8sgpt-operator-system

# Check for the REST API server
kubectl get deployment -A | grep k8sgpt

# Check for the k8sgpt CRD and any results
kubectl get k8sgpt -A
kubectl get results -A

Step 2: Expose the operator health endpoint

The k8sgpt Operator container exposes a health check on port 8081. Create a NodePort service to expose it for external monitoring:

# k8sgpt-health-svc.yaml
apiVersion: v1
kind: Service
metadata:
  name: k8sgpt-operator-health
  namespace: k8sgpt-operator-system
spec:
  type: NodePort
  selector:
    control-plane: controller-manager
  ports:
    - name: healthz
      port: 8081
      targetPort: 8081
      nodePort: 30810
      protocol: TCP

kubectl apply -f k8sgpt-health-svc.yaml

# Verify
curl http://<node-ip>:30810/healthz
# Returns: ok

Step 3: Expose the k8sgpt REST API server (if deployed)

If you deploy k8sgpt serve as a Kubernetes workload, it listens on port 8080 by default. Expose it:

# k8sgpt-api-svc.yaml
apiVersion: v1
kind: Service
metadata:
  name: k8sgpt-api
  namespace: k8sgpt-operator-system
spec:
  type: NodePort
  selector:
    app: k8sgpt-api
  ports:
    - name: api
      port: 8080
      targetPort: 8080
      nodePort: 30808
      protocol: TCP

kubectl apply -f k8sgpt-api-svc.yaml

# Verify
curl http://<node-ip>:30808/healthz
# Returns: ok

# List recent analysis results
curl http://<node-ip>:30808/analyze

If you expose the API via an Ingress with a domain name, that's the URL to monitor.

Step 4: Monitor the operator and API with Vigilmon

Operator health

  1. Log in to vigilmon.online and go to Monitors → New Monitor
  2. Choose HTTP / HTTPS
  3. URL: http://<node-ip>:30810/healthz
  4. Check interval: 1 minute
  5. Expected status code: 200
  6. Response body contains: ok
  7. Save the monitor

REST API server (if deployed)

  1. Create a new HTTP / HTTPS monitor
  2. URL: http://<node-ip>:30808/healthz (or your ingress URL)
  3. Check interval: 1 minute
  4. Expected status code: 200
  5. Response body contains: ok
  6. Save

TCP port monitors (faster detection)

Add TCP monitors for both ports to catch networking failures independent of HTTP health logic:

  1. Go to Monitors → New Monitor → TCP Port
  2. Host: <node-ip>, Port: 30810
  3. Save, then repeat for port 30808

Step 5: Monitor the AI backend connectivity

k8sgpt's analysis depends on an external AI provider. If OpenAI, Anthropic, or your Ollama instance is unreachable, k8sgpt runs but produces no results. Monitor these endpoints directly.

OpenAI API

If you use OpenAI as your backend, monitor its API status endpoint:

  1. New HTTP / HTTPS monitor
  2. URL: https://api.openai.com (returns 200 if reachable)
  3. Check interval: 5 minutes
  4. Expected status: 200

Self-hosted Ollama

If you self-host Ollama as the AI backend for k8sgpt:

  1. New HTTP / HTTPS monitor
  2. URL: http://ollama.internal:11434 (or your Ollama service URL)
  3. Check interval: 1 minute
  4. Expected status: 200
  5. Response body contains: Ollama is running

When Ollama goes down, k8sgpt produces no new analysis results — but there's no clear signal in the cluster. This monitor fills that gap.

Step 6: Heartbeat monitoring for CronJob deployments

If you run k8sgpt as a CronJob rather than a persistent operator, use Vigilmon's heartbeat monitor to detect when it stops running.

In your k8sgpt CronJob, add a curl call to the Vigilmon heartbeat URL at the end of each successful run:

# k8sgpt-cronjob.yaml
apiVersion: batch/v1
kind: CronJob
metadata:
  name: k8sgpt-analyze
  namespace: k8sgpt-operator-system
spec:
  schedule: "*/15 * * * *"
  jobTemplate:
    spec:
      template:
        spec:
          containers:
            - name: k8sgpt
              image: ghcr.io/k8sgpt-ai/k8sgpt:latest
              command:
                - /bin/sh
                - -c
                - |
                  k8sgpt analyze --explain --output json && \
                  curl -fsS "https://heartbeat.vigilmon.online/ping/YOUR-HEARTBEAT-ID"
          restartPolicy: OnFailure

Replace YOUR-HEARTBEAT-ID with the ID from your Vigilmon heartbeat monitor. If k8sgpt fails or the CronJob is suspended, the heartbeat URL stops receiving pings and Vigilmon alerts you.

To create the heartbeat monitor in Vigilmon:

  1. Go to Monitors → New Monitor → Heartbeat
  2. Set expected interval: 15 minutes (matching your CronJob schedule)
  3. Set grace period: 5 minutes
  4. Save and copy the ping URL into your CronJob

Step 7: Configure alert channels

k8sgpt failures are operational blind spots — you stop getting diagnostic insights for cluster failures without knowing it. Treat k8sgpt alerts as high-priority.

Email alerts

  1. Go to Alert Channels → Add Channel → Email
  2. Enter your platform or SRE team email
  3. Assign to all k8sgpt monitors

Webhook (Slack or PagerDuty)

  1. Go to Alert Channels → Add Channel → Webhook
  2. Enter your Slack webhook URL
  3. Assign to all k8sgpt monitors

When you receive a k8sgpt alert, run:

# Check operator pod status
kubectl get pods -n k8sgpt-operator-system

# Check operator logs for errors
kubectl logs -n k8sgpt-operator-system \
  -l control-plane=controller-manager --tail=50

# Check if recent Results are being created
kubectl get results -A --sort-by='.metadata.creationTimestamp' | tail -10

# Check the k8sgpt custom resource configuration
kubectl describe k8sgpt -n k8sgpt-operator-system

# Verify RBAC permissions are intact
kubectl auth can-i list pods \
  --as=system:serviceaccount:k8sgpt-operator-system:k8sgpt-operator-controller-manager

Step 8: Create a status page

If your team uses k8sgpt as part of incident response or cluster health dashboards, a status page lets everyone see whether the diagnostics system itself is healthy:

  1. Go to Status Pages → New Status Page
  2. Name: "AI Cluster Diagnostics (k8sgpt)"
  3. Add monitors:
    • k8sgpt Operator health
    • k8sgpt REST API
    • Ollama (if self-hosted)
  4. Publish and share with your platform team

Monitor summary

| Monitor | Type | What it catches | |---------|------|-----------------| | http://<node-ip>:30810/healthz | HTTP | Operator pod crash, restart loop | | http://<node-ip>:30808/healthz | HTTP | REST API server down | | <node-ip>:30810 | TCP | Operator port unreachable | | <node-ip>:30808 | TCP | API server port unreachable | | https://api.openai.com | HTTP | OpenAI backend unreachable | | http://ollama.internal:11434 | HTTP | Self-hosted Ollama down | | k8sgpt CronJob | Heartbeat | CronJob stopped or suspended |

What's next

  • Result freshness alerting — combine a Vigilmon heartbeat with a script that checks kubectl get results -A and pings the heartbeat URL only when fresh results exist; if the cluster has degraded resources but no new Results appear, the heartbeat stops and you get alerted
  • Multi-backend monitoring — if you switch k8sgpt backends (Ollama → OpenAI) for failover, add monitors for both so you know when the fallback is also degraded
  • Status page integration — embed the Vigilmon status page URL in your k8sgpt dashboards so your team always knows whether the AI diagnostics pipeline is operational

Get started free at vigilmon.online — no credit card required, monitors start running in under a minute.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles