tutorial

Monitoring K3s with Vigilmon: Lightweight Kubernetes API Health, Agent Node Connectivity, Embedded SQLite/etcd & TLS Certificate Alerts

How to monitor K3s with Vigilmon — Kubernetes API server health, K3s agent node connectivity, embedded database availability, workload HTTP monitoring, and TLS certificate expiry alerting for lightweight Kubernetes clusters.

K3s is a lightweight, CNCF-certified Kubernetes distribution from Rancher (now part of SUSE) that packages everything needed to run a full Kubernetes cluster into a single binary under 100 MB. K3s replaces etcd with SQLite by default (or embedded etcd for HA clusters), removes cloud-provider integrations, and bundles Flannel, CoreDNS, Traefik, and local-path-provisioner to create a minimal but production-capable Kubernetes environment. K3s is purpose-built for edge computing, IoT deployments, CI environments, and resource-constrained systems where running a full kubeadm cluster is impractical. When the K3s server process crashes on the control plane node, all API operations stop and existing workloads continue running but cannot be managed — new pods cannot be scheduled and existing ones cannot be drained. When an embedded SQLite database becomes corrupted, the K3s API server cannot start and the entire control plane is unavailable. Vigilmon gives you external visibility into K3s: API server reachability, application HTTP health through the Traefik ingress, TLS certificate monitoring, and TCP probes on the Kubernetes API port that catch control plane failures within seconds.

What You'll Build

  • HTTP monitors on the K3s Kubernetes API server to detect control plane failures
  • HTTP monitors on applications served through K3s's built-in Traefik ingress
  • TCP monitors on the Kubernetes API port (6443) for fast failure detection
  • SSL certificate monitors for Traefik ingress HTTPS endpoints
  • A failure mode table covering K3s-specific failure scenarios including SQLite corruption and single-node control plane loss
  • An alerting runbook mapping Vigilmon alerts to K3s diagnostic commands

Prerequisites

  • A K3s cluster (single-server or HA multi-server configuration)
  • At least one application deployed with an Ingress resource (using Traefik)
  • External access to your K3s server (or the LoadBalancer service, if applicable)
  • A free account at vigilmon.online

Step 1: Understand K3s's Health Surface

K3s exposes health information through its API server's /healthz and /readyz endpoints, and through the k3s CLI. Check the cluster state before setting up external monitors:

# Check K3s server status (on control plane node)
sudo systemctl status k3s

# Verify API server health
curl -k https://localhost:6443/healthz
# Returns: ok

# Check node status
kubectl get nodes -o wide

# Inspect K3s embedded SQLite database health (single-server)
sudo ls -la /var/lib/rancher/k3s/server/db/
# For HA embedded etcd:
kubectl get pods -n kube-system -l component=etcd

The key external health signals for K3s are:

  1. The Kubernetes API server /healthz endpoint (indicates control plane health)
  2. Application HTTP responses through the Traefik ingress (end-to-end workload health)
  3. TCP connectivity to port 6443 (fast detection of K3s server process failure)
  4. TLS certificates for Traefik ingress hostnames

Step 2: Monitor the K3s API Server Health Endpoint

The K3s Kubernetes API server's /healthz endpoint is the single most important monitor for a K3s cluster. When this endpoint fails, no kubectl commands work, no new pods can be scheduled, and no existing workloads can be managed — even if the workloads themselves are still running:

# Test the API server health endpoint
curl -k https://your-k3s-server:6443/healthz
# Returns: ok

# If you have a kubeconfig, verify connectivity
kubectl cluster-info
  1. Log in to Vigilmon → Add Monitor → HTTP.
  2. URL: https://your-k3s-server.example.com:6443/healthz (your K3s server's external IP or hostname).
  3. Check interval: 60 seconds.
  4. Response timeout: 10 seconds.
  5. Expected status: 200.
  6. Keyword: ok.
  7. Label: K3s API server /healthz.
  8. Click Save.

TLS note: The K3s API server uses a self-signed certificate by default. If Vigilmon's HTTP monitor fails due to certificate validation, configure your K3s server with a proper TLS certificate by setting --tls-san your-k3s-server.example.com in the K3s server configuration, then issue a certificate from Let's Encrypt or your CA for that hostname.

Step 3: Monitor K3s API TCP Connectivity

A TCP monitor on port 6443 gives you the fastest possible detection of K3s server process failure — it doesn't wait for an HTTP response body, just tests whether the port is accepting connections:

  1. Add Monitor → TCP.
  2. Host: your-k3s-server.example.com.
  3. Port: 6443.
  4. Check interval: 60 seconds.
  5. Label: K3s API server TCP (6443).
  6. Click Save.

In HA multi-server configurations, add a TCP monitor for each server node. A TCP monitor that fires on one server while others pass indicates a per-node K3s process failure rather than a cluster-wide outage.

Step 4: Monitor Applications Through the Traefik Ingress

K3s ships with Traefik as its built-in ingress controller. Applications exposed through Traefik IngressRoutes or standard Kubernetes Ingress resources are the end-user facing surface — and the layer most likely to catch configuration errors, certificate issues, or workload crashes that aren't visible through the API server health endpoint:

# Check Traefik service and external IP
kubectl get svc -n kube-system traefik

# Verify your application is reachable through Traefik
curl https://myapp.example.com/health
  1. Add Monitor → HTTP.
  2. URL: https://myapp.example.com/health (your application's health endpoint).
  3. Check interval: 60 seconds.
  4. Response timeout: 10 seconds.
  5. Expected status: 200.
  6. Keyword: your health response field (e.g., ok, status).
  7. Label: myapp.example.com (K3s Traefik).
  8. Click Save.

Add one HTTP monitor per application or per IngressRoute. Traefik-level failures — misconfigured middleware, broken certificates, or routing rule conflicts — are only visible through these application-level monitors, not through the API server health check.

Step 5: Monitor Traefik Dashboard and TCP Port 80/443

The Traefik dashboard shows active routers, services, and middleware. Monitoring Traefik's HTTP port directly (separate from your application) helps isolate Traefik process failures from application failures:

# Check Traefik dashboard (if enabled)
kubectl port-forward -n kube-system svc/traefik 8080:8080 &
curl http://localhost:8080/api/overview
  1. Add Monitor → TCP.
  2. Host: your-k3s-server.example.com.
  3. Port: 443.
  4. Check interval: 60 seconds.
  5. Label: Traefik HTTPS TCP (443).
  6. Click Save.

Repeat for port 80. A TCP monitor failure on port 443 while the K3s API TCP monitor (port 6443) is passing isolates the failure to Traefik rather than the K3s control plane.

Step 6: Monitor TLS Certificates for Traefik Ingress Hostnames

Traefik in K3s can automatically provision Let's Encrypt certificates for Ingress hostnames using its built-in ACME client. When ACME renewal fails — due to DNS misconfiguration, rate limiting, or network issues on the K3s node — the certificate expires and all HTTPS traffic to that hostname fails:

For each HTTPS hostname in your Traefik Ingress:

  1. Add Monitor → SSL Certificate.
  2. Domain: myapp.example.com.
  3. Alert when expiry is within: 30 days.
  4. Alert again: 14 days, 7 days, 3 days, 1 day.
  5. Click Save.

K3s + Traefik ACME storage: Traefik stores Let's Encrypt certificates in /var/lib/rancher/k3s/server/tls/ by default. In HA multi-server deployments, certificate state must be shared — typically via a shared file system or by disabling ACME on follower nodes. Unexpected certificate expiry in HA clusters is often caused by ACME certificate state not being synchronized across server nodes.

Step 7: Configure Alerting

In Vigilmon under Settings → Notifications, configure alert channels and response runbooks:

| Monitor | Trigger | Immediate action | |---|---|---| | K3s API server /healthz | Non-200 | SSH to control plane node; check sudo systemctl status k3s; review logs: sudo journalctl -u k3s -n 100 | | K3s API TCP (6443) | Connection refused | K3s server process crashed; restart: sudo systemctl restart k3s; check disk space and SQLite health | | Application HTTP (Traefik) | Non-200 | Check pod health: kubectl get pods -n default; review Traefik logs: kubectl logs -n kube-system -l app=traefik | | Traefik HTTPS TCP (443) | Connection refused | Traefik process crash or misconfiguration; check Traefik pod: kubectl get pods -n kube-system | | SSL certificate | < 30 days | Check ACME logs in Traefik; verify DNS resolution for the hostname; check Let's Encrypt rate limits |

Alert grouping: Create a k3s monitor group. When the K3s server crashes, both the API /healthz HTTP monitor and the API TCP monitor fire simultaneously — the group view immediately identifies this as a control plane failure rather than an application issue.

Common K3s Failure Modes and What Vigilmon Catches

| Scenario | Vigilmon signal | |---|---| | K3s server process crash (single-server) | API /healthz HTTP monitor and TCP 6443 monitor both fire immediately | | SQLite database corruption | K3s fails to start; API /healthz and TCP 6443 monitors fire; workloads continue running | | Embedded etcd quorum loss (HA) | API /healthz monitor fires; some server node TCP monitors pass (single-node majority lost) | | Traefik pod crash | Application HTTP monitors fire; K3s API monitors continue passing | | Let's Encrypt certificate renewal failure | SSL monitor fires at 30-day threshold; HTTP monitor fires with TLS error on expiry | | K3s node (agent) unreachable | Pods scheduled on that agent fail; application HTTP monitors for those pods fire | | Disk full on K3s server node | SQLite writes fail; new pod scheduling stops; API /healthz may still pass briefly | | Network policy blocks external ingress | HTTP and TCP monitors for Traefik fire; API monitor continues passing | | K3s upgrade breaks kube-proxy | Application HTTP monitors show intermittent failures; API monitor passes | | CoreDNS pod crash | DNS-based HTTP monitors timeout; IP-based monitors continue passing |

K3s makes Kubernetes accessible on edge devices, IoT gateways, CI environments, and small-scale production clusters — but its single-binary simplicity can mask failures that would be more visible in a multi-component kubeadm cluster. SQLite corruption takes down the entire control plane with a single file failure. A Traefik ACME renewal failure silently expires the certificate for every hosted application. Vigilmon gives you external visibility across all of these: API server health checks that fire within 60 seconds of a K3s process crash, application HTTP monitors through Traefik that catch routing failures the API server can't see, TLS certificate expiry alerts before your users see browser warnings, and TCP probes that distinguish control plane failures from data plane failures. When your K3s cluster is running on a remote edge device with no local team, Vigilmon is your eyes on the ground.

Start monitoring your K3s cluster in under 5 minutes — register free at vigilmon.online.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles