tutorial

Monitoring Talos Linux with Vigilmon: Immutable Kubernetes OS, API Server & Machine Health

How to monitor Talos Linux with Vigilmon — machine API health, Kubernetes control plane availability, node readiness, and alerting for immutable Kubernetes OS outages.

Talos Linux is an immutable, API-driven operating system built exclusively for running Kubernetes, developed by Sidero Labs. Unlike conventional Linux distributions, Talos has no shell, no SSH access, and no package manager — every interaction happens through the Talos API (talosctl) or the Kubernetes API. The OS is built to be read-only, secure by default, and always consistent between machines. Teams running Talos replace manual SSH troubleshooting with API-driven observability, which means external monitoring of the Talos machine API and Kubernetes control plane is not just useful — it is the primary operational interface when things go wrong. Vigilmon gives you external visibility into the Talos machine API, the Kubernetes API server, cluster nodes, and the workloads running inside, so you catch degradation before it becomes a cluster-wide incident.

What You'll Build

  • An HTTP monitor for the Talos machine API health
  • Kubernetes API server health monitors for the control plane running on Talos
  • Node readiness monitors for Talos worker nodes
  • Application workload monitors for services running inside the cluster
  • SSL certificate monitoring for the Talos API and Kubernetes API
  • An alerting setup that distinguishes OS-layer failures from Kubernetes-layer failures

Prerequisites

  • A running Talos Linux cluster (single-node or multi-node)
  • talosctl configured with a talosconfig pointing to your cluster
  • kubectl configured with a kubeconfig for the cluster
  • A free account at vigilmon.online

Step 1: Understand Talos API Endpoints

Talos exposes two key API layers:

  1. Talos machine API — port 50000 (gRPC), used by talosctl for OS-level operations (upgrades, configuration, reboots)
  2. Kubernetes API server — port 6443 (HTTPS), the standard kubectl endpoint

Both must be reachable for full cluster operability. The Kubernetes API handles workloads; the Talos machine API handles the OS hosting those workloads. Monitor both independently:

# Check Talos machine API reachability
talosctl --nodes your-talos-host health

# Check Kubernetes API server
kubectl get --raw='/healthz'

Find your Talos node IPs:

talosctl --nodes your-talos-host get members

Step 2: Monitor the Kubernetes API Server Health Endpoint

On Talos, the Kubernetes API server runs as a static pod controlled by the Talos OS. Monitor it at the standard Kubernetes health endpoint:

  1. Log in to Vigilmon → Add Monitor → HTTP.
  2. URL: https://your-talos-control-plane-ip:6443/healthz.
  3. Check interval: 60 seconds.
  4. Response timeout: 15 seconds.
  5. Expected status: 200.
  6. Keyword: ok.
  7. Label: Talos Kubernetes API (/healthz).
  8. Click Save.

This monitor catches the API server pod crashing on the Talos node, which can happen after an OS upgrade if the configuration is incompatible with the Kubernetes version.

Step 3: Monitor the Kubernetes API Server Readiness

The /readyz endpoint verifies that the API server has completed initialization and all controllers are ready to serve traffic. This is the definitive check for a fully operational control plane:

  1. Add Monitor → HTTP.
  2. URL: https://your-talos-control-plane-ip:6443/readyz.
  3. Check interval: 60 seconds.
  4. Response timeout: 15 seconds.
  5. Expected status: 200.
  6. Keyword: ok.
  7. Label: Talos Kubernetes API (/readyz).
  8. Click Save.

On Talos, control plane static pods (API server, controller manager, scheduler) are defined in the machine configuration and restarted by the Talos OS manager if they crash. A degraded /readyz means the OS restarted the pod but it hasn't completed initialization — check talosctl logs -n your-node kube-apiserver.

Step 4: Monitor etcd Cluster Health

Talos runs etcd as a managed service, distributed across control plane nodes. etcd failure immediately renders the Kubernetes API server read-only and prevents new writes (pod scheduling, ConfigMap updates, etc.):

# Check etcd health via the Kubernetes API
kubectl get --raw='/healthz/etcd'

# Check via talosctl
talosctl --nodes your-control-plane-node service etcd

For external monitoring:

  1. Add Monitor → HTTP.
  2. URL: https://your-talos-control-plane-ip:6443/healthz/etcd.
  3. Check interval: 2 minutes.
  4. Response timeout: 15 seconds.
  5. Expected status: 200.
  6. Keyword: ok.
  7. Label: Talos etcd health.
  8. Click Save.

Talos bootstraps etcd automatically on the first control plane node during talosctl bootstrap. If etcd loses quorum (more than half of control plane nodes lost), the cluster is in a recovery state — external monitoring from Vigilmon catches this immediately from the API server's degraded /readyz response.

Step 5: Monitor Talos Node Readiness via Kubernetes

On Talos, worker nodes join the cluster by loading a machine configuration generated by talosctl gen config. Once joined, their readiness is reflected in Kubernetes node conditions. Monitor workloads on specific nodes by targeting NodePort services:

# List node status
kubectl get nodes

# Get NodePort services
kubectl get services --all-namespaces | grep NodePort

For a workload exposed on a Talos worker node:

  1. Add Monitor → HTTP.
  2. URL: http://your-talos-worker-ip:30080.
  3. Check interval: 60 seconds.
  4. Response timeout: 10 seconds.
  5. Expected status: 200.
  6. Label: App on Talos worker (NodePort 30080).
  7. Click Save.

Step 6: Monitor Applications via Ingress

Most Talos clusters use an ingress controller (Nginx, Traefik, or Cilium's Gateway API) to expose applications. Monitor ingress endpoints directly to catch both application failures and ingress controller failures:

# Find ingress endpoints
kubectl get ingress --all-namespaces

# Check ingress controller service
kubectl get services -n ingress-nginx

For an application exposed through ingress:

  1. Add Monitor → HTTP.
  2. URL: https://app.your-domain.com.
  3. Check interval: 60 seconds.
  4. Response timeout: 10 seconds.
  5. Expected status: 200.
  6. Keyword: (your app's expected keyword, e.g. dashboard).
  7. Label: App via Talos ingress (app.your-domain.com).
  8. Click Save.

Step 7: Monitor SSL Certificates

Talos generates TLS certificates for both the machine API (port 50000) and the Kubernetes API server (port 6443). Both expire and must be rotated, either automatically (Talos manages Kubernetes certs via kubeadm-style logic) or manually for the Talos OS certificates:

# Check Kubernetes API server cert expiry
openssl s_client -connect your-talos-control-plane-ip:6443 < /dev/null 2>/dev/null | openssl x509 -noout -dates

# Check Talos machine API cert (gRPC/TLS on port 50000)
openssl s_client -connect your-talos-node-ip:50000 < /dev/null 2>/dev/null | openssl x509 -noout -dates

For the Kubernetes API server:

  1. Add Monitor → SSL Certificate.
  2. Domain: your-talos-control-plane-ip (port 6443).
  3. Alert when expiry is within: 30 days.
  4. Alert again: 14 days, 7 days, 3 days, 1 day.
  5. Click Save.

For any custom domains fronting the cluster (e.g., a load balancer):

  1. Add Monitor → SSL Certificate.
  2. Domain: k8s.your-domain.com.
  3. Alert when expiry is within: 30 days.
  4. Click Save.

Step 8: Configure Alerting

In Vigilmon under Settings → Notifications, configure your alert channels:

| Monitor | Trigger | Action | |---|---|---| | Kubernetes API /healthz | Non-200 or keyword missing | API server pod crashed; check talosctl logs kube-apiserver | | Kubernetes API /readyz | Non-200 or keyword missing | API server degraded; check etcd connectivity | | Talos etcd health | Non-200 or keyword missing | etcd cluster unhealthy; check disk I/O and quorum | | App workload (NodePort) | Non-200 or timeout | Pod failure; check kubectl get pods and logs | | Ingress endpoint | Non-200 or timeout | Ingress controller or upstream pod failure | | SSL certificate (API server) | < 30 days to expiry | kubectl and talosctl operations will fail on expiry |

Alert routing: Route Kubernetes API and etcd alerts to the platform team on-call immediately — these affect all workloads. Application monitors route to the application team. Talos's immutable design means OS-level issues (kernel panics, hardware failures) show up indirectly as API server unreachability, not SSH alerts.

Common Talos Linux Failure Modes and What Vigilmon Catches

| Scenario | Vigilmon monitor | |---|---| | Talos OS upgrade breaks API server static pod config | Kubernetes API /healthz fires during upgrade window | | etcd loses quorum after two control plane nodes lost | etcd health check fires; API server enters read-only mode | | Worker node reboots for Talos OS upgrade | NodePort workload monitor fires briefly during reboot | | Ingress controller misconfigured after upgrade | Ingress endpoint monitor fires; all exposed apps return 404 | | Control plane node hardware failure | All API server monitors fire; cluster is unresponsive | | Kubernetes API server TLS cert expires | SSL monitor alerts 30 days before kubectl commands fail | | Load balancer misconfigured after change | External URL monitor fires; internal access still works | | DNS failure for cluster hostname | External monitor fires; internal kubelet DNS still resolves |

Talos Linux trades the flexibility of a general-purpose OS for operational consistency, security, and repeatability — but that trade makes external monitoring even more important. Without SSH, a Vigilmon external check is often the first signal that a Talos node or API server has degraded. Combine Kubernetes API health checks with workload monitors, SSL expiry alerts, and etcd health to build complete observability for a Talos Linux cluster without ever needing shell access.

Start monitoring your Talos Linux cluster in under 5 minutes — register free at vigilmon.online.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles