tutorial

Monitoring RKE2 Clusters with Vigilmon

RKE2 is Rancher's hardened, CNCF-conformant Kubernetes distribution. Here's how to monitor your RKE2 control plane, etcd, node health, and deployed workloads with Vigilmon.

RKE2 (Rancher Kubernetes Engine 2) is a CNCF-conformant Kubernetes distribution designed with security and compliance in mind. It ships with CIS Kubernetes Benchmark hardening out of the box, an embedded etcd datastore, and no external dependencies on Docker — everything runs via containerd under systemd. This makes RKE2 an excellent choice for production and regulated environments, but it also means you own the full observability stack. Vigilmon complements your existing cluster monitoring (Prometheus, Grafana) with external uptime checks: API server reachability, ingress health, node TCP probes, and heartbeats for RKE2 CronJobs — all from outside your cluster boundary.

What You'll Set Up

  • External uptime monitors for the Kubernetes API server
  • HTTP health checks for ingress-exposed workloads
  • TCP port monitors for control plane and etcd ports
  • Cron heartbeat monitors for RKE2 CronJobs
  • SSL certificate expiry alerts for cluster endpoints

Prerequisites

  • A running RKE2 cluster (single-node or multi-node)
  • At least one workload exposed via an ingress controller (nginx-ingress, Traefik, etc.)
  • kubectl access to the cluster
  • A free Vigilmon account

Step 1: Monitor the Kubernetes API Server

The API server is the single entry point to your cluster. If it is unreachable, kubectl stops working, controllers cannot reconcile, and any CI/CD pipelines that apply manifests will fail. RKE2 exposes the API server on port 6443 by default.

Add an external HTTP monitor for the API server health endpoint:

  1. Log in to vigilmon.online and click Add Monitor.
  2. Set Type to HTTP / HTTPS.
  3. Enter https://YOUR_SERVER_IP:6443/healthz.
  4. Set Expected HTTP status to 200.
  5. Set Check interval to 1 minute.
  6. Click Save.

The /healthz endpoint does not require authentication — it returns ok when the API server is healthy. If your API server IP is not publicly accessible, use your cluster's load balancer VIP or the first control plane node's public IP.

For high-availability RKE2 setups with multiple server nodes behind a load balancer, monitor the load balancer endpoint rather than individual nodes:

https://k8s-lb.yourdomain.com:6443/healthz

This catches both load balancer failures and scenarios where all server nodes are unreachable.

Step 2: TCP Port Monitoring for Control Plane Components

Beyond the API server, RKE2's control plane components communicate on several ports. TCP monitors on these ports give you early warning of component-level failures that the API server health endpoint may not surface immediately.

| Component | Port | Protocol | |---|---|---| | API server | 6443 | TCP/HTTPS | | etcd client | 2379 | TCP | | etcd peer | 2380 | TCP | | Kubelet API | 10250 | TCP | | Node metrics | 10255 | TCP |

Add TCP monitors for each port you want to track:

  1. Click Add Monitor → TCP Port.
  2. Enter your server IP and port (e.g. 192.168.1.10:2379).
  3. Set Check interval to 1 minute.
  4. Click Save.

For etcd, monitoring port 2379 (client traffic) is the most actionable — a failure here means the API server cannot persist state. The peer port 2380 is useful for diagnosing etcd cluster split-brain in multi-server deployments.

Step 3: Monitor Ingress-Exposed Workloads

RKE2 ships with an optional Nginx ingress controller that exposes your workloads externally. Each critical workload should have its own Vigilmon monitor.

First, ensure your workloads have health endpoints. For a deployment called myapp:

# deployment.yaml
livenessProbe:
  httpGet:
    path: /healthz
    port: 8080
  initialDelaySeconds: 10
  periodSeconds: 30
readinessProbe:
  httpGet:
    path: /readyz
    port: 8080
  initialDelaySeconds: 5
  periodSeconds: 10

Then add a Vigilmon monitor pointing to the ingress-exposed URL:

  1. Click Add Monitor → HTTP / HTTPS.
  2. Enter https://myapp.yourdomain.com/healthz.
  3. Set Expected HTTP status to 200.
  4. Set Check interval to 1 minute.
  5. Click Save.

This external probe is different from Kubernetes liveness/readiness probes — it checks the full request path including DNS, ingress, and TLS termination, not just container health. A failing Vigilmon check when Kubernetes probes pass indicates a networking issue outside the cluster (DNS misconfiguration, ingress misconfiguration, or upstream firewall change).

Step 4: SSL Certificate Alerts for Cluster Endpoints

RKE2 auto-generates TLS certificates for cluster components (API server, etcd, kubelet) using its embedded CA. These are internal certificates with a 1-year validity by default. External-facing endpoints served through your ingress controller typically use Let's Encrypt or your own PKI.

Add SSL monitors for every externally accessible HTTPS endpoint:

  1. Open any HTTP monitor you created in Step 3.
  2. Enable Monitor SSL certificate under the SSL section.
  3. Set Alert when certificate expires in less than 21 days.
  4. Click Save.

For the API server itself, also add an SSL monitor:

  1. Create a new HTTP monitor for https://k8s-lb.yourdomain.com:6443/healthz.
  2. Enable the SSL certificate check on the same monitor.
  3. Set the alert threshold to 30 days — longer lead time for an internal CA certificate that requires manual rotation.

RKE2's auto-generated certificates are renewed by the rke2 certificate rotate command; Vigilmon's SSL alert gives you the runway to schedule that maintenance window before expiry causes an outage.

Step 5: Heartbeat Monitoring for RKE2 CronJobs

Kubernetes CronJobs running on your RKE2 cluster are invisible to external monitors by default. If a CronJob fails silently, there is no HTTP endpoint to probe — the job simply stops running. Use Vigilmon's cron heartbeat to confirm scheduled jobs complete successfully.

  1. In Vigilmon, click Add Monitor → Cron Heartbeat.
  2. Set Expected ping interval to match your CronJob schedule in minutes (e.g. 60 for an hourly job).
  3. Copy the heartbeat URL.

Add the ping to your CronJob's container command:

# cronjob.yaml
apiVersion: batch/v1
kind: CronJob
metadata:
  name: backup-etcd
spec:
  schedule: "0 * * * *"
  jobTemplate:
    spec:
      template:
        spec:
          containers:
          - name: backup
            image: bitnami/etcd:latest
            command:
            - /bin/sh
            - -c
            - |
              etcdctl snapshot save /backup/snapshot.db \
                --endpoints=https://127.0.0.1:2379 \
                --cacert=/etc/kubernetes/ssl/kube-ca.pem \
                --cert=/etc/kubernetes/ssl/kube-node.pem \
                --key=/etc/kubernetes/ssl/kube-node-key.pem
              curl -s https://vigilmon.online/heartbeat/YOUR_ID

The curl call at the end only runs if the backup command exits successfully. If the backup fails, the job exits with a non-zero code, the shell stops executing, and the heartbeat ping never fires. Vigilmon alerts after the expected interval passes — giving you immediate notification of a missed etcd backup.

Step 6: Node Health with TCP Kubelet Probes

RKE2 agent nodes (worker nodes) expose the Kubelet API on port 10250. Adding a TCP probe per node gives you a quick signal when a node goes completely dark — before Kubernetes itself marks the node NotReady (which can take several minutes by default):

  1. Click Add Monitor → TCP Port for each node IP.
  2. Enter <NODE_IP>:10250.
  3. Set Check interval to 1 minute.
  4. Click Save.

For clusters with many nodes, focus TCP monitoring on control plane nodes and any stateful worker nodes hosting persistent volumes. Ephemeral worker nodes can be monitored at the workload level instead (if a workload is healthy, the nodes underneath are running).

Step 7: Configure Alert Channels

  1. Go to Alert Channels in Vigilmon and add Slack, PagerDuty, or email.
  2. Set Consecutive failures before alert to 2 for all monitors — RKE2 components can briefly flap during rolling upgrades.
  3. Set up Maintenance windows before running rke2-upgrade or certificate rotation. You can automate this with the Vigilmon API from your upgrade script:
# Pause monitoring before upgrade
curl -X POST https://vigilmon.online/api/maintenance \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -d '{"monitor_id": "YOUR_API_SERVER_MONITOR_ID", "duration_minutes": 30}'

# Run RKE2 upgrade
systemctl restart rke2-server

Summary

| Monitor | Target | What It Catches | |---|---|---| | HTTP health | https://k8s-lb:6443/healthz | API server down, LB failure | | TCP port | <node>:2379 | etcd client port unreachable | | HTTP workload | https://myapp.domain.com/healthz | Ingress misconfiguration, app crash | | SSL certificate | API server + ingress endpoints | Certificate expiry | | Cron heartbeat | CronJob backup script | Missed backup, job failure | | TCP kubelet | <node>:10250 | Worker node completely unreachable |

RKE2's hardening and embedded etcd make it an excellent production-grade Kubernetes distribution — but hardened infrastructure still needs external eyes. Vigilmon watches your cluster from outside: API server reachability, workload health, certificates, and scheduled jobs. Combined with in-cluster Prometheus metrics, you get complete observability coverage from the kernel to the application layer.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles