tutorial

How to Monitor Virtual Kubelet with Vigilmon (HTTP + TCP + Alerts)

Virtual Kubelet is a Kubernetes node implementation that masquerades as a regular Kubernetes node, routing pod scheduling to external APIs such as Azure Cont...

Virtual Kubelet is a Kubernetes node implementation that masquerades as a regular Kubernetes node, routing pod scheduling to external APIs such as Azure Container Instances, AWS Fargate, HashiCorp Nomad, and custom providers. Your Kubernetes control plane sees Virtual Kubelet as a standard node, but the actual workloads run outside your cluster — in a serverless backend, a different cloud, or a custom environment. When the Virtual Kubelet process fails or its upstream provider becomes unavailable, pods that appeared to schedule successfully stop receiving traffic and Kubernetes shows no error.

In this tutorial you'll set up comprehensive uptime monitoring for your Virtual Kubelet deployment using Vigilmon — free tier, no credit card required.

Why Virtual Kubelet deployments need external monitoring

Virtual Kubelet introduces a unique class of failure that the Kubernetes control plane cannot surface on its own:

  • Virtual Kubelet process crashes — the node registration disappears; pods previously assigned to the virtual node enter Unknown phase; Kubernetes marks them for eviction after the node timeout, but no alert fires for the time window in between
  • Upstream provider quota exhausted — your ACI subscription or Fargate account hits its capacity limit; new pods are rejected by the provider while Kubernetes believes the node can still accept workloads
  • Provider API unavailable — the external API that backs the virtual node is down; running pods may continue executing but lifecycle operations (create, delete, update) all fail silently
  • Authentication token expiry — Virtual Kubelet's credentials to the upstream provider expire; the process stays running and the node appears Ready, but every pod operation fails with auth errors that Kubernetes doesn't propagate to pod events
  • Network isolation between Virtual Kubelet and provider — the proxy or VPN link between your cluster and the provider drops; pods are stranded in a partially-created state with no path to resolution

These failures are especially damaging because the Kubernetes node condition Ready reflects the Virtual Kubelet process liveness, not whether the backing provider is functional.

What you'll need

  • A Kubernetes cluster with Virtual Kubelet deployed
  • At least one workload running on the virtual node exposed via Service or Ingress
  • A free Vigilmon account (sign up takes 30 seconds)

Step 1: Expose a health endpoint for monitoring

Virtual Kubelet exposes its own health and metrics endpoints. First, confirm the virtual node is registered and its pods are running:

# Verify Virtual Kubelet node is registered
kubectl get nodes
# NAME                STATUS   ROLES    AGE   VERSION
# virtual-node-aci    Ready    agent    5d    v1.19.10-vk-azure-aci-v1.4.7
# node-1              Ready    worker   14d   v1.23.5

# Check Virtual Kubelet pod
kubectl get pods -n virtual-kubelet
# NAME                               READY   STATUS    RESTARTS   AGE
# virtual-kubelet-7f9c8b6d5-xp2mn    1/1     Running   0          5d

# Check Virtual Kubelet health endpoint
kubectl port-forward -n virtual-kubelet deployment/virtual-kubelet 10255:10255
curl http://localhost:10255/healthz
# ok

Deploy a test workload on the virtual node to validate the full provider path:

# vk-health.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: vk-health
  namespace: default
spec:
  replicas: 2
  selector:
    matchLabels:
      app: vk-health
  template:
    metadata:
      labels:
        app: vk-health
    spec:
      nodeSelector:
        kubernetes.io/role: agent
        beta.kubernetes.io/os: linux
        type: virtual-kubelet
      tolerations:
        - key: virtual-kubelet.io/provider
          operator: Exists
          effect: NoSchedule
      containers:
        - name: health
          image: nginx:alpine
          ports:
            - containerPort: 80
          volumeMounts:
            - name: config
              mountPath: /etc/nginx/conf.d
          readinessProbe:
            httpGet:
              path: /health
              port: 80
            initialDelaySeconds: 10
            periodSeconds: 15
      volumes:
        - name: config
          configMap:
            name: vk-health-nginx
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: vk-health-nginx
  namespace: default
data:
  default.conf: |
    server {
      listen 80;
      location /health {
        return 200 '{"status":"ok","provider":"virtual-kubelet"}';
        add_header Content-Type application/json;
      }
    }
---
apiVersion: v1
kind: Service
metadata:
  name: vk-health
  namespace: default
spec:
  type: LoadBalancer
  selector:
    app: vk-health
  ports:
    - port: 80
      targetPort: 80
      protocol: TCP

kubectl apply -f vk-health.yaml

# Get the LoadBalancer IP
kubectl get svc vk-health
# NAME        TYPE           CLUSTER-IP      EXTERNAL-IP      PORT(S)   AGE
# vk-health   LoadBalancer   10.96.55.200    203.0.113.90    80/TCP    2m

# Verify pods landed on the virtual node
kubectl get pods -l app=vk-health -o wide
# NAME                         READY   STATUS    NODE
# vk-health-5c8d7f9b4-abcde    1/1     Running   virtual-node-aci

# Confirm the health endpoint responds
curl http://203.0.113.90/health
# {"status":"ok","provider":"virtual-kubelet"}

Step 2: Set up HTTP monitoring in Vigilmon

With your health endpoint exposed, add it to Vigilmon:

  1. Log in to vigilmon.online and go to Monitors → New Monitor
  2. Choose HTTP / HTTPS as the type
  3. Set the URL to your service endpoint: http://203.0.113.90/health
  4. Set the check interval to 1 minute
  5. Under Expected response, set:
    • Status code: 200
    • Response body contains: "status":"ok"
  6. Save the monitor

This monitor validates the entire Virtual Kubelet data path: the VK process must be running, the upstream provider API must be reachable, provider authentication must be valid, the pod must be running in the provider, and routing back to the LoadBalancer must work — all for a 200 to reach Vigilmon.

Why external monitoring catches what Kubernetes misses

When a Virtual Kubelet node goes NotReady, Kubernetes waits up to 5 minutes (node-monitor-grace-period) before evicting pods. During this window, your workloads are silently failing to serve traffic. Vigilmon detects the outage in under a minute, before Kubernetes has even begun remediation. Its multi-region consensus model eliminates false positives from single-region transient failures.

Step 3: Monitor TCP ports and the Virtual Kubelet health endpoint

Add monitors to detect failures at multiple levels:

Service TCP check:

  1. Go to Monitors → New Monitor
  2. Choose TCP Port
  3. Enter: Host 203.0.113.90, Port 80
  4. Save the monitor

Virtual Kubelet process health via Ingress:

Expose the Virtual Kubelet /healthz endpoint externally via an Ingress so Vigilmon can probe it directly:

# vk-health-ingress.yaml
apiVersion: v1
kind: Service
metadata:
  name: virtual-kubelet-health
  namespace: virtual-kubelet
spec:
  selector:
    app: virtual-kubelet
  ports:
    - port: 10255
      targetPort: 10255
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: virtual-kubelet-health
  namespace: virtual-kubelet
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /healthz
spec:
  rules:
    - host: vk-health.your-cluster.com
      http:
        paths:
          - path: /healthz
            pathType: Exact
            backend:
              service:
                name: virtual-kubelet-health
                port:
                  number: 10255

kubectl apply -f vk-health-ingress.yaml

Add an HTTP monitor in Vigilmon for https://vk-health.your-cluster.com/healthz expecting 200 and body ok.

Step 4: Configure alert channels

A Virtual Kubelet failure means workloads that appear scheduled are not serving traffic — users notice before on-call does.

Email alerts

  1. In Vigilmon, go to Alert Channels → Add Channel → Email
  2. Enter your platform team's on-call email
  3. Assign the channel to all Virtual Kubelet monitors

Webhook alerts (Slack, PagerDuty, etc.)

  1. Go to Alert Channels → Add Channel → Webhook
  2. Enter your webhook URL
  3. Assign the channel to your monitors

Example payload when the VK health endpoint goes down:

{
  "monitor_name": "Virtual Kubelet Health",
  "status": "down",
  "url": "http://203.0.113.90/health",
  "started_at": "2024-01-15T09:45:00Z",
  "duration_seconds": 180
}

Run these diagnostics immediately when an alert fires:

# Check Virtual Kubelet process
kubectl get pods -n virtual-kubelet
kubectl logs -n virtual-kubelet -l app=virtual-kubelet --tail=100

# Check virtual node status
kubectl describe node virtual-node-aci

# Check pods on the virtual node
kubectl get pods -A --field-selector spec.nodeName=virtual-node-aci

# Check for provider API errors in VK logs
kubectl logs -n virtual-kubelet -l app=virtual-kubelet --tail=200 | grep -i "error\|failed\|unauthorized"

# Check provider-specific status (ACI example)
az container list --resource-group your-rg --output table

Step 5: Create a public status page

Serverless backends behind Virtual Kubelet often serve end users directly. Give them a status page.

  1. In Vigilmon, go to Status Pages → New Status Page
  2. Give it a name: "Virtual Kubelet Provider Status"
  3. Add your monitors: VK Health HTTP, Service TCP, VK Process Health
  4. Group them: Workload Availability, Provider Connectivity, VK Process
  5. Publish the page

Share this URL with your development teams whose workloads run on the virtual node.

Putting it all together

Here's a summary of the monitors to create for a Virtual Kubelet deployment:

| Monitor | Type | What it catches | |---------|------|-----------------| | http://203.0.113.90/health | HTTP | Full provider path (VK → provider API → pod → response) | | 203.0.113.90:80 | TCP | Service reachability | | https://vk-health.your-cluster.com/healthz | HTTP | Virtual Kubelet process health | | VK workload heartbeat | Heartbeat | Provider scheduling still works end-to-end |

And the Virtual Kubelet diagnostics to run when an alert fires:

# Quick triage: VK process or provider?
kubectl get pods -n virtual-kubelet
kubectl get node virtual-node-aci -o yaml | grep -A10 conditions

# Provider quota / auth check
kubectl logs -n virtual-kubelet deploy/virtual-kubelet | grep -E "quota|throttl|401|403|limit"

# Pod status on virtual node
kubectl get pods -A --field-selector spec.nodeName=virtual-node-aci -o wide

# Force VK node status refresh
kubectl annotate node virtual-node-aci kubectl.kubernetes.io/last-applied-configuration-

# Check events for VK-related issues
kubectl get events -A | grep -i "virtual\|vk\|provider"

What's next

  • SSL certificate monitoring — add SSL monitors for any HTTPS endpoints served from provider-backed workloads
  • Heartbeat monitoring — use Vigilmon heartbeats to verify scheduled jobs on the virtual node complete on time
  • Multi-provider coverage — if you run Virtual Kubelet with multiple providers, create a monitor set per provider to isolate failures

Get started free at vigilmon.online — no credit card, monitors start running in under a minute.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles