How to Monitor HyperShift with Vigilmon (HTTP + TCP + Alerts)

HyperShift is Red Hat's architecture for running multiple OpenShift or Kubernetes control planes as ordinary pods within a single management cluster. Instead of dedicating a set of control-plane nodes to each tenant cluster, HyperShift collapses every hosted cluster's API server, etcd, controller manager, and scheduler into a namespace of pods on the management cluster. Worker nodes (NodePools) connect back to that hosted control plane over the network. This design enables dramatically higher control-plane density and faster cluster provisioning, but it introduces a failure topology that conventional cluster monitoring tools are blind to: a hosted cluster's API server is just a pod, and pod restarts are invisible to the tenant's users and to the hosted cluster's own internal health checks. The management cluster's health now determines the availability of every hosted cluster simultaneously, making external monitoring not just useful but essential.

Why HyperShift deployments need external monitoring

HyperShift operator crash stops all HostedCluster reconciliation: The hypershift-operator pod on the management cluster reconciles every HostedCluster object and reacts to NodePool changes. When it crashes or enters a restart loop, new HostedCluster creation stalls, NodePool scaling freezes, and configuration changes to hosted control planes are silently ignored — with no error surfaced inside the hosted clusters themselves.
Hosted API server pod crash makes the tenant cluster unreachable: Each HostedCluster's API server runs as a pod in a namespace like clusters-my-hosted-cluster. A crash or OOMKill of that pod immediately makes kubectl and oc commands against the hosted cluster return connection refused, blocking all CI/CD pipelines and operator access for that tenant.
NodePool machine provisioning failures leave workers in NotReady: When the machine-provider (AWS, Azure, bare metal) fails to provision new nodes, NodePool's MachineDeployment stalls silently. The hosted cluster scheduler cannot place workloads on new nodes, but no alert fires inside the hosted cluster because the failure lives entirely within the management cluster's machine-api namespace.
etcd pod crash halts all hosted cluster state changes: Each HostedCluster runs its own etcd pod (or pods) inside the management cluster. An etcd OOMKill or storage I/O error causes the hosted cluster's API server to reject all write operations, stalling deployments, config changes, and secret rotations for that tenant while reads may continue to succeed.
Management cluster API server outage cascades to all hosted clusters simultaneously: Unlike standalone clusters where a single control plane serves one tenant, a management cluster API server outage simultaneously disrupts reconciliation of every HostedCluster it hosts. All NodePool scaling, all HostedCluster status updates, and the hypershift-operator itself stop functioning — a single point of failure with tenant-wide blast radius.

What you'll need

A management cluster running the HyperShift operator (hypershift-operator pod in the hypershift namespace) with cluster-admin access
At least one HostedCluster deployed and accessible (use hypershift create cluster or the Assisted Installer)
A Vigilmon account (free tier covers HTTP, TCP monitors, and status pages)

Step 1: Expose a health endpoint for monitoring

The HyperShift operator pod serves a metrics and health endpoint on port 9000. Expose it via a Service and Route on the management cluster:

# Verify the hypershift-operator pod and its health port
oc -n hypershift get pods -l name=operator
oc -n hypershift get pod -l name=operator -o jsonpath='{.items[0].spec.containers[0].ports}'

Create a Service and Route to expose the operator health endpoint:

# hypershift-operator-healthcheck.yaml
apiVersion: v1
kind: Service
metadata:
  name: hypershift-operator-health
  namespace: hypershift
spec:
  selector:
    name: operator
  ports:
    - name: healthz
      port: 9000
      targetPort: 9000
  type: ClusterIP
---
apiVersion: route.openshift.io/v1
kind: Route
metadata:
  name: hypershift-operator-health
  namespace: hypershift
spec:
  host: hypershift-health.apps.management.example.com
  port:
    targetPort: healthz
  to:
    kind: Service
    name: hypershift-operator-health
  tls:
    termination: edge
    insecureEdgeTerminationPolicy: Redirect

oc apply -f hypershift-operator-healthcheck.yaml
curl -f https://hypershift-health.apps.management.example.com/healthz

Expose the hosted cluster API server's /healthz endpoint. The hosted API server listens on port 6443 inside the management cluster namespace and is fronted by a load balancer or NodePort service:

# Find the hosted API server service for your HostedCluster
oc -n clusters-my-hosted-cluster get service kube-apiserver

# Get the external hostname or IP assigned to the hosted API server
oc -n clusters-my-hosted-cluster get service kube-apiserver \
  -o jsonpath='{.status.loadBalancer.ingress[0].hostname}'

The hosted cluster API server's external health endpoint is:

https://api.my-hosted-cluster.hypershift.example.com:6443/healthz

Verify reachability:

curl -k https://api.my-hosted-cluster.hypershift.example.com:6443/healthz
# Expected output: ok

Step 2: Set up HTTP monitoring in Vigilmon

Log in to vigilmon.online and open the Monitors tab.
Click Add Monitor and select HTTP(S).
Enter the HyperShift operator health URL: https://hypershift-health.apps.management.example.com/healthz.
Set the Check interval to 60 seconds and the Timeout to 10 seconds.
Under Expected status code, enter 200. Add a Keyword check for ok to verify the response body.
Click Save Monitor. Vigilmon begins issuing checks from external probes immediately.

Repeat steps 2–6 for the hosted cluster API server:

URL: https://api.my-hosted-cluster.hypershift.example.com:6443/healthz
Expected status: 200
Keyword check: ok
Note: Select Skip TLS verification if using a self-signed certificate, or upload the cluster CA under Advanced settings.

Why external monitoring catches what Kubernetes misses

HyperShift's internal readiness and liveness probes only detect whether a pod restarts, not whether the pod is serving valid responses to real client requests. A hosted API server can pass its own liveness probe while returning 503 errors to tenant kubectl calls due to an etcd connection failure. Vigilmon's probes send real HTTP requests from outside the cluster network — the same path a developer's laptop or CI runner takes — catching TLS errors, load balancer misrouting, and application-layer failures that in-cluster probes never see.

Step 3: Monitor TCP ports and the HyperShift health endpoint

Add TCP monitors for the hosted cluster API server port and the management cluster API server:

In Vigilmon, click Add Monitor and select TCP:

Management cluster API server: Host api.management.example.com, Port 6443
Hosted cluster API server: Host api.my-hosted-cluster.hypershift.example.com, Port 6443
Hosted cluster etcd (if the etcd client port is externally reachable): Host 203.0.113.51, Port 2379

Expose the hosted cluster's etcd health endpoint for HTTP monitoring. The etcd pod runs inside clusters-my-hosted-cluster:

# hosted-etcd-health-service.yaml
apiVersion: v1
kind: Service
metadata:
  name: etcd-health
  namespace: clusters-my-hosted-cluster
spec:
  selector:
    app: etcd
  ports:
    - name: client
      port: 2379
      targetPort: 2379
    - name: metrics
      port: 2381
      targetPort: 2381
  type: ClusterIP
---
apiVersion: route.openshift.io/v1
kind: Route
metadata:
  name: etcd-health
  namespace: clusters-my-hosted-cluster
spec:
  host: etcd-health-my-hosted-cluster.apps.management.example.com
  port:
    targetPort: metrics
  to:
    kind: Service
    name: etcd-health
  tls:
    termination: edge
    insecureEdgeTerminationPolicy: Redirect

oc apply -f hosted-etcd-health-service.yaml
curl https://etcd-health-my-hosted-cluster.apps.management.example.com/metrics | grep etcd_server_is_leader

Add the etcd metrics endpoint as an HTTP monitor in Vigilmon:

URL: https://etcd-health-my-hosted-cluster.apps.management.example.com/metrics
Expected status: 200
Keyword check: etcd_server_is_leader 1

Step 4: Configure alert channels

Email alerts:

In Vigilmon, go to Alert Channels and click Add Channel.
Select Email and enter your platform-engineering team address (e.g. platform@example.com).
Set Alert on: Down, Recovered, Certificate expiry (30 days).
Click Save, then click Send test alert to confirm delivery.
Assign this channel to all monitors created above under each monitor's Alerts tab.

Webhook alerts (for Slack, PagerDuty, or custom runbooks):

In Vigilmon, click Add Channel and select Webhook.
Enter your endpoint URL (e.g. https://hooks.slack.com/services/YOUR/SLACK/WEBHOOK).
Vigilmon sends a POST request with a JSON payload on every state change:

{
  "monitor": "HyperShift Hosted API Server (my-hosted-cluster)",
  "status": "down",
  "url": "https://api.my-hosted-cluster.hypershift.example.com:6443/healthz",
  "checked_at": "2026-07-11T09:14:33Z",
  "response_code": 0,
  "response_time_ms": null,
  "message": "Connection refused"
}

Route alerts to different channels per HostedCluster by creating separate webhook channels for each tenant team.
Assign the webhook channel to all monitors.

Run these commands to diagnose HyperShift alerts when they fire:

# Check hypershift-operator pod status on management cluster
oc -n hypershift get pods -l name=operator
oc -n hypershift logs -l name=operator --tail=50

# List all HostedClusters and their conditions
oc get hostedcluster -n clusters
oc describe hostedcluster my-hosted-cluster -n clusters | grep -A 10 "Conditions:"

# Check hosted control plane pods for a specific cluster
oc -n clusters-my-hosted-cluster get pods

# Check the hosted API server pod logs
oc -n clusters-my-hosted-cluster logs -l app=kube-apiserver --tail=50

# Check etcd pod health inside the hosted control plane namespace
oc -n clusters-my-hosted-cluster get pods -l app=etcd
oc -n clusters-my-hosted-cluster exec -it etcd-0 -- etcdctl endpoint health \
  --cacert /etc/etcd/tls/etcd-ca/ca.crt \
  --cert /etc/etcd/tls/client/etcd-client.crt \
  --key /etc/etcd/tls/client/etcd-client.key

# Check NodePool status
oc get nodepool -n clusters
oc describe nodepool my-hosted-cluster -n clusters | grep -A 10 "Status:"

# Check management cluster nodes (underlying NodePool capacity)
oc get nodes --show-labels | grep hypershift

Step 5: Create a public status page

In Vigilmon, go to Status Pages and click New Status Page.
Name it HyperShift Platform Status and choose a subdomain such as status.hypershift.example.com.
Add the following monitors to the page, grouped by component:
- HyperShift Operator (HTTP) — platform layer
- Management Cluster API Server (TCP) — platform layer
- Hosted Cluster API Server — my-hosted-cluster (HTTP)
- Hosted Cluster etcd Metrics — my-hosted-cluster (HTTP)
Create a second status page per tenant (e.g. status-tenant-a.hypershift.example.com) showing only that tenant's hosted cluster monitors, so tenant teams have self-service visibility without seeing management cluster internals.
Enable 90-day uptime history bars and incident history on all pages.
Click Publish and distribute the URLs to platform engineering and tenant teams.

Putting it all together

| Monitor | Type | What it catches | |---|---|---| | HyperShift Operator /healthz | HTTP | Operator crash, reconciliation loop failure, new HostedCluster creation stalled | | Management Cluster API Server | TCP | Management control plane down, cascading impact on all hosted clusters | | Hosted API Server /healthz | HTTP | Tenant API server pod crash, OOMKill, etcd disconnection, TLS cert expiry | | Hosted API Server port | TCP | Load balancer misrouting, network policy blocking tenant access | | Hosted etcd metrics | HTTP | etcd leader election failure, storage I/O errors, quorum loss | | NodePool Machine Endpoint | TCP | Provisioner unreachable, worker node join failures |

# Full HyperShift platform health snapshot
echo "=== HyperShift Operator ==="
oc -n hypershift get pods -l name=operator

echo "=== All HostedClusters ==="
oc get hostedcluster -A

echo "=== All NodePools ==="
oc get nodepool -A

echo "=== Hosted Control Plane Pods (my-hosted-cluster) ==="
oc -n clusters-my-hosted-cluster get pods

echo "=== etcd Health (my-hosted-cluster) ==="
oc -n clusters-my-hosted-cluster get pods -l app=etcd

echo "=== Management Cluster Nodes ==="
oc get nodes

echo "=== Recent Management Cluster Events ==="
oc get events -A --sort-by='.lastTimestamp' \
  | grep -i "warning\|error\|failed\|oomkill" | tail -30

echo "=== HostedCluster Conditions ==="
oc get hostedcluster -n clusters -o jsonpath=\
'{range .items[*]}{.metadata.name}{"\t"}{range .status.conditions[*]}{.type}={.status}{" "}{end}{"\n"}{end}'

What's next

Add a certificate expiry monitor in Vigilmon for each hosted cluster API server certificate (api.my-hosted-cluster.hypershift.example.com) — certificate rotation failures in HyperShift are hard to detect because the cert lives in a management cluster secret, not on a node.
Scale monitoring to multiple HostedClusters by duplicating the monitor set for each cluster name and grouping them under per-tenant status pages — Vigilmon's monitor groups make this manageable at dozens of hosted clusters.
Add a response-time alert threshold on the hosted API server /healthz check: a degraded etcd consistently inflates API server response times to over 500 ms before requests start failing, giving you early warning to intervene before tenant workloads are impacted.

Get started free at vigilmon.online — no credit card, monitors start running in under a minute.