tutorial

How to Monitor OpenShift with Vigilmon (HTTP + TCP + Alerts)

Red Hat OpenShift is an enterprise Kubernetes platform that layers a rich set of additional components on top of upstream Kubernetes: an OAuth authentication...

Red Hat OpenShift is an enterprise Kubernetes platform that layers a rich set of additional components on top of upstream Kubernetes: an OAuth authentication server, a built-in container image registry, an HAProxy-based Router that replaces the standard Nginx Ingress controller, cluster operators that reconcile platform components, and etcd as the authoritative state store. Because OpenShift manages its own control-plane services as cluster operators, failures often cascade in ways that are invisible inside the cluster — a degraded OAuth server still reports Running pods while every oc login and kubectl call silently fails. External monitoring is the only reliable way to catch these outages before your users do.

Why OpenShift deployments need external monitoring

  • OAuth server outage blocks all cluster access: OpenShift's built-in OAuth server handles every oc login and token refresh. When it goes down, developers lose kubectl/oc access entirely, CI/CD pipelines fail to authenticate, and operators cannot respond to incidents — even if the workloads themselves are still running.
  • HAProxy Router crash drops all Route traffic: OpenShift uses Routes (not Kubernetes Ingress) as the primary external-access mechanism. A crashed or misconfigured HAProxy Router pod in the openshift-ingress namespace silently drops all incoming HTTP/HTTPS traffic to every application in the cluster, with no in-cluster alert firing.
  • Image registry unavailability breaks new pod startup: New deployments, rolling updates, and CronJob pods all pull images from the built-in registry. Registry downtime causes ImagePullBackOff errors across every namespace simultaneously, stalling all rollouts without any obvious single point of failure.
  • Cluster operator degraded state cascades silently: OpenShift's cluster operators (network, storage, console, DNS, etc.) manage platform infrastructure. An operator stuck in Degraded state can block upgrades, prevent new node joins, and degrade routing — yet the workload pods themselves keep running, masking the problem.
  • etcd quorum loss halts all cluster state changes: Losing a majority of etcd members causes the API server to return errors on all write operations. Pods already running continue, but no new deployments, config maps, or secrets can be applied, bringing CI/CD and incident response to a standstill.

What you'll need

  • An OpenShift cluster (OCP 4.x) with cluster-admin access and the oc CLI installed
  • A Vigilmon account (free tier includes HTTP, TCP, and status pages)
  • Network reachability from the public internet (or a Vigilmon probe on your network) to your cluster's router IP and API endpoint

Step 1: Expose a health endpoint for monitoring

OpenShift's HAProxy Router already exposes a stats and health port on every router pod. Expose it via a Service and a Route so Vigilmon can reach it externally.

Check the current router deployment:

oc -n openshift-ingress get pods -l ingresscontroller.operator.openshift.io/deployment-ingresscontroller=default
oc -n openshift-ingress get service router-internal-default

Create a dedicated health-check Service and Route:

# openshift-router-healthcheck.yaml
apiVersion: v1
kind: Service
metadata:
  name: router-health
  namespace: openshift-ingress
spec:
  selector:
    ingresscontroller.operator.openshift.io/deployment-ingresscontroller: default
  ports:
    - name: healthz
      port: 1936
      targetPort: 1936
  type: ClusterIP
---
apiVersion: route.openshift.io/v1
kind: Route
metadata:
  name: router-health
  namespace: openshift-ingress
spec:
  host: router-health.apps.cluster.example.com
  port:
    targetPort: healthz
  to:
    kind: Service
    name: router-health
  tls:
    termination: edge
    insecureEdgeTerminationPolicy: Redirect

oc apply -f openshift-router-healthcheck.yaml
oc -n openshift-ingress get route router-health

Also expose a lightweight health endpoint for your own application using a Route:

# app-health-route.yaml
apiVersion: route.openshift.io/v1
kind: Route
metadata:
  name: app-health
  namespace: my-app
spec:
  host: my-app.apps.cluster.example.com
  path: /healthz
  port:
    targetPort: http
  to:
    kind: Service
    name: my-app
  tls:
    termination: edge
    insecureEdgeTerminationPolicy: Redirect

oc apply -f app-health-route.yaml
curl -f https://my-app.apps.cluster.example.com/healthz

Step 2: Set up HTTP monitoring in Vigilmon

  1. Log in to vigilmon.online and open the Monitors tab.
  2. Click Add Monitor and select HTTP(S).
  3. Enter the router health URL: https://router-health.apps.cluster.example.com/healthz.
  4. Set the Check interval to 60 seconds and the Timeout to 10 seconds.
  5. Under Expected status code, enter 200. Optionally add a Keyword check for the string alive to verify the HAProxy stats page content.
  6. Click Save Monitor. Vigilmon begins issuing checks from external probes immediately.

Repeat steps 2–6 for the OpenShift console URL:

  • URL: https://console-openshift-console.apps.cluster.example.com
  • Expected status: 200
  • Keyword check: OpenShift

And for your application Route:

  • URL: https://my-app.apps.cluster.example.com/healthz
  • Expected status: 200

Why external monitoring catches what Kubernetes misses

OpenShift's internal probes — liveness, readiness, and cluster operator health checks — run inside the cluster network. They cannot detect a BGP route withdrawal that makes your router IP unreachable from the internet, a TLS certificate expiry that causes browsers to refuse the connection, or a DNS misconfiguration that points your Route hostname to a dead IP. Vigilmon's probes hit your endpoints from the public internet the same way your users and pipelines do, catching the full class of external-facing failures.

Step 3: Monitor TCP ports and the OpenShift health endpoint

Add TCP monitors for the Kubernetes API server and the OAuth server, which do not serve HTTP on their primary ports:

In Vigilmon, click Add Monitor and select TCP:

  • Kubernetes API server: Host api.cluster.example.com, Port 6443
  • OAuth server: Host oauth-openshift.apps.cluster.example.com, Port 443
  • etcd peer port (if reachable from your probe): Host 203.0.113.50, Port 2380

Check the OAuth server's /healthz endpoint directly via HTTP:

# oauth-health-route.yaml
apiVersion: route.openshift.io/v1
kind: Route
metadata:
  name: oauth-health
  namespace: openshift-authentication
spec:
  host: oauth-health.apps.cluster.example.com
  port:
    targetPort: 6443
  to:
    kind: Service
    name: oauth-openshift
  tls:
    termination: reencrypt
    insecureEdgeTerminationPolicy: Redirect

oc apply -f oauth-health-route.yaml
curl -k https://oauth-openshift.apps.cluster.example.com/healthz

Add this as a Vigilmon HTTP monitor:

  • URL: https://oauth-openshift.apps.cluster.example.com/healthz
  • Expected status: 200
  • Keyword check: ok

Also add the image registry health endpoint:

oc -n openshift-image-registry get route default-route
  • URL: https://default-route-openshift-image-registry.apps.cluster.example.com/healthz
  • Expected status: 200

Step 4: Configure alert channels

Email alerts:

  1. In Vigilmon, go to Alert Channels and click Add Channel.
  2. Select Email and enter your operations team address (e.g. ops@example.com).
  3. Set Alert on: Down, Recovered, Certificate expiry (30 days).
  4. Click Save, then click Send test alert to confirm delivery.
  5. Assign this channel to all four monitors created above under each monitor's Alerts tab.

Webhook alerts (for Slack, PagerDuty, or custom runbooks):

  1. In Vigilmon, click Add Channel and select Webhook.
  2. Enter your endpoint URL (e.g. https://hooks.slack.com/services/YOUR/SLACK/WEBHOOK).
  3. Vigilmon sends a POST request with a JSON payload on every state change:
{
  "monitor": "OpenShift HAProxy Router",
  "status": "down",
  "url": "https://router-health.apps.cluster.example.com/healthz",
  "checked_at": "2026-07-11T08:23:01Z",
  "response_code": 0,
  "response_time_ms": null,
  "message": "Connection timed out"
}
  1. Use this payload in a PagerDuty Events API integration or a custom webhook receiver that pages on-call engineers.
  2. Assign the webhook channel to all monitors.

Run these commands to diagnose router alerts when they fire:

# Check router pod status
oc -n openshift-ingress get pods -l ingresscontroller.operator.openshift.io/deployment-ingresscontroller=default

# Check cluster operators for degraded state
oc get clusteroperators | grep -v "True.*False.*False"

# Check OAuth server pods
oc -n openshift-authentication get pods

# Check image registry operator
oc -n openshift-image-registry get pods

# Verify etcd member health
oc -n openshift-etcd get pods -l app=etcd
oc rsh -n openshift-etcd etcd-master-0 etcdctl member list --write-out=table

# Check recent router events
oc -n openshift-ingress get events --sort-by='.lastTimestamp' | tail -20

# Pull router logs
oc -n openshift-ingress logs -l ingresscontroller.operator.openshift.io/deployment-ingresscontroller=default --tail=50

Step 5: Create a public status page

  1. In Vigilmon, go to Status Pages and click New Status Page.
  2. Name it OpenShift Cluster Status and choose a subdomain such as status.cluster.example.com.
  3. Add the following monitors to the page:
    • OpenShift HAProxy Router (HTTP)
    • OpenShift Console (HTTP)
    • Kubernetes API Server (TCP)
    • OAuth Server /healthz (HTTP)
    • Image Registry (HTTP)
    • Application Health (HTTP)
  4. Enable 90-day uptime history bars and incident history.
  5. Click Publish. Share the URL with your development teams and stakeholders so they can self-serve during incidents rather than filing support tickets.

Putting it all together

| Monitor | Type | What it catches | |---|---|---| | HAProxy Router /healthz | HTTP | Router pod crash, HAProxy config reload failure, Route traffic blackhole | | OpenShift Console | HTTP | Console operator degraded, ingress misconfiguration, TLS cert expiry | | Kubernetes API Server | TCP | API server pod down, etcd quorum loss, network partition to control plane | | OAuth Server /healthz | HTTP | OAuth server crash, OIDC provider misconfiguration, all oc login failures | | Image Registry | HTTP | Registry pod down, storage backend failure, new pod startup stalls | | Application Route | HTTP | App-specific failures visible from outside the cluster |

# Quick OpenShift cluster health snapshot
echo "=== Cluster Operators ==="
oc get clusteroperators

echo "=== Router Pods ==="
oc -n openshift-ingress get pods

echo "=== OAuth Pods ==="
oc -n openshift-authentication get pods

echo "=== Image Registry Pods ==="
oc -n openshift-image-registry get pods

echo "=== etcd Pods ==="
oc -n openshift-etcd get pods -l app=etcd

echo "=== Node Status ==="
oc get nodes

echo "=== Recent Cluster Events ==="
oc get events -A --sort-by='.lastTimestamp' | grep -i "warning\|error\|failed" | tail -30

echo "=== Router External IP ==="
oc -n openshift-ingress get service router-external-default -o jsonpath='{.status.loadBalancer.ingress[0].ip}'

What's next

  • Add a certificate expiry monitor in Vigilmon for your cluster's wildcard certificate (*.apps.cluster.example.com) and the API server certificate (api.cluster.example.com) to get 30-day advance warnings before TLS outages.
  • Configure response-time alerting on the HAProxy Router health check — a spike from under 100 ms to over 2 seconds reliably predicts imminent router overload before requests start dropping.
  • Extend monitoring to individual application namespaces by adding HTTP monitors for each team's Route, and group them under a single status page per team so cluster operations and application teams have separate visibility.

Get started free at vigilmon.online — no credit card, monitors start running in under a minute.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles