HNC (Hierarchical Namespace Controller) is the Kubernetes add-on that enables hierarchical namespace trees — letting platform teams organize namespaces into parent-child relationships where resources, RBAC, network policies, and resource quotas propagate automatically from parent to child namespaces. When HNC is healthy, multi-team resource sharing and policy propagation happen automatically and invisibly. When it fails, one of two things happens: either namespace propagation silently stops and child namespaces drift out of sync with their parent's policies (a security gap that's invisible until audit time), or the HNC webhook fails and namespace creation operations are rejected. Vigilmon monitors HNC's health endpoints from outside the cluster, alerting your platform team the moment either failure mode appears.
What You'll Build
- HTTP monitors for HNC's controller manager health endpoint
- HTTP monitors for HNC's admission webhook
- A heartbeat monitor to verify namespace propagation is running
- Tiered alert routing: webhook failures page on-call, propagation drift notifies the platform team
Prerequisites
- HNC installed in a Kubernetes cluster (version 1.1+)
- At least one parent-child namespace hierarchy configured
- An Ingress controller exposing HNC health endpoints over HTTPS
- A free account at vigilmon.online
Why HNC Monitoring Is Not Optional
HNC's failure modes are uniquely dangerous in multi-team environments:
Silent policy drift. HNC propagates RoleBindings, NetworkPolicies, and ResourceQuotas from parent namespaces to children. If the HNC controller crashes, the propagation reconciliation loop stops. New resources added to parent namespaces after the crash won't appear in children — but existing propagated resources remain, so nothing visibly breaks until you need the newly added policy to be enforced.
Namespace creation webhook failures block tenant onboarding. HNC registers a ValidatingWebhookConfiguration and MutatingWebhookConfiguration that intercepts namespace creation and modification. If the webhook is unreachable, creating new subnamespaces (via SubnamespaceAnchor resources) fails with connection refused, blocking all tenant onboarding operations.
Hierarchy violations create security gaps. HNC enforces that child namespaces can't have resources that conflict with parent-level restrictions. If the controller fails while a hierarchy modification is in progress, namespaces can end up in an inconsistent state where the hierarchy tree no longer matches the propagated resources — creating unintended permissions.
Cross-namespace resource sharing breaks invisibly. If your teams rely on HNC's propagate.hnc.x-k8s.io annotations to share ConfigMaps or Secrets across namespace boundaries, a controller failure means changes to those resources in the parent stop reaching children — silently, without any error on the consumers.
Step 1: Verify HNC Health Endpoints
HNC's controller manager exposes health endpoints on port 8080 (metrics) and 9443 (webhook):
# Check HNC controller manager health
kubectl exec -n hnc-system deploy/hnc-controller-manager -- \
wget -qO- http://localhost:8080/healthz
# Check the webhook is responding
kubectl get validatingwebhookconfigurations | grep hnc
kubectl get mutatingwebhookconfigurations | grep hnc
# Verify namespace hierarchy is healthy
kubectl get hierarchyconfigurations -A
# Check for any HNC condition errors
kubectl get namespaces -l "hnc.x-k8s.io/managed-by=hnc"
Look for HierarchyConfigurationCondition objects with Active: False — these indicate propagation issues:
kubectl get hierarchyconfiguration hierarchy -n <namespace> -o yaml | grep -A5 conditions
Step 2: Expose Health Endpoints via Ingress
# hnc-health-services.yaml
---
apiVersion: v1
kind: Service
metadata:
name: hnc-controller-health
namespace: hnc-system
spec:
selector:
control-plane: controller-manager
app.kubernetes.io/name: hnc
ports:
- name: health
port: 8080
targetPort: 8080
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: hnc-health
namespace: hnc-system
spec:
rules:
- host: hnc-health.internal.yourdomain.com
http:
paths:
- path: /healthz
pathType: Exact
backend:
service:
name: hnc-controller-health
port:
number: 8080
- path: /readyz
pathType: Exact
backend:
service:
name: hnc-controller-health
port:
number: 8080
Apply and verify:
kubectl apply -f hnc-health-services.yaml
curl https://hnc-health.internal.yourdomain.com/healthz
# Expected: 200 OK with "ok"
Step 3: Add Vigilmon HTTP Monitors
In the Vigilmon dashboard:
- Go to Add Monitor → HTTP
- Configure the following monitors:
| Monitor Name | URL | Priority |
|---|---|---|
| hnc controller liveness | https://hnc-health.yourdomain.com/healthz | Critical |
| hnc controller readiness | https://hnc-health.yourdomain.com/readyz | Critical |
Settings for all monitors:
- Check interval: 1 minute
- Timeout: 10 seconds
- Expected status code: 200
- Alert after: 2 consecutive failures
HNC's webhook directly gates namespace creation and hierarchy changes — an HNC outage during a new tenant onboarding can block your entire platform provisioning pipeline.
Step 4: Monitor HNC Webhook TLS and Namespace Propagation
The HNC webhook uses TLS. An expired certificate causes all SubnamespaceAnchor creation and hierarchy mutations to fail with cryptic x509 errors:
# Check HNC webhook certificate expiry
kubectl get secret hnc-webhook-server-cert -n hnc-system \
-o jsonpath='{.data.tls\.crt}' | base64 -d | \
openssl x509 -noout -enddate
In Vigilmon, enable SSL certificate monitoring on the HNC webhook domain with a 14-day expiry alert.
Verify propagation is working by checking propagated object counts:
# Count propagated RoleBindings in child namespaces
kubectl get rolebindings -A \
-l "hnc.x-k8s.io/inherited-from" --no-headers | wc -l
If this count drops suddenly, propagation has stalled. Wire this into your heartbeat monitor (Step 5).
Step 5: Heartbeat Monitor for Namespace Propagation
Create a CronJob that verifies HNC is actively propagating resources from parent to child namespaces:
# hnc-propagation-heartbeat.yaml
apiVersion: batch/v1
kind: CronJob
metadata:
name: hnc-propagation-heartbeat
namespace: hnc-system
spec:
schedule: "*/15 * * * *"
jobTemplate:
spec:
template:
spec:
restartPolicy: OnFailure
serviceAccountName: hnc-heartbeat
containers:
- name: heartbeat
image: bitnami/kubectl:latest
env:
- name: PARENT_NS
value: "your-parent-namespace"
- name: CHILD_NS
value: "your-child-namespace"
- name: HEARTBEAT_URL
valueFrom:
secretKeyRef:
name: vigilmon-secrets
key: hnc-heartbeat-url
command:
- /bin/sh
- -c
- |
set -e
# Verify parent-child relationship is active
HNC_STATUS=$(kubectl get hierarchyconfiguration hierarchy \
-n "$CHILD_NS" \
-o jsonpath='{.status.conditions[?(@.type=="ActivitiesHalted")].status}' 2>/dev/null)
if [ "$HNC_STATUS" = "True" ]; then
echo "FAIL: HNC propagation halted in $CHILD_NS"
exit 1
fi
# Verify propagated resources exist in child namespace
INHERITED=$(kubectl get rolebindings -n "$CHILD_NS" \
-l "hnc.x-k8s.io/inherited-from=$PARENT_NS" \
--no-headers 2>/dev/null | wc -l)
if [ "$INHERITED" -eq 0 ]; then
echo "WARN: No inherited RoleBindings in $CHILD_NS — propagation may be stalled"
exit 1
fi
wget -qO- "$HEARTBEAT_URL"
echo "HNC propagation heartbeat sent"
Create the required RBAC and secret:
kubectl create serviceaccount hnc-heartbeat -n hnc-system
kubectl create clusterrolebinding hnc-heartbeat-reader \
--clusterrole=view \
--serviceaccount=hnc-system:hnc-heartbeat
kubectl create secret generic vigilmon-secrets \
--from-literal=hnc-heartbeat-url='https://vigilmon.online/heartbeat/YOUR_TOKEN' \
-n hnc-system
In Vigilmon, create a Heartbeat monitor:
- Name:
hnc namespace propagation - Expected interval: 15 minutes
- Grace period: 5 minutes
Step 6: Monitor SubnamespaceAnchor Status
HNC uses SubnamespaceAnchor resources to represent child namespaces. An anchor in Conflict or Forbidden state indicates a broken hierarchy:
# Check for SubnamespaceAnchor issues across all namespaces
kubectl get subnamespaceanchors -A \
-o custom-columns='NS:.metadata.namespace,NAME:.metadata.name,STATUS:.status.state' | \
grep -v "Ok"
Add a monitoring CronJob that alerts on non-Ok anchor states:
# Add to the heartbeat CronJob command:
ANCHORS_NOT_OK=$(kubectl get subnamespaceanchors -A \
-o jsonpath='{.items[?(@.status.state!="Ok")].metadata.name}' 2>/dev/null)
if [ -n "$ANCHORS_NOT_OK" ]; then
echo "WARN: SubnamespaceAnchors not Ok: $ANCHORS_NOT_OK"
exit 1
fi
Step 7: Configure Alert Routing
Route alerts by impact:
Critical — HNC controller down or webhook failing (page on-call):
- Alert Channels → Add Channel → PagerDuty
- Assign to:
hnc controller liveness,hnc controller readiness
High — Propagation stalled or hierarchy broken (notify platform team Slack):
- Alert Channels → Add Channel → Slack Webhook
- Paste your
#platform-alertswebhook URL - Assign to:
hnc namespace propagationheartbeat
This separation reflects the operational reality: HNC controller downtime blocks namespace operations immediately; propagation drift is a serious but slower-burning problem that the platform team needs to investigate.
What You're Now Monitoring
| Component | Monitor Type | Failure Detected | |---|---|---| | HNC controller liveness | HTTP health | Controller crash stops all reconciliation | | HNC controller readiness | HTTP readiness | Controller running but not reconciling | | HNC webhook TLS | SSL cert expiry | Certificate expiry blocks namespace operations | | Namespace propagation | Heartbeat | Inherited resources no longer propagating | | SubnamespaceAnchor status | Heartbeat | Hierarchy conflicts or forbidden states |
Conclusion
HNC is the invisible foundation of multi-team Kubernetes — when it works, teams get automatic policy inheritance and clean namespace isolation. When it fails, the effects are diffuse and delayed: policies drift, onboarding breaks, and security guarantees quietly erode. Vigilmon gives your platform team the external vantage point to catch HNC failures before they become audit findings or security incidents.
Start monitoring HNC for free at vigilmon.online. Your first monitor is running in under two minutes.