tutorial

Monitoring Kubernetes Hierarchical Namespace Controller (HNC) with Vigilmon

The Hierarchical Namespace Controller (HNC) brings namespace trees to Kubernetes—child namespaces that inherit RBAC roles, network policies, resource quotas,...

The Hierarchical Namespace Controller (HNC) brings namespace trees to Kubernetes—child namespaces that inherit RBAC roles, network policies, resource quotas, and config maps from their parents automatically. When HNC works, multi-tenant clusters become dramatically easier to manage. When HNC breaks, propagation silently stops: child namespaces no longer inherit parent policies, RBAC grants disappear, network policies fail to propagate, and tenants gain or lose access to resources without warning.

This tutorial shows how to monitor HNC's propagation health, sync errors, namespace tree integrity, and RBAC propagation status using Vigilmon—so you know immediately when your namespace hierarchy stops working.

Why HNC needs monitoring

HNC fails in ways that have major security and operational consequences:

  • Propagation controller crashes — the HNC controller exits or enters CrashLoopBackOff; propagation stops for all namespaces while the cluster appears healthy
  • Anchor synchronization stalls — child namespace anchors diverge from parent state; a child namespace is supposed to inherit a NetworkPolicy but the propagated copy is stale or missing
  • Cycle detection triggers — a misconfigured namespace hierarchy creates a cycle; HNC halts propagation for the affected subtree and surfaces a condition that most operators don't monitor
  • RBAC propagation lag — new RoleBindings added to a parent namespace take minutes or hours to appear in children, or never propagate due to an HNC webhook failure
  • Resource quota overrides — a namespace admin overrides a propagated ResourceQuota, which HNC then tries to re-propagate, creating a conflict loop visible only in the HNC controller logs

What you'll need

  • A Kubernetes cluster with HNC installed (kubectl-hns plugin recommended)
  • HNC controller running in hnc-system namespace
  • A free Vigilmon account

Step 1: Verify HNC controller status and metrics

Check the controller is running and expose its metrics:

# Check HNC controller status
kubectl get pods -n hnc-system

# Verify HNC is installed and working
kubectl hns tree default

# Check for propagation errors across all namespaces
kubectl get hierarchyconfigurations --all-namespaces -o wide

HNC exposes Prometheus metrics on port 8080. Create a metrics service:

# hnc-metrics-service.yaml
apiVersion: v1
kind: Service
metadata:
  name: hnc-metrics
  namespace: hnc-system
spec:
  selector:
    control-plane: controller-manager
  ports:
    - name: metrics
      port: 8080
      targetPort: 8080

Verify key metrics:

kubectl port-forward -n hnc-system svc/hnc-metrics 8080:8080
curl http://localhost:8080/metrics | grep hnc

Expected output:

# HELP hnc_reconciler_errors_total Number of reconciler errors
hnc_reconciler_errors_total{reconciler="HierarchyConfig"} 0
hnc_reconciler_errors_total{reconciler="ObjectReconciler"} 0

# HELP hnc_objects_propagated_total Objects propagated to descendant namespaces
hnc_objects_propagated_total{group="",version="v1",resource="configmaps"} 84
hnc_objects_propagated_total{group="rbac.authorization.k8s.io",version="v1",resource="rolebindings"} 156

Step 2: Monitor namespace tree health

Build a health bridge that inspects the HNC hierarchy for errors and propagation lag:

# hnc-health-bridge.py
import os, subprocess, json
from flask import Flask, jsonify

app = Flask(__name__)

def get_hierarchy_conditions():
    """Check all HierarchyConfiguration objects for conditions."""
    result = subprocess.run(
        ["kubectl", "get", "hierarchyconfigurations", "--all-namespaces", "-o", "json"],
        capture_output=True, text=True, timeout=15
    )
    if result.returncode != 0:
        return None, f"kubectl error: {result.stderr}"

    data = json.loads(result.stdout)
    errors = []
    warnings = []

    for item in data.get("items", []):
        ns = item["metadata"]["namespace"]
        conditions = item.get("status", {}).get("conditions", [])
        for cond in conditions:
            if cond.get("status") == "True" and cond.get("type") in ("ParentMissing", "CriticalPathParentMissing", "BadConfiguration", "HasCycles"):
                errors.append(f"{ns}: {cond['type']} — {cond.get('message', '')}")
            elif cond.get("status") == "True" and cond.get("type") in ("SubnamespaceAnchorMissing",):
                warnings.append(f"{ns}: {cond['type']}")

    return {"errors": errors, "warnings": warnings, "total_ns": len(data.get("items", []))}, None

def check_propagated_objects():
    """Verify propagated RoleBindings exist in child namespaces."""
    result = subprocess.run(
        ["kubectl", "get", "rolebindings", "--all-namespaces", "-o", "json"],
        capture_output=True, text=True, timeout=15
    )
    if result.returncode != 0:
        return None, result.stderr

    data = json.loads(result.stdout)
    propagated = [
        item for item in data.get("items", [])
        if item["metadata"].get("annotations", {}).get("hnc.x-k8s.io/inherited-from")
    ]
    return len(propagated), None

@app.route("/health")
def health():
    conditions, err = get_hierarchy_conditions()
    if err:
        return jsonify({"status": "error", "message": err}), 503

    if conditions["errors"]:
        return jsonify({
            "status": "error",
            "message": f"{len(conditions['errors'])} hierarchy error(s)",
            "errors": conditions["errors"][:5]
        }), 503

    propagated_count, err = check_propagated_objects()
    if err:
        return jsonify({"status": "error", "message": f"Cannot list RoleBindings: {err}"}), 503

    return jsonify({
        "status": "ok",
        "namespaces": conditions["total_ns"],
        "warnings": len(conditions["warnings"]),
        "propagated_rolebindings": propagated_count
    }), 200

@app.route("/tree")
def tree():
    """Return the namespace tree for visibility."""
    result = subprocess.run(
        ["kubectl", "hns", "tree", "--all-namespaces"],
        capture_output=True, text=True, timeout=10
    )
    if result.returncode != 0:
        return jsonify({"status": "error", "message": result.stderr}), 503
    return jsonify({"status": "ok", "tree": result.stdout}), 200

if __name__ == "__main__":
    app.run(host="0.0.0.0", port=8080)

Deploy it:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: hnc-health-bridge
  namespace: monitoring
spec:
  replicas: 1
  selector:
    matchLabels:
      app: hnc-health-bridge
  template:
    metadata:
      labels:
        app: hnc-health-bridge
    spec:
      serviceAccountName: hnc-health-reader
      containers:
        - name: bridge
          image: python:3.12-slim
          command: ["python", "/app/hnc-health-bridge.py"]
          ports:
            - containerPort: 8080
---
# RBAC for the bridge to read HNC resources
apiVersion: v1
kind: ServiceAccount
metadata:
  name: hnc-health-reader
  namespace: monitoring
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: hnc-health-reader
rules:
  - apiGroups: ["hnc.x-k8s.io"]
    resources: ["hierarchyconfigurations", "subnamespaceanchors"]
    verbs: ["get", "list"]
  - apiGroups: ["rbac.authorization.k8s.io"]
    resources: ["rolebindings"]
    verbs: ["list"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: hnc-health-reader
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: hnc-health-reader
subjects:
  - kind: ServiceAccount
    name: hnc-health-reader
    namespace: monitoring

Step 3: Monitor RBAC propagation lag

Propagation lag is the gap between when a policy is set on a parent and when it appears on children. Detect it by checking annotation timestamps:

# Check when a RoleBinding was propagated vs. when the source was created
kubectl get rolebinding -n child-namespace my-role \
  -o jsonpath='{.metadata.annotations}'

# Output should show:
# {"hnc.x-k8s.io/inherited-from":"parent-namespace"}

Add a propagation lag check to your monitoring script:

#!/bin/bash
# check-hnc-propagation.sh
# Verify a specific RoleBinding is propagated to all expected child namespaces

PARENT_NS="${1:-platform}"
BINDING_NAME="${2:-developer-access}"
EXPECTED_CHILDREN=("team-alpha" "team-beta" "team-gamma")

ERRORS=0
for child in "${EXPECTED_CHILDREN[@]}"; do
  if ! kubectl get rolebinding "$BINDING_NAME" -n "$child" \
    --ignore-not-found -o name | grep -q rolebinding; then
    echo "MISSING: $BINDING_NAME not propagated to $child"
    ERRORS=$((ERRORS + 1))
  fi
done

if [ "$ERRORS" -gt 0 ]; then
  echo "PROPAGATION ERRORS: $ERRORS namespace(s) missing RoleBinding"
  exit 1
fi
echo "OK: All child namespaces have propagated RoleBinding"

Step 4: Set up Vigilmon monitors

Monitor 1 — HNC controller liveness:

  1. Log in to vigilmon.online → Add Monitor
  2. Type: HTTP
  3. URL: your HNC metrics endpoint
  4. Keyword: hnc_reconciler_errors_total
  5. Interval: 1 minute

Monitor 2 — Hierarchy health bridge:

  1. Add Monitor → HTTP
  2. URL: https://your-cluster/hnc-health
  3. Expected status: 200
  4. Interval: 2 minutes

Monitor 3 — HNC webhook (admission webhook must stay up):

HNC registers a validating admission webhook. If it goes down, namespace creation fails:

  1. Add Monitor → HTTP
  2. URL: https://hnc-webhook-service.hnc-system.svc:443/validate-hnc-x-k8s-io-v1alpha2-subnamespaceanchor
  3. Expected status: 200 or 400 (healthy — returns 400 for invalid requests, which means it's up)
  4. Interval: 1 minute

Step 5: Alert on propagation failures

# hnc-alerts.yaml
groups:
  - name: hnc-propagation
    rules:
      - alert: HNCReconcilerErrors
        expr: hnc_reconciler_errors_total > 0
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "HNC reconciler errors detected"
          description: "HNC is failing to reconcile namespace hierarchy — propagation may have stopped"

      - alert: HNCObjectPropagationStalled
        expr: rate(hnc_objects_propagated_total[10m]) == 0
        for: 15m
        labels:
          severity: warning
        annotations:
          summary: "HNC object propagation rate is zero"
          description: "No objects propagated in 15m — hierarchy may be static or controller stalled"

      - alert: HNCWebhookDown
        expr: up{job="hnc-webhook"} == 0
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "HNC admission webhook is down"
          description: "Namespace creation and HNC changes will fail until the webhook recovers"

What you're now monitoring

With this setup running:

  • Controller health — HNC reconciler errors trigger alerts within 2 minutes
  • Propagation integrity — RBAC and policy propagation gaps are caught before tenants notice missing access
  • Cycle detection — namespace hierarchy cycles surface as health bridge failures
  • Webhook availability — HNC admission webhook downtime is caught immediately, before kubectl create namespace starts returning 500 errors

Hierarchical namespaces dramatically simplify Kubernetes multi-tenancy — but only when the propagation engine is healthy. Vigilmon makes sure you're always the first to know when it isn't.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles