Monitoring ytt Kubernetes Deployments with Vigilmon: Deployed Service Health, Overlay Drift, Ingress Availability & SSL Certificate Alerts

ytt (YAML Templating Tool), part of the VMware Tanzu Carvel suite, uses YAML annotations and Starlark scripting to produce Kubernetes configuration that is composable, type-safe, and overlay-aware. Teams use ytt to template Deployment specs, Service configurations, and Ingress rules across environments — but when a ytt overlay silently changes a Service's port, a Deployment's selector, or an Ingress's hostname, Kubernetes applies the change and breaks traffic without any error-level event. When a ytt-rendered ConfigMap changes an application's database URL at next rollout, the pods crash but the previous revision's pods continue serving until they are replaced. When the kapp-controller (often used to apply ytt output) fails to reconcile an App CR, resources drift from their desired state without alerting anyone. Vigilmon gives you external visibility into the services your ytt configurations deploy: continuous health monitoring, SSL certificate expiry tracking, and TCP-level port checks that catch failures whether they come from a ytt overlay change, a kapp-controller reconciliation error, or an out-of-band cluster event.

What You'll Build

• HTTP monitors on the health endpoints of services produced by ytt templates
• SSL certificate monitoring for every hostname in ytt-generated Ingress resources
• TCP monitors on service ports to detect Service selector or port changes from ytt overlays
• An alerting runbook linking ytt/kapp failure modes to Kubernetes inspection commands

Prerequisites

• A Kubernetes cluster receiving configurations generated by ytt (applied via kapp, kapp-controller, or kubectl)
• At least one service exposed externally (Ingress or LoadBalancer)
• HTTPS configured for production services
• A free account at vigilmon.online

Step 1: Understand ytt's Monitoring Surface

ytt is a configuration generator — it produces YAML that is applied to Kubernetes via kapp, kapp-controller, or kubectl. Unlike a running Kubernetes controller, ytt itself does not have a health endpoint or a status API. What you monitor are the Kubernetes resources that ytt's output created.

Check the state of ytt-managed resources:

# If using kapp to apply ytt output:
kapp inspect -a my-app

# Or apply ytt output and inspect with kubectl:
ytt -f config/ | kapp deploy -a my-app -f- --diff-changes

# Check resource status after apply:
kubectl get deployments,services,ingresses -n production -l "kapp.k14s.io/app=my-app"

When using kapp-controller with an App CR:

# Check App CR reconciliation status
kubectl get app my-app -n production -o yaml | grep -A 20 "status:"
# Look for: status.deploy.exitCode: 0 and status.fetch.exitCode: 0

The key insight for monitoring ytt deployments is that the ytt pipeline has no ongoing runtime health — monitoring must target the Kubernetes services that the rendered YAML creates. Common failure modes to monitor for:

1. A ytt overlay changes a Deployment label selector → pods become unmanaged, traffic drops
2. A ytt overlay changes a Service's targetPort → HTTP returns connection refused
3. A ytt overlay updates an Ingress hostname → DNS no longer resolves for the old hostname
4. kapp-controller App CR reconciliation fails → resources drift from desired state

Step 2: Monitor Deployed Service Health Endpoints

The most reliable health signal for a ytt-managed Kubernetes service is an HTTP check against the application's health endpoint. This validates that the ytt configuration produced a working Deployment, that the pods passed readiness checks, and that the Service routes to healthy pod endpoints.

A typical ytt config for a service with a health endpoint:

#@ load("@ytt:data", "data")

apiVersion: apps/v1
kind: Deployment
metadata:
  name: #@ data.values.name
  namespace: #@ data.values.namespace
spec:
  selector:
    matchLabels:
      app: #@ data.values.name
  template:
    spec:
      containers:
        - name: #@ data.values.name
          image: #@ data.values.image
          readinessProbe:
            httpGet:
              path: /health
              port: 8080
            initialDelaySeconds: 5
            periodSeconds: 10
---
apiVersion: v1
kind: Service
metadata:
  name: #@ data.values.name
  namespace: #@ data.values.namespace
spec:
  selector:
    app: #@ data.values.name  #@ overlays on this field can silently break routing
  ports:
    - port: 80
      targetPort: 8080

Add a Vigilmon monitor:

1. Log in to Vigilmon → Add Monitor → HTTP.
2. URL: https://my-service.example.com/health.
3. Check interval: 60 seconds.
4. Response timeout: 10 seconds.
5. Expected status: 200.
6. Keyword: your health response value (e.g., healthy, ok).
7. Label: my-service (ytt config).
8. Click Save.

Add one monitor per ytt-rendered service that is exposed externally. When using ytt overlays across multiple environments, each environment needs its own monitor — a staging ytt overlay might work while a production overlay has a different port or selector.

Step 3: Monitor for Service Selector and Port Changes

ytt overlays are powerful precisely because they can patch any field in a Kubernetes manifest — including fields that silently break routing if changed incorrectly. Monitor TCP-level connectivity to detect these failures:

# Test if the Service is accepting connections at the TCP level
nc -zv my-service.example.com 443

# Compare the Service selector against the Deployment's pod labels:
kubectl get svc my-service -n production -o jsonpath='{.spec.selector}'
kubectl get pods -n production -l app=my-service
# If the label selector changed in a ytt overlay, these won't match

For each externally exposed service:

1. Add Monitor → TCP.
2. Host: my-service.example.com.
3. Port: 443.
4. Check interval: 2 minutes.
5. Label: my-service TCP (port 443).
6. Click Save.

TCP passes but HTTP fails: This pattern typically indicates a ytt overlay changed the application's listening port (containerPort or targetPort) without updating the Service's targetPort. TCP connects to the Service, but the pod is not listening on the expected port. Check: kubectl describe endpoints my-service -n production — if the endpoint's port does not match the container's actual listening port, a ytt overlay misalignment caused the failure.

Step 4: Monitor Ingress Availability

ytt templates frequently manage Ingress resources, parameterizing hostnames and TLS secret references across environments. When a ytt overlay changes an Ingress hostname or removes a TLS block, the Ingress controller silently applies the change — leaving the old hostname unrouted.

A ytt Ingress template with overlay support:

#@ load("@ytt:overlay", "overlay")
#@ load("@ytt:data", "data")

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: my-service-ingress
  namespace: #@ data.values.namespace
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-prod
spec:
  tls:
    - hosts:
        - #@ data.values.hostname
      secretName: #@ data.values.tls_secret
  rules:
    - host: #@ data.values.hostname
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: my-service
                port:
                  number: 80

Monitor the Ingress-routed URL:

1. Add Monitor → HTTP.
2. URL: https://my-service.example.com (the Ingress hostname from data.values.hostname).
3. Check interval: 2 minutes.
4. Response timeout: 15 seconds.
5. Expected status: 200.
6. Label: my-service Ingress (ytt template).
7. Click Save.

Per-environment monitoring: If you render ytt with different data.values.hostname per environment (e.g., my-service.staging.example.com vs. my-service.example.com), create a separate Vigilmon monitor for each hostname.

Step 5: Monitor SSL Certificates for ytt-Managed Ingresses

ytt templates that reference cert-manager annotations or TLS secrets create a dependency on automatic certificate renewal. If the cert-manager ClusterIssuer hits a rate limit, or if a ytt overlay removes the cert-manager annotation and replaces it with a manually managed secret, certificates can expire without any alert.

For each hostname in your ytt-templated Ingress resources:

1. Add Monitor → SSL Certificate.
2. Domain: my-service.example.com.
3. Alert when expiry is within: 30 days.
4. Alert again: 14 days, 7 days, 3 days, 1 day.
5. Click Save.

ytt overlay removes cert-manager annotation: A common accident is a ytt overlay that targets metadata.annotations and accidentally replaces (rather than merges) the existing annotations — removing cert-manager.io/cluster-issuer. The existing certificate continues serving until it expires in 90 days (for Let's Encrypt). The SSL monitor at 30 days catches this window before users see certificate errors.

List all TLS hosts across your ytt-rendered Ingresses to find every domain that needs an SSL monitor:

ytt -f config/ -f values/production.yaml | \
  kubectl neat | \
  yq eval 'select(.kind == "Ingress") | .spec.tls[].hosts[]' -

Step 6: Monitor kapp-Controller App CR Reconciliation

When using kapp-controller to apply ytt-rendered configurations via App CRs, a failed reconciliation leaves resources in a stale state. The App CR status shows the error, but only if you query it:

# Check all App CRs for reconciliation failures
kubectl get apps -A -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name,READY:.status.conditions[0].status,MSG:.status.conditions[0].message'

# Detailed reconciliation status for a specific App:
kubectl describe app my-app -n production

If you expose the kapp-controller metrics externally:

1. Add Monitor → HTTP.
2. URL: https://kapp-controller-metrics.example.com/metrics.
3. Check interval: 5 minutes.
4. Expected status: 200.
5. Label: kapp-controller metrics.
6. Click Save.

Alternatively, set up a Kubernetes CronJob that alerts when any App CR has a non-zero exit code:

apiVersion: batch/v1
kind: CronJob
metadata:
  name: kapp-app-health-check
  namespace: monitoring
spec:
  schedule: "*/10 * * * *"
  jobTemplate:
    spec:
      template:
        spec:
          containers:
            - name: checker
              image: bitnami/kubectl:latest
              command:
                - /bin/sh
                - -c
                - |
                  FAILED=$(kubectl get apps -A -o json | jq '[.items[] | select(.status.deploy.exitCode != 0)] | length')
                  if [ "$FAILED" -gt "0" ]; then
                    echo "ALERT: $FAILED App CR(s) have failed reconciliation"
                    exit 1
                  fi
          restartPolicy: Never

Step 7: Configure Alerting

In Vigilmon under Settings → Notifications, configure alert channels with a ytt-aware runbook:

| Monitor | Trigger | Immediate action | |---|---|---| | Service health endpoint | Non-200 or timeout | Check kubectl get pods -n production; verify ytt overlay did not change selector | | TCP service port | Connection refused | Check kubectl get endpoints my-service -n production; compare port with container | | Ingress HTTP | Non-200 | Check Ingress: kubectl describe ingress -n production; verify hostname matches | | SSL certificate | < 30 days | Check cert-manager: kubectl get certificates -n production; verify ytt overlay did not remove annotation |

Common ytt Kubernetes Failure Modes and What Vigilmon Catches

| Scenario | Vigilmon signal | |---|---| | ytt overlay changes Service selector label | HTTP monitor times out; TCP may pass; kubectl get endpoints shows empty | | ytt overlay changes Service targetPort | TCP passes; HTTP monitor returns connection refused from pod side | | ytt overlay removes Ingress TLS block | HTTP monitor returns redirect to HTTP; SSL monitor fires at 30 days | | kapp-controller App CR reconciliation loop failure | HTTP monitor detects stale pod crash; kapp status shows non-zero exit code | | ytt data.values wrong environment loaded | HTTP monitor fires on unexpected hostname; Ingress routing breaks | | ConfigMap rendered by ytt changes app config | Health endpoint returns 500 after rolling restart | | ytt overlay adds NetworkPolicy that blocks ingress | All HTTP monitors fire simultaneously; TCP may also fail | | Image tag changed in ytt data values → pull failure | Health endpoint fails after rollout starts | | Port mismatch between ytt Service port and Ingress backend port | Ingress returns 502; HTTP monitor fires; TCP may pass | | cert-manager annotation removed by ytt overlay | SSL monitor fires at 30-day threshold; HTTP fails on expiry |

ytt's overlay and templating power makes it easy to manage Kubernetes configuration across many environments — but the same flexibility that lets a single overlay patch a service port across ten environments can silently break routing when an overlay targets the wrong field. Vigilmon monitors your ytt-deployed services from the outside, catching configuration drift, selector mismatches, port changes, and expired certificates the moment they start affecting traffic — not when engineers notice a deployment is broken.

Start monitoring your ytt Kubernetes deployments in under 5 minutes — register free at vigilmon.online.