Monitoring kpt Kubernetes Deployments with Vigilmon: Package Health, Live Resource Status, Service Availability & SSL Certificate Alerts

kpt is Google's open-source Kubernetes configuration toolkit for packaging, fetching, displaying, querying, and mutating Kubernetes resources using a declarative, git-native workflow. Teams use kpt to fetch and apply upstream Kubernetes package configurations, and to apply local customizations using KRM (Kubernetes Resource Model) functions. When a kpt live apply run completes with resource errors but exits zero, Kubernetes silently runs the broken configuration. When a kpt function mutates a Service's port or a Deployment's replica count in a way that conflicts with the live cluster state, the reconciliation loop may thrash without alerting anyone. When kpt packages reference container images that are updated upstream and automatically pulled into the cluster, a bad image version can crash all pods without any kpt-side notification. Vigilmon gives you external visibility into the services your kpt configurations deploy: continuous HTTP health monitoring, SSL certificate expiry tracking, and TCP-level connectivity checks that catch failures regardless of whether they originate from a kpt package update, a KRM function mutation, or a cluster-side event.

What You'll Build

• HTTP monitors on health endpoints of services deployed via kpt packages
• SSL certificate monitoring for every Ingress hostname in kpt-managed configurations
• TCP monitors on service ports to detect port changes from kpt package updates
• An alerting runbook that maps kpt deployment failure modes to Kubernetes inspection commands

Prerequisites

• A Kubernetes cluster receiving configurations applied via kpt live apply
• At least one service exposed externally (Ingress or LoadBalancer)
• HTTPS configured for production services
• A free account at vigilmon.online

Step 1: Understand kpt's Monitoring Surface

kpt manages Kubernetes resources through a ResourceGroup inventory (a Kubernetes ConfigMap or CR that tracks which resources belong to a package). After kpt live apply completes, kpt has no continuous runtime presence — it is a CLI tool, not a controller. Monitor the Kubernetes resources the package created, not the kpt process.

Check the status of kpt-managed resources:

# Check the status of all resources in a kpt package
kpt live status ./my-package/

# Inspect resources grouped by the kpt inventory
kubectl get resourcegroups -n production

# List all resources owned by a specific kpt inventory
kubectl get all -n production -l "cli-utils.sigs.k8s.io/inventory-id=my-package-inventory"

When using the kpt status command after an apply:

# Detailed status with conditions
kpt live status ./my-package/ --poll-until current --timeout 60s
# Expected output per resource:
#   NAMESPACE   NAME                    STATUS    CONDITIONS
#   production  Deployment/my-service   Current   -
#   production  Service/my-service      Current   -

A resource showing InProgress or Failed in kpt live status output indicates the Kubernetes resource has not reached its desired state. External monitoring catches what happens after kpt exits: pods being OOM killed, Services losing their endpoints, or Ingresses being overridden by other controllers.

Step 2: Monitor the Primary Service Health Endpoint

Every kpt-deployed service should be monitored via its external health endpoint. This validates the complete stack: the Deployment's pods are running, the Service has healthy pod endpoints, and the application is responding to requests.

A typical kpt package directory:

my-service-package/
├── Kptfile
├── deployment.yaml
├── service.yaml
├── ingress.yaml
└── setters.yaml

With deployment.yaml containing a health endpoint:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-service
  namespace: production  # kpt setter: namespace
spec:
  selector:
    matchLabels:
      app: my-service
  template:
    spec:
      containers:
        - name: my-service
          image: my-org/my-service:v1.2.3  # kpt setter: image-tag
          readinessProbe:
            httpGet:
              path: /healthz
              port: 8080

Add a Vigilmon monitor:

1. Log in to Vigilmon → Add Monitor → HTTP.
2. URL: https://my-service.example.com/healthz.
3. Check interval: 60 seconds.
4. Response timeout: 10 seconds.
5. Expected status: 200.
6. Keyword: ok or your health response body.
7. Label: my-service (kpt package).
8. Click Save.

Add one monitor per kpt package you deploy to production. When a kpt package update fails partway through (some resources updated, others not), monitors on each service detect which services are affected.

Step 3: Monitor Resource Actuation with TCP Checks

kpt's live apply uses server-side apply to push resources into the cluster. When a package update changes a Service's spec.ports field (e.g., a new upstream package version changes the default port), the Service silently accepts the port change — and clients that were working before start getting connection refused.

# Test TCP connectivity to a kpt-managed LoadBalancer service
nc -zv my-service.example.com 443

# Check what port the Service is actually listening on after a package update:
kubectl get svc my-service -n production -o jsonpath='{.spec.ports[*].port}'

1. Add Monitor → TCP.
2. Host: my-service.example.com.
3. Port: 443.
4. Check interval: 2 minutes.
5. Label: my-service TCP (kpt package).
6. Click Save.

Upstream package port drift: kpt packages fetched from upstream repos (via kpt pkg get) can change service ports in new versions. If your team runs kpt pkg update and applies without reviewing the diff, a port change can break connectivity silently. A TCP monitor fires within 2 minutes of the port change taking effect, while the HTTP monitor simultaneously returns connection refused — the combination tells you the problem is at the network/service layer, not the application.

Step 4: Monitor kpt-Managed Ingress Resources

kpt packages for production services typically include Ingress resources. When a kpt package update changes the Ingress host field (a common version-bump mistake in upstream packages) or removes a TLS section, the old hostname stops routing.

A kpt-managed Ingress:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: my-service-ingress
  namespace: production
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-prod
spec:
  tls:
    - hosts:
        - my-service.example.com  # kpt setter: ingress-hostname
      secretName: my-service-tls
  rules:
    - host: my-service.example.com  # kpt setter: ingress-hostname
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: my-service
                port:
                  number: 80

Monitor the Ingress route:

1. Add Monitor → HTTP.
2. URL: https://my-service.example.com.
3. Check interval: 2 minutes.
4. Response timeout: 15 seconds.
5. Expected status: 200.
6. Label: my-service Ingress (kpt package).
7. Click Save.

Post-update monitoring: Run kpt live apply → wait for the health check to confirm it passes before considering the deployment complete. If the monitor fires within 5 minutes of a package update, roll back with kpt live apply --dry-run to inspect what changed.

Step 5: Monitor SSL Certificates for kpt Ingress Packages

kpt packages that include cert-manager annotations rely on automatic certificate renewal. A kpt package update that changes the cert-manager annotation value, or that replaces a cert-manager-managed secret with a manually created one, can cause certificate expiry to go undetected.

For each Ingress hostname in your kpt packages:

1. Add Monitor → SSL Certificate.
2. Domain: my-service.example.com.
3. Alert when expiry is within: 30 days.
4. Alert again: 14 days, 7 days, 3 days, 1 day.
5. Click Save.

List all Ingress hostnames across your kpt packages to find every domain needing an SSL monitor:

# Find all Ingress hosts in the local kpt package directory
grep -r "host:" ./my-packages/ | grep -v "#" | awk '{print $2}' | sort -u

# Or query the live cluster for kpt-managed Ingress hosts:
kubectl get ingress -A -l "cli-utils.sigs.k8s.io/inventory-id" \
  -o jsonpath='{range .items[*]}{.spec.rules[*].host}{"\n"}{end}'

kpt package update and cert-manager: If an upstream kpt package version removes the cert-manager.io/cluster-issuer annotation (e.g., a package author switches to pre-provisioned certs), and your team applies the update without a diff review, the existing certificate stops auto-renewing. The 30-day SSL alert gives you enough time to catch this before users see certificate errors.

Step 6: Monitor the ResourceGroup Inventory

kpt tracks its managed resources via a ResourceGroup inventory object. If this object is deleted or corrupted (e.g., by a namespace deletion outside of kpt), kpt loses track of the owned resources and subsequent kpt live apply runs may leave orphaned resources or fail to prune deleted ones.

# Check for ResourceGroup inventory objects
kubectl get resourcegroups -A

# Verify the inventory matches expected resource count
kpt live status ./my-package/ | wc -l

Set up a monitoring CronJob to verify inventory health:

apiVersion: batch/v1
kind: CronJob
metadata:
  name: kpt-inventory-health-check
  namespace: monitoring
spec:
  schedule: "*/15 * * * *"
  jobTemplate:
    spec:
      template:
        spec:
          containers:
            - name: checker
              image: bitnami/kubectl:latest
              command:
                - /bin/sh
                - -c
                - |
                  GROUPS=$(kubectl get resourcegroups -A -o json | jq '.items | length')
                  if [ "$GROUPS" -eq "0" ]; then
                    echo "ALERT: No kpt ResourceGroup inventories found"
                    exit 1
                  fi
          restartPolicy: Never

Step 7: Configure Alerting

In Vigilmon under Settings → Notifications, configure your alert channels with a kpt-aware runbook:

| Monitor | Trigger | Immediate action | |---|---|---| | Service health endpoint | Non-200 or timeout | Check kpt live status ./my-package/; run kubectl get pods -n production | | TCP service port | Connection refused | Check kubectl get svc -n production; review port diff from last kpt pkg update | | Ingress HTTP | Non-200 or 404 | Check kubectl describe ingress -n production; verify host field unchanged | | SSL certificate | < 30 days | Check cert-manager: kubectl get certificates -n production; verify kpt package annotation |

Alert grouping: Create a monitor group per kpt package (e.g., my-service-package). When all monitors in a package group fire simultaneously, the likely cause is a failed kpt live apply that broke all resources in the package, or a namespace-level event (quota exhaustion, node failure) affecting all package resources.

Common kpt Failure Modes and What Vigilmon Catches

| Scenario | Vigilmon signal | |---|---| | kpt pkg update changes Service targetPort | TCP passes; HTTP monitor returns connection refused | | Upstream package changes Ingress host field | HTTP monitor fires; SSL monitor may also fire if hostname changed | | kpt live apply fails mid-run (partial apply) | Some monitors fire; others pass — indicates partial resource update | | ResourceGroup inventory object deleted | No kpt-side alert; services continue running from last apply until next apply fails | | Image tag setter updated to broken image | Health endpoint fails after rollout; pods in ImagePullBackOff | | kpt function mutates NetworkPolicy to block ingress | All HTTP monitors fire simultaneously | | Namespace resource quota exceeded, pods not scheduled | HTTP monitor fails; kubectl describe quota -n production shows breach | | cert-manager annotation removed in upstream package | SSL monitor fires at 30-day threshold | | Service port removed from package YAML (pruning) | TCP monitor fires; HTTP returns connection refused | | kpt setter applied wrong value (e.g., staging hostname in prod) | HTTP monitor fires for wrong hostname; DNS does not resolve |

kpt's git-native, package-based approach to Kubernetes configuration makes it easy to pull in upstream best practices and customize for your environment — but the same mechanism that lets a single kpt pkg update refresh 50 resources can silently break service routing, change TLS configuration, or introduce port mismatches across your production cluster. Vigilmon monitors the services your kpt packages deploy from the outside, giving you real-time visibility into health endpoint responses, TCP connectivity, Ingress routing, and SSL certificate expiry — alerting you the moment a package update breaks something, before users notice.

Start monitoring your kpt Kubernetes deployments in under 5 minutes — register free at vigilmon.online.