Monitoring Pulumi Kubernetes with Vigilmon: Stack Health, Deployed Service Availability, Resource Drift & SSL Certificate Alerts

Pulumi Kubernetes lets you define, deploy, and manage Kubernetes resources using real programming languages — TypeScript, Python, Go, and .NET — instead of YAML templates. When a Pulumi stack applies successfully but a deployed Deployment's pods fail their readiness checks, your services are down without any Pulumi-side alert. When a stack update partially fails and leaves resources in an inconsistent state, Kubernetes continues serving traffic from the previous revision while your stack shows a failure — a split state that silently breaks new deployments. When the Kubernetes services Pulumi manages become unreachable due to a Service misconfiguration or a lost LoadBalancer IP, the stack itself shows green while users cannot connect. Vigilmon gives you external visibility into what matters: the availability of the services your Pulumi stacks deploy, the health endpoints they expose, and the SSL certificates they terminate — independent of whether the Pulumi state backend is reachable.

What You'll Build

• HTTP monitors on the health endpoints of services deployed by your Pulumi stacks
• SSL certificate monitoring for every domain managed by Pulumi-provisioned ingresses
• TCP monitors on critical Kubernetes service ports to detect LoadBalancer or NodePort failures
• An alerting runbook that maps Pulumi deployment failure modes to Kubernetes inspection commands

Prerequisites

• A Pulumi Kubernetes stack deploying services to a Kubernetes cluster
• At least one service exposed externally (LoadBalancer, Ingress, or NodePort)
• HTTPS configured for production services
• A free account at vigilmon.online

Step 1: Understand Pulumi's Kubernetes Health Surface

Pulumi Kubernetes uses the Kubernetes API to create and update resources. When you run pulumi up, Pulumi waits for Deployments to reach their desired state before marking the update as complete. However, this readiness check happens only during the pulumi up run — after the update completes, Pulumi has no continuous awareness of whether your services stay healthy.

Check the current state of a Pulumi-managed stack's resources:

# View all resources managed by a Pulumi stack
pulumi stack --show-urns

# Check the status of Kubernetes deployments in your stack
pulumi stack output --json

# Inspect deployed Kubernetes resources directly
kubectl get deployments,services,ingresses -n production -l "app.kubernetes.io/managed-by=pulumi"

A Pulumi stack can show Status: Current in pulumi stack ls while the Kubernetes Deployment it manages has 0/3 Ready pods. External monitoring closes this gap — Vigilmon monitors the service endpoint, not the Pulumi state.

The primary health signals for Pulumi Kubernetes deployments are:

1. HTTP response from the application's health endpoint (end-to-end availability)
2. TCP connectivity to the Kubernetes Service's external port
3. SSL certificate validity for Ingress-terminated TLS
4. Individual microservice health endpoints when a stack manages multiple services

Step 2: Monitor the Primary Service Health Endpoint

Every Pulumi-deployed service should have a health endpoint. Monitoring this endpoint validates the full path: Pulumi successfully applied the Deployment, pods passed readiness checks, the Service has healthy endpoints, and the application is responding to requests.

For a typical Pulumi stack deploying a web service:

// Example Pulumi TypeScript — Deployment with a /healthz endpoint
const appDeployment = new k8s.apps.v1.Deployment("app", {
  spec: {
    selector: { matchLabels: { app: "my-service" } },
    template: {
      spec: {
        containers: [{
          name: "app",
          image: "my-org/my-service:latest",
          livenessProbe: {
            httpGet: { path: "/healthz", port: 8080 },
          },
        }],
      },
    },
  },
});

const appService = new k8s.core.v1.Service("app-svc", {
  spec: {
    type: "LoadBalancer",
    selector: { app: "my-service" },
    ports: [{ port: 443, targetPort: 8080 }],
  },
});

Add a Vigilmon monitor for the service:

1. Log in to Vigilmon → Add Monitor → HTTP.
2. URL: https://my-service.example.com/healthz.
3. Check interval: 60 seconds.
4. Response timeout: 10 seconds.
5. Expected status: 200.
6. Keyword: ok or your health response body.
7. Label: my-service (Pulumi stack: production).
8. Click Save.

Add one monitor per service your Pulumi stack deploys. A single Pulumi stack may deploy five microservices — each needs its own monitor, since a partially failed pulumi up can leave individual services broken while others are healthy.

Step 3: Monitor External Service Endpoints via TCP

Pulumi stacks often provision Kubernetes Services of type LoadBalancer. When a cloud load balancer loses its health check configuration, or when the NodePort range is exhausted, the Service accepts TCP connections but returns errors. A TCP-level monitor catches connectivity failures independent of the application layer:

# Get the external IP of a LoadBalancer Service
kubectl get svc my-service -n production \
  -o jsonpath='{.status.loadBalancer.ingress[0].ip}'
# Returns: 203.0.113.55

1. Add Monitor → TCP.
2. Host: my-service.example.com (or the raw LoadBalancer IP).
3. Port: 443.
4. Check interval: 2 minutes.
5. Label: my-service TCP (port 443).
6. Click Save.

TCP vs. HTTP monitors for Pulumi stacks: When a Pulumi update changes a Service's selector labels (a common refactoring mistake), the Service loses its Endpoints — TCP connects but HTTP requests hang indefinitely. A TCP monitor passes while the HTTP monitor times out. Running both gives you the signal to look at kubectl get endpoints my-service -n production.

For stacks managing databases or message brokers alongside web services, add TCP monitors for those ports too:

| Resource | Port | Monitor type | |---|---|---| | PostgreSQL (via ExternalName or NodePort) | 5432 | TCP | | Redis | 6379 | TCP | | NATS / RabbitMQ | 4222 / 5672 | TCP | | gRPC services | 50051 | TCP |

Step 4: Monitor Ingress and SSL Certificates

When a Pulumi stack manages an Ingress resource (with cert-manager annotations or pre-provisioned TLS secrets), certificate rotation and renewal happen in the background. Pulumi does not alert you when cert-manager fails to renew a certificate after the initial deployment:

// Example Pulumi Ingress with cert-manager TLS
const ingress = new k8s.networking.v1.Ingress("app-ingress", {
  metadata: {
    annotations: {
      "cert-manager.io/cluster-issuer": "letsencrypt-prod",
      "nginx.ingress.kubernetes.io/ssl-redirect": "true",
    },
  },
  spec: {
    tls: [{ hosts: ["my-service.example.com"], secretName: "my-service-tls" }],
    rules: [{
      host: "my-service.example.com",
      http: { paths: [{ path: "/", pathType: "Prefix", backend: { service: { name: "my-service", port: { number: 443 } } } }] },
    }],
  },
});

Add SSL certificate monitoring for every host in your Ingress:

1. Add Monitor → SSL Certificate.
2. Domain: my-service.example.com.
3. Alert when expiry is within: 30 days.
4. Alert again: 14 days, 7 days, 3 days, 1 day.
5. Click Save.

cert-manager and Pulumi drift: Pulumi imports cert-manager-issued certificates into its state when you reference them as stack resources. If someone manually deletes the Certificate resource outside of Pulumi, cert-manager may not renew it even though Pulumi still has it in state. The 30-day SSL alert gives you a buffer to detect this before expiry.

Step 5: Monitor Multi-Stack Deployments

Large Pulumi Kubernetes setups use multiple stacks for different environments or service tiers. Monitor each stack's deployed services independently:

# List all stacks in a Pulumi project
pulumi stack ls

# Output:
# production  ✓  Current
# staging     ✓  Current
# feature-x   !  Update failed

A Update failed stack may still have running services from the previous successful update. Monitor the services, not the stack status:

1. Create a monitor group in Vigilmon for each Pulumi stack (e.g., production-stack, staging-stack).
2. Add all service monitors for that stack to the corresponding group.
3. Enable group alerting so a single notification fires when multiple monitors in the same stack fail simultaneously.

Simultaneous failures across all monitors in a group typically indicate:

• A failed pulumi up that corrupted Service or Ingress resources
• A Kubernetes node failure that evicted multiple pods
• A namespace-level NetworkPolicy change that blocked all ingress traffic

Step 6: Configure Alerting

In Vigilmon under Settings → Notifications, configure your alert channels with a Pulumi-aware runbook:

| Monitor | Trigger | Immediate action | |---|---|---| | Service health endpoint | Non-200 or timeout | Check kubectl get pods -n production; run pulumi stack output to see last apply state | | TCP service port | Connection refused | Check kubectl get svc -n production; verify LoadBalancer IP with cloud console | | SSL certificate | < 30 days to expiry | Check cert-manager: kubectl get certificaterequests -A; verify Pulumi Ingress resource | | All stack monitors simultaneously | Multiple alerts at once | Check pulumi up history: pulumi history; look for failed updates |

Alert thresholds: Set service health endpoints to alert after 1 consecutive failure — Pulumi stacks managing production workloads should have zero tolerance for undetected downtime. TCP monitors and SSL alerts can tolerate 2 consecutive failures to reduce false positives during brief network interruptions.

Common Pulumi Kubernetes Failure Modes and What Vigilmon Catches

| Scenario | Vigilmon signal | |---|---| | pulumi up partially fails, Service selector changed | HTTP health check fails; TCP may still pass | | Pod OOM killed after successful deployment | HTTP health check times out or returns 500 | | LoadBalancer IP de-provisioned by cloud provider | TCP monitor fires; all HTTP monitors for that service fire | | cert-manager fails to renew Let's Encrypt certificate | SSL monitor fires at 30-day threshold | | Ingress controller upgrade breaks routing | HTTP monitors fire; TCP monitor on port 443 may still pass | | Namespace deleted and re-created outside Pulumi | All monitors for that namespace fire simultaneously | | ConfigMap change breaks application startup | HTTP health endpoint returns 500 or connection refused | | Resource quota exceeded, pods cannot be scheduled | HTTP monitor fails; kubectl describe quota -n production shows quota breach | | Image pull failure on new deployment | Health endpoint returns 200 from old pods until rollout completes; then fails | | PersistentVolume detach during node migration | Application-level health check fails for stateful services |

Pulumi makes Kubernetes infrastructure feel like application code — but the same principle applies to monitoring: your stack's Status: Current does not mean your services are available right now. Vigilmon monitors the services your Pulumi stacks deploy, not the state of the stacks themselves, giving you real external visibility into whether your Kubernetes workloads are serving traffic, certificates are valid, and critical ports are reachable — independent of whether the Pulumi CLI or state backend is accessible.

Start monitoring your Pulumi Kubernetes deployments in under 5 minutes — register free at vigilmon.online.