tutorial

How to Monitor Jenkins X with Vigilmon

Jenkins X is a Kubernetes-native CI/CD platform built on Tekton pipelines, Lighthouse for webhook processing, and automated environment promotion via GitOps....

Jenkins X is a Kubernetes-native CI/CD platform built on Tekton pipelines, Lighthouse for webhook processing, and automated environment promotion via GitOps. Unlike traditional Jenkins, there's no monolithic server to monitor — health is distributed across webhook processors, pipeline pods, and promotion controllers. That makes monitoring more important, not less.

This tutorial covers monitoring Jenkins X with Vigilmon: Lighthouse webhook processor health, Tekton pipeline pod success rates, environment promotion status, and pipeline activity availability.

Why Jenkins X needs external monitoring

Jenkins X is composed of loosely coupled components that can fail independently:

  • Lighthouse webhook processor down — incoming GitHub/GitLab webhooks are dropped silently; PRs stop triggering pipelines
  • Tekton pipeline pod failures — builds fail but no one is paged; queues back up
  • Jx-controller pod crash — environment promotion halts; staging/production environments diverge from Git
  • Pipeline activity API unreachable — jx get activity commands fail; teams lose visibility into build history
  • Chart museum or OCI registry unreachable — Helm chart promotion fails; releases cannot deploy to environments

External monitoring catches these failures because it probes from outside the cluster — the same view your Git provider has when it posts webhooks.

What you'll need

  • Kubernetes cluster with Jenkins X installed (v3.x with Helmfile boot)
  • Lighthouse, Tekton, and at least one environment configured
  • A free Vigilmon account

Step 1: Expose and monitor the Lighthouse webhook endpoint

Lighthouse is your inbound webhook processor. It receives push events and pull request events from your Git provider and triggers Tekton PipelineRuns. If Lighthouse is down, no pipelines start.

# Confirm Lighthouse pods are running
kubectl get pods -n jx -l app=lighthouse-webhooks

# Get the Lighthouse webhook URL
kubectl get ingress -n jx lighthouse-webhooks \
  -o jsonpath='{.spec.rules[0].host}'

Lighthouse exposes a /healthz endpoint:

curl https://lighthouse.example.com/healthz
# {"status":"ok"}

If you're using a private cluster, ensure the Lighthouse Ingress is reachable from Vigilmon's probe locations. For private clusters, consider using Vigilmon's private agent or a VPN endpoint.

Test the webhook endpoint returns 200:

# Simulate a ping event (GitHub sends these on webhook creation)
curl -X POST https://lighthouse.example.com/hook \
  -H "Content-Type: application/json" \
  -H "X-GitHub-Event: ping" \
  -d '{"zen":"Non-blocking is better than blocking."}' \
  -w "\nHTTP Status: %{http_code}\n"

Step 2: Monitor Tekton pipeline pod success rates

Tekton runs pipelines as Pods. A high failure rate in TaskRun pods indicates broken pipelines:

# Count failed TaskRun pods in the last hour
kubectl get pods -n jx \
  --field-selector=status.phase=Failed \
  --sort-by=.metadata.creationTimestamp \
  | tail -20

Write a failure-rate checker:

# tekton_health_server.py
import subprocess, json, time, http.server, threading
from datetime import datetime, timezone, timedelta

CHECK_INTERVAL = 60
FAILURE_THRESHOLD = 5  # > 5 failed pods in last 30 min = unhealthy

state = {"recent_failures": 0, "failed_pods": [], "last_check": 0}

def check_tekton():
    while True:
        try:
            result = subprocess.run(
                ["kubectl", "get", "pods", "-n", "jx",
                 "--field-selector=status.phase=Failed",
                 "-o", "json"],
                capture_output=True, text=True, timeout=15
            )
            pods = json.loads(result.stdout).get("items", [])
            now = datetime.now(timezone.utc)
            cutoff = now - timedelta(minutes=30)

            recent_failures = []
            for pod in pods:
                ts_str = pod["metadata"].get("creationTimestamp", "")
                if ts_str:
                    ts = datetime.fromisoformat(ts_str.replace("Z", "+00:00"))
                    if ts > cutoff:
                        recent_failures.append({
                            "name": pod["metadata"]["name"],
                            "age_minutes": int((now - ts).total_seconds() / 60)
                        })

            state["recent_failures"] = len(recent_failures)
            state["failed_pods"] = recent_failures
            state["last_check"] = time.time()
        except Exception as e:
            state["recent_failures"] = -1
            state["failed_pods"] = [{"error": str(e)}]
        time.sleep(CHECK_INTERVAL)

class Handler(http.server.BaseHTTPRequestHandler):
    def do_GET(self):
        if self.path != "/healthz":
            self.send_response(404); self.end_headers(); return
        body = json.dumps(state).encode()
        unhealthy = (state["recent_failures"] < 0 or
                     state["recent_failures"] >= FAILURE_THRESHOLD)
        self.send_response(503 if unhealthy else 200)
        self.send_header("Content-Type", "application/json")
        self.end_headers()
        self.wfile.write(body)
    def log_message(self, *args): pass

threading.Thread(target=check_tekton, daemon=True).start()
http.server.HTTPServer(("", 9093), Handler).serve_forever()

Deploy as a Deployment in the jx namespace with appropriate RBAC:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: tekton-pod-reader
rules:
  - apiGroups: [""]
    resources: ["pods"]
    verbs: ["get", "list"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: tekton-health-reader
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: tekton-pod-reader
subjects:
  - kind: ServiceAccount
    name: tekton-health-sa
    namespace: jx

Step 3: Monitor environment promotion health

Jenkins X promotes releases through environments (staging → production). The jx-controller-environment pod manages this. Check its health:

# Check environment controller is running
kubectl get pods -n jx -l app=jxl-controller-environment

# Check for stuck environment promotions
kubectl get environments -n jx -o json | \
  jq '.items[] | {name: .metadata.name, status: .status.conditions}'

Expose the environment controller health via its metrics port or a wrapper endpoint. If you use jx promote and see environments stuck in Waiting, the controller is likely down.

Step 4: Monitor the pipeline activity API

The Pipeline Activity API surfaces build history. It's also the backend for jx get activity and the Jenkins X dashboard:

# Check pipeline activity endpoint (in-cluster)
kubectl get svc -n jx | grep activity

# Direct health check against the dashboard
curl -I https://dashboard.jx.example.com/

Add a Vigilmon monitor for the JX dashboard or pipeline activity endpoint so you know immediately when build visibility goes dark.

Step 5: Set up Vigilmon monitors

Monitor 1 — Lighthouse webhook processor

| Field | Value | |---|---| | Type | HTTP | | URL | https://lighthouse.example.com/healthz | | Expected status | 200 | | Keyword check | "status":"ok" | | Interval | 30 seconds | | Alert after | 1 failure |

Monitor 2 — Tekton pipeline failure rate

| Field | Value | |---|---| | Type | HTTP | | URL | http://<tekton-health-IP>:9093/healthz | | Expected status | 200 | | Interval | 60 seconds |

Monitor 3 — JX dashboard / pipeline activity

| Field | Value | |---|---| | Type | HTTP | | URL | https://dashboard.jx.example.com/ | | Expected status | 200 | | Interval | 60 seconds |

Monitor 4 — Lighthouse webhook ingest (TCP)

| Field | Value | |---|---| | Type | TCP | | Host | lighthouse.example.com | | Port | 443 | | Interval | 30 seconds |

This catches cases where HTTPS terminates but the underlying Lighthouse service is unhealthy.

Step 6: Key metrics for Prometheus alerting

Pair Vigilmon with in-cluster Prometheus for defense in depth:

# Lighthouse: webhook events received per minute
rate(lighthouse_webhook_counter_total[5m])

# Tekton: PipelineRun completion rate
rate(tekton_pipelines_controller_pipelinerun_count{status="Succeeded"}[10m])
  / rate(tekton_pipelines_controller_pipelinerun_count[10m])

# Tekton: queue depth (pending PipelineRuns)
tekton_pipelines_controller_workqueue_depth{name="pipeline"}

# Lighthouse: webhook processing errors
rate(lighthouse_webhook_counter_total{status="error"}[5m])

Step 7: Configure escalation policies

Lighthouse down              → P1 immediately (all pipelines stop)
Tekton failure rate > 50%    → P1 (builds are broken)
Environment controller down  → P1 (promotions halted)
JX dashboard unreachable     → P2 (team visibility lost)
Webhook TCP unreachable      → P1 (Git provider can't post events)

Plug these into Vigilmon's alert channels:

Slack:    #ci-alerts
PagerDuty: platform-oncall escalation policy
Email:    devops@example.com

What to monitor — quick reference

| Component | Signal | Vigilmon check | Alert | |---|---|---|---| | Lighthouse | /healthz 200 | HTTP | 1 failure = P1 | | Lighthouse | TCP port 443 | TCP | 1 failure = P1 | | Tekton pods | Failure rate < threshold | Custom HTTP | ≥5 failed/30min | | Env controller | Pod running | Kubernetes + HTTP | Any restart | | JX dashboard | HTTP 200 | HTTP | 2 failures |

Vigilmon for Kubernetes-native CI/CD

Jenkins X distributes CI/CD responsibility across multiple Kubernetes controllers. Any one component failing silently breaks the delivery pipeline — developers push code, nothing happens, and no one knows why until they check kubectl get pods. Vigilmon's external probes give you the first alert because they hit the same webhook URLs your Git provider uses. A 30-second health-check means you know about Lighthouse failures before your first PR sits unreviewed.

Start monitoring at vigilmon.online — free plan, no credit card, your pipeline monitoring is live in minutes.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles