How to Monitor Goldpinger with Vigilmon

Goldpinger is a network connectivity monitoring tool for Kubernetes clusters. It runs as a DaemonSet where each pod periodically pings all other Goldpinger pods on every other node and reports whether the connections succeeded. The result is a complete node-to-node connectivity matrix that makes it immediately visible when a specific node loses the ability to reach a subset of other nodes — a class of failure that is invisible to standard Kubernetes health checks but causes Pod evictions, failed DNS lookups, and service mesh connectivity errors.

But Goldpinger itself needs external monitoring. If the Goldpinger DaemonSet becomes unhealthy, your network connectivity visibility disappears exactly when it is most likely to be needed — during a network partition or node failure. In this tutorial you'll set up uptime and response-time monitoring for Goldpinger using Vigilmon — free tier, no credit card required.

Why Goldpinger needs external monitoring

Goldpinger's own health checks miss several failure modes:

• DaemonSet pod crash on a subset of nodes — a node where Goldpinger is not running appears as "not pinged" in the connectivity matrix, but the absence looks identical to a real network failure; you cannot tell whether the node has a network issue or whether Goldpinger itself is missing
• Goldpinger HTTP server down — each pod serves /check, /metrics, and /healthz on port 8080; if this server stops responding, the pod continues to be counted as Running while producing no connectivity data
• Check endpoint returning stale data — Goldpinger pings on a configurable interval; a deadlock in the check loop means the /check endpoint continues returning HTTP 200 with stale results from the last successful ping cycle
• DNS resolution failure inside pods — Goldpinger relies on DNS to resolve peer pod addresses; a CoreDNS issue can cause the connectivity matrix to show all peers as unreachable even when the node network is healthy
• Prometheus scrape missing — if Goldpinger's /metrics stops being scraped (port blocked, Prometheus relabeling change), connectivity alerts based on the goldpinger_peers_response_time_ms metric silently stop firing

External monitoring from Vigilmon watches the Goldpinger HTTP endpoints from outside the cluster, giving you an independent signal that Goldpinger is healthy and producing data.

What you'll need

• Goldpinger deployed as a DaemonSet in your Kubernetes cluster
• The Goldpinger HTTP service reachable externally or via Ingress
• A free Vigilmon account

Step 1: Deploy Goldpinger and expose its endpoints

If Goldpinger is not yet deployed, install it via Helm:

helm repo add okgolove https://okgolove.github.io/helm-charts
helm repo update
helm install goldpinger okgolove/goldpinger \
  --namespace monitoring \
  --create-namespace \
  --set serviceAccount.create=true

Expose the Goldpinger Service for external monitoring:

# goldpinger-ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: goldpinger-ingress
  namespace: monitoring
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /$2
spec:
  ingressClassName: nginx
  rules:
    - host: network.yourdomain.com
      http:
        paths:
          - path: /goldpinger(/|$)(.*)
            pathType: Prefix
            backend:
              service:
                name: goldpinger
                port:
                  number: 8080

kubectl apply -f goldpinger-ingress.yaml

# Verify each endpoint
curl https://network.yourdomain.com/goldpinger/healthz
# OK

curl https://network.yourdomain.com/goldpinger/check | jq '.podResults | length'
# 5  (number of nodes being monitored)

curl https://network.yourdomain.com/goldpinger/metrics | grep goldpinger_peers_response_time
# goldpinger_peers_response_time_ms_bucket{...}

If you prefer a NodePort:

kubectl patch svc goldpinger -n monitoring \
  -p '{"spec":{"type":"NodePort","ports":[{"port":8080,"targetPort":8080,"nodePort":30808}]}}'

Step 2: Understand the Goldpinger check endpoint

The /check endpoint returns a JSON connectivity report that summarizes whether each node can reach all peers:

curl https://network.yourdomain.com/goldpinger/check | jq '.podResults[] | {pod: .podResults.podName, ok: .OK}'

A healthy response looks like:

{
  "podResults": {
    "goldpinger-abc12": {
      "podResults": {
        "goldpinger-def34": {"OK": true, "response-time-ms": 1},
        "goldpinger-ghi56": {"OK": true, "response-time-ms": 2}
      }
    }
  }
}

When any peer shows "OK": false, Goldpinger has detected a real connectivity failure. Configure Vigilmon's body check to match the healthy state.

Step 3: Set up HTTP monitoring in Vigilmon

With your Goldpinger endpoints accessible, add them to Vigilmon:

1. Log in to vigilmon.online and go to Monitors → New Monitor
2. Choose HTTP / HTTPS
3. Add a monitor for each endpoint:

| Monitor name | URL | Expected status | |---|---|---| | Goldpinger health | https://network.yourdomain.com/goldpinger/healthz | 200 | | Goldpinger connectivity check | https://network.yourdomain.com/goldpinger/check | 200 | | Goldpinger metrics | https://network.yourdomain.com/goldpinger/metrics | 200 |

4. Set the check interval to 1 minute
5. For the health monitor, under Expected response, match OK in the response body
6. For the connectivity check monitor, set status code 200 — a non-200 means Goldpinger itself is unhealthy; connectivity failures within the cluster are reported in the JSON body with 200 status
7. For the metrics monitor, optionally match goldpinger_peers_response_time_ms in the response body to confirm peer data is present
8. Save each monitor

Vigilmon probes from multiple geographic regions. If the Goldpinger Ingress is unreachable from outside the cluster, this independently confirms an external connectivity issue beyond what the internal connectivity matrix shows.

Step 4: Monitor the Goldpinger TCP port

Add a TCP-layer monitor to catch port-level failures independently of HTTP content:

1. Go to Monitors → New Monitor
2. Choose TCP Port
3. Enter your Goldpinger hostname and port 8080 (or NodePort 30808)
4. Save the monitor

A TCP failure when Goldpinger pods are Running typically indicates a Service selector label mismatch, a NetworkPolicy restricting external access, or a host-port conflict on the DaemonSet nodes.

Step 5: Configure alert channels

Network connectivity failures in Kubernetes often precede cascading failures — they cause pod evictions, failed health checks, and service mesh errors. Goldpinger should alert before your services do.

Email alerts

1. Go to Alert Channels → Add Channel → Email
2. Enter your network operations or SRE on-call address
3. Assign the channel to all Goldpinger monitors

Webhook alerts for incident routing

1. Go to Alert Channels → Add Channel → Webhook
2. Enter your incident management webhook URL (PagerDuty, Opsgenie, Slack)
3. The payload Vigilmon sends:

{
  "monitor_name": "Goldpinger connectivity check",
  "status": "down",
  "url": "https://network.yourdomain.com/goldpinger/check",
  "started_at": "2024-01-15T10:23:00Z",
  "duration_seconds": 120
}

Wire this alert to a runbook that queries the Goldpinger /check JSON response to identify which specific node-to-node paths have failed, correlates with recent node events, and checks underlying network infrastructure (CNI plugin, cloud VPC routes, BGP peers).

Step 6: Correlate Vigilmon alerts with Goldpinger diagnostics

When you receive a Goldpinger downtime alert, run these checks:

# 1. Check DaemonSet pod status across all nodes
kubectl get pods -n monitoring -l app=goldpinger -o wide

# 2. Check for nodes where Goldpinger is missing
kubectl describe ds goldpinger -n monitoring | grep -E "Desired|Ready|Scheduled"

# 3. Get the full connectivity matrix
kubectl port-forward svc/goldpinger 8080:8080 -n monitoring &
curl -s http://localhost:8080/check | jq '
  .podResults | to_entries[] |
  {
    from: .key,
    failed_to: [.value.podResults | to_entries[] | select(.value.OK == false) | .key]
  } | select(.failed_to | length > 0)'

# 4. Check Goldpinger logs for DNS or connectivity errors
kubectl logs -n monitoring -l app=goldpinger --tail=100 | grep -i "error\|fail\|timeout\|dns"

# 5. Identify which nodes are affected
kubectl get pods -n monitoring -l app=goldpinger -o jsonpath='{range .items[*]}{.metadata.name}{" "}{.spec.nodeName}{"\n"}{end}'

# 6. Check CNI plugin health on affected nodes
kubectl get pods -n kube-system -l k8s-app=calico-node -o wide  # for Calico
# or for Cilium:
kubectl get pods -n kube-system -l k8s-app=cilium -o wide

# 7. Run a manual ping from one Goldpinger pod to another
SOURCE_POD=$(kubectl get pods -n monitoring -l app=goldpinger -o name | head -1 | cut -d/ -f2)
TARGET_IP=$(kubectl get pods -n monitoring -l app=goldpinger -o jsonpath='{.items[1].status.podIP}')
kubectl exec -n monitoring "$SOURCE_POD" -- ping -c 3 "$TARGET_IP"

If Vigilmon shows the /healthz endpoint down but pods are Running, the Goldpinger HTTP server has crashed internally — check the pod logs for a panic or OOM event. If /check returns 200 but the JSON contains failed peers, Goldpinger is healthy but has detected a real network connectivity failure between specific nodes.

Step 7: Create a status page for network connectivity monitoring

Network connectivity is foundational infrastructure. Make Goldpinger's status visible to engineering leadership and service teams without requiring cluster access:

1. Go to Status Pages → New Status Page
2. Name it: "Cluster Network Connectivity"
3. Add your monitors: Goldpinger health, connectivity check, metrics
4. Share the URL with SRE, network operations, and on-call engineers

Summary

| What you set up | What it catches | |---|---| | HTTP monitor on /healthz | Goldpinger HTTP server crash, pod failure | | HTTP monitor on /check | Goldpinger unavailable (connectivity data gaps) | | HTTP monitor on /metrics | Prometheus scrape failures, missing peer data | | TCP monitor on port 8080 | NetworkPolicy blocks, Service selector drift | | Email + webhook alert channels | Immediate notification when network visibility drops | | Status page | Engineering visibility into cluster connectivity health |

Goldpinger provides the node-to-node connectivity matrix that makes otherwise invisible network partitions immediately obvious. External monitoring ensures that Goldpinger itself is healthy and that your network visibility layer does not fail silently exactly when you need it most.

Get started at vigilmon.online — free tier, no credit card required.