How to Monitor Perses with Vigilmon (Dashboard Server, Datasource & GitOps Health)

Perses is a CNCF sandbox project that brings open, GitOps-friendly dashboarding to the Prometheus ecosystem. Unlike Grafana, Perses stores every dashboard as a versioned Kubernetes CRD or plain JSON file — which means your dashboards live in Git, get reviewed in pull requests, and can be deployed with Argo CD or Flux just like any other workload.

That model is powerful, but it also introduces failure points that only external monitoring can reliably catch: the Perses API server going down, datasource connectivity breaking silently, or a GitOps sync stalling without anyone noticing. In this tutorial you'll add comprehensive uptime and endpoint monitoring for your Perses installation using Vigilmon — free tier, no credit card required.

Why Perses needs external monitoring

Perses runs as a server process that exposes a REST API consumed by the UI and by CI/CD pipelines. When it's unhealthy, the failure modes look like this:

• API server crash — the dashboard UI shows a blank screen or 502; no internal alerting fires because there is no internal alerting
• Datasource connectivity lost — panels fail to render but the server itself reports healthy; users see empty charts with cryptic "query failed" messages
• GitOps sync stalls — a network partition or RBAC change stops the sync controller from reconciling new dashboards; the server stays up while your config drifts silently
• Panel rendering degraded — a slow Prometheus instance causes query timeouts that cascade into slow page loads; without latency monitoring you won't notice until users complain

External monitoring from Vigilmon adds the layer that Perses itself cannot provide: end-to-end reachability from outside your cluster, tracked over time with alerting.

What you'll need

• A running Perses server (self-hosted via Docker, Kubernetes, or binary)
• A free Vigilmon account
• (Optional) Perses CLI (percli) installed locally for the GitOps sync check

Step 1: Verify the Perses health endpoint

Perses exposes a health endpoint out of the box at /api/v1/health. Confirm it's reachable:

curl -s http://localhost:8080/api/v1/health | jq .

A healthy response looks like:

{
  "buildTime": "2024-09-12T14:32:00Z",
  "version": "0.8.0",
  "status": "ok"
}

If your Perses instance sits behind an ingress or reverse proxy, use the public URL instead of localhost. Note that URL — you'll add it to Vigilmon in the next step.

Step 2: Add the Perses health check to Vigilmon

1. Log in to Vigilmon and click Add Monitor.
2. Choose HTTP(S) as the monitor type.
3. Enter your Perses health URL: https://perses.your-domain.com/api/v1/health
4. Set the check interval to 60 seconds (or 30 seconds if you want tighter coverage).
5. Under Expected Status Code, enter 200.
6. Under Response Body Contains, enter "status":"ok" — this ensures Vigilmon only considers the check passing when Perses actually reports healthy, not just when the reverse proxy returns 200.
7. Click Save.

Vigilmon will now ping your Perses server every minute from multiple geographic regions. If the server goes down or the health endpoint starts returning errors, you'll receive an alert within two minutes.

Step 3: Monitor datasource connectivity

Perses validates datasource connectivity via its /api/v1/globaldatasources and per-project /api/v1/projects/{project}/datasources endpoints. You can expose a simple health aggregator with a small script, or use Perses's built-in datasource probe endpoint if your version supports it.

Here's a lightweight Node.js sidecar that checks each configured datasource and exposes a /datasource-health endpoint:

// datasource-health-check.js
import http from "http";
import { execSync } from "child_process";

const PERSES_URL = process.env.PERSES_URL || "http://localhost:8080";
const PORT = process.env.PORT || 9090;

async function checkDatasources() {
  const resp = await fetch(`${PERSES_URL}/api/v1/globaldatasources`);
  if (!resp.ok) throw new Error(`Perses API returned ${resp.status}`);
  const datasources = await resp.json();

  const results = await Promise.allSettled(
    datasources.map(async (ds) => {
      const probe = await fetch(
        `${PERSES_URL}/api/v1/proxy/globaldatasources/${ds.metadata.name}/api/v1/query?query=up`
      );
      return { name: ds.metadata.name, ok: probe.ok, status: probe.status };
    })
  );

  const failed = results
    .filter((r) => r.status === "rejected" || !r.value?.ok)
    .map((r) => r.reason?.message || r.value?.name);

  return { total: datasources.length, failed };
}

http
  .createServer(async (req, res) => {
    if (req.url !== "/datasource-health") {
      res.writeHead(404);
      res.end();
      return;
    }
    try {
      const { total, failed } = await checkDatasources();
      if (failed.length > 0) {
        res.writeHead(503, { "Content-Type": "application/json" });
        res.end(
          JSON.stringify({ status: "degraded", total, failed })
        );
      } else {
        res.writeHead(200, { "Content-Type": "application/json" });
        res.end(JSON.stringify({ status: "ok", total, failed: [] }));
      }
    } catch (err) {
      res.writeHead(503, { "Content-Type": "application/json" });
      res.end(JSON.stringify({ status: "error", error: err.message }));
    }
  })
  .listen(PORT, () => console.log(`Datasource health check listening on :${PORT}`));

Deploy this sidecar alongside your Perses server, then add a second Vigilmon monitor:

• URL: https://perses.your-domain.com/datasource-health (or the sidecar's exposed URL)
• Expected status: 200
• Response body contains: "status":"ok"

Step 4: Monitor panel rendering performance

Slow Prometheus queries are the most common cause of degraded Perses dashboards. You can detect this before users notice by checking the query latency on a known-fast query:

#!/bin/bash
# panel-latency-check.sh — exit non-zero if query takes more than 2 seconds

START=$(date +%s%3N)
STATUS=$(curl -s -o /dev/null -w "%{http_code}" \
  "http://localhost:8080/api/v1/proxy/globaldatasources/prometheus/api/v1/query?query=up")
END=$(date +%s%3N)
LATENCY=$((END - START))

if [ "$STATUS" != "200" ]; then
  echo "FAIL: datasource returned HTTP $STATUS"
  exit 1
fi

if [ "$LATENCY" -gt 2000 ]; then
  echo "SLOW: query took ${LATENCY}ms (threshold 2000ms)"
  exit 1
fi

echo "OK: ${LATENCY}ms"
exit 0

Expose this script via a small HTTP wrapper (or use the sidecar pattern above) and add a Vigilmon monitor for the latency check endpoint. Set the response body check to "OK:" so that slow or failed queries immediately trigger an alert.

Step 5: Monitor GitOps sync status

If you're using Perses with a GitOps sync controller (e.g., the official perses-operator or a custom Argo CD app), you need to know when syncs stall. The simplest approach is to check whether the operator's last-sync timestamp is recent:

# perses-sync-health.yaml — deploy as a CronJob that touches a status endpoint
apiVersion: batch/v1
kind: CronJob
metadata:
  name: perses-sync-probe
  namespace: monitoring
spec:
  schedule: "*/5 * * * *"
  jobTemplate:
    spec:
      template:
        spec:
          restartPolicy: Never
          containers:
            - name: probe
              image: curlimages/curl:8.5.0
              command:
                - /bin/sh
                - -c
                - |
                  # Check that the operator deployment rolled out recently
                  LAST_SYNC=$(kubectl get configmap perses-sync-status -n monitoring \
                    -o jsonpath='{.data.lastSync}' 2>/dev/null || echo "0")
                  NOW=$(date +%s)
                  AGE=$((NOW - LAST_SYNC))
                  if [ "$AGE" -gt 600 ]; then
                    # Update a status endpoint that Vigilmon probes
                    curl -sf -X POST http://perses-sync-status-svc/stale
                  else
                    curl -sf -X POST http://perses-sync-status-svc/healthy
                  fi

Add a Vigilmon HTTP monitor for your sync-status service. When a sync hasn't completed in more than 10 minutes, the status endpoint returns 503 and Vigilmon fires an alert.

Step 6: Set up alert routing in Vigilmon

With all four monitors in place — server health, datasource connectivity, panel rendering latency, and GitOps sync — configure Vigilmon to route alerts where your team actually looks:

1. In the Vigilmon dashboard, click Alert Channels.
2. Add a Slack channel for your #platform-alerts Slack channel (paste the incoming webhook URL).
3. Add an Email channel for your on-call distribution list.
4. Back in each monitor's settings, under Alert Channels, select both channels.
5. Set Alert after N failures to 2 to avoid false positives from transient network blips.

What you're monitoring now

| Monitor | What it catches | |---|---| | /api/v1/health | Perses server crash, OOM kill, port binding failure | | /datasource-health | Prometheus or Thanos unreachable, auth expiry | | Panel latency endpoint | Slow queries, upstream overload | | GitOps sync status | Operator stuck, RBAC change, network partition |

Keeping dashboards healthy long-term

A few Perses-specific operational practices pair well with this monitoring setup:

• Pin your Prometheus version alongside Perses releases — the proxy API shape changes between major Prometheus versions and can silently break datasource connectivity.
• Use percli lint in CI before merging dashboard PRs — invalid dashboard JSON that passes schema validation can still cause render failures at runtime.
• Set a dashboard TTL alert — if you use GitOps, alert when a dashboard hasn't been reconciled within two sync cycles. Stale dashboards are usually the first sign of a broken sync pipeline.

External monitoring with Vigilmon gives you the ground truth about whether Perses is working from your users' perspective — something no internal health check can replicate.

Start monitoring your Perses instance with Vigilmon →