Monitoring Graphite.dev with Vigilmon

Graphite is a developer productivity platform for stacked pull requests — a workflow where developers break large changes into a series of small, sequential PRs that stack on top of each other. Graphite provides a CLI and web dashboard that manages stacked PR dependencies, automatically rebases dependent PRs when a base branch updates, and streamlines the review process for incremental changes.

The self-hosted Graphite deployment runs a Node.js web application backed by PostgreSQL. When it's healthy, developers can visualize their entire stack, request reviews with a single command, and trust that Graphite will keep their stack rebased. When it breaks, stack management operations fail silently, rebase conflicts compound, and PRs get out of sync with each other. Vigilmon gives you the proactive monitoring to catch Graphite failures before your team's stacks become tangled.

What You'll Set Up

• Web application uptime monitor (port 3000)
• PostgreSQL connectivity check
• GitHub API connectivity probe
• Stack rebase automation heartbeat
• SSL certificate expiry alert
• Alert channels with appropriate thresholds

Prerequisites

• Graphite self-hosted deployment with the web application running (default port 3000)
• PostgreSQL running and accessible
• A free Vigilmon account

Step 1: Monitor Web Application Availability (Port 3000)

The Graphite web dashboard is the interface where developers manage their PR stacks, view review queues, and request reviews. If the dashboard is down, all stack management operations are blocked — developers can't visualize their stacks, request reviews, or see where their PRs stand in the review queue.

1. Log in to vigilmon.online and click Add Monitor.
2. Set Type to HTTP / HTTPS.
3. Enter your Graphite URL: https://graphite.yourdomain.com/ (or /health if Graphite exposes a health endpoint).
4. Set Check interval to 1 minute.
5. Set Expected HTTP status to 200.
6. Click Save.

Add a TCP port monitor for a lower-level signal:

1. Click Add Monitor → TCP Port.
2. Enter your Graphite server hostname and port 3000.
3. Set Check interval to 1 minute.
4. Click Save.

If Graphite exposes a health API endpoint, use it instead of the dashboard root:

# Verify your health endpoint returns 200
curl -I https://graphite.yourdomain.com/api/health

Step 2: Monitor PostgreSQL Database Connectivity

Graphite stores all stack relationships, PR metadata, team membership, and review assignments in PostgreSQL. If the database becomes unreachable — due to a connectivity issue, storage exhaustion, or a crashed PostgreSQL process — the Graphite web application will fail to load stack data, serve blank dashboards, or throw 500 errors.

Add a TCP monitor for PostgreSQL:

1. Click Add Monitor → TCP Port.
2. Enter your PostgreSQL server hostname and port 5432.
3. Set Check interval to 1 minute.
4. Click Save.

For a more robust check, add a cron heartbeat driven by a lightweight database probe script:

#!/bin/bash
# /etc/cron.d/graphite-db-heartbeat — runs every 5 minutes
HEARTBEAT="https://vigilmon.online/heartbeat/graphite-db-abc123"

# Attempt a simple query against the Graphite database
if psql "$GRAPHITE_DATABASE_URL" -c "SELECT 1;" > /dev/null 2>&1; then
  curl -s "$HEARTBEAT"
fi

1. In Vigilmon, create a Cron Heartbeat monitor with a 10-minute expected interval.
2. Use the heartbeat URL in the script above.

If PostgreSQL is down or the Graphite database is unreachable, the heartbeat goes silent.

Step 3: Monitor GitHub API Connectivity

Graphite creates PRs, manages branch dependencies, performs rebase operations, and posts stack visualization comments through the GitHub API. Rate limit exhaustion, a stale GitHub token, or GitHub API degradation causes Graphite's rebase automation to fail and stack diagrams to stop updating.

1. Click Add Monitor → HTTP / HTTPS.
2. URL: https://api.github.com/zen.
3. Set Check interval to 5 minutes.
4. Set Expected HTTP status to 200.
5. Click Save.

Monitor GitHub API rate limit headroom by adding a secondary heartbeat:

#!/bin/bash
# /etc/cron.d/graphite-github-ratelimit — runs every 10 minutes
GITHUB_TOKEN="$GRAPHITE_GITHUB_TOKEN"
HEARTBEAT="https://vigilmon.online/heartbeat/github-ratelimit-abc123"

REMAINING=$(curl -s -H "Authorization: token $GITHUB_TOKEN" \
  https://api.github.com/rate_limit | jq '.rate.remaining')

# Ping heartbeat if we have healthy rate limit headroom
if [ "${REMAINING:-0}" -gt 1000 ]; then
  curl -s "$HEARTBEAT"
fi

Set the heartbeat interval to 30 minutes. If rate limit drops below 1000 remaining, the heartbeat goes silent and you get an alert before Graphite starts failing API calls.

Step 4: Monitor Stack Rebase Automation

When a base branch updates (e.g., main gets new commits), Graphite automatically rebases all dependent PRs in the stack. This is one of Graphite's most critical features — without it, stacks drift out of sync and developers get merge conflicts on every PR in the chain.

Rebase jobs run asynchronously in Graphite's background worker. A backed-up rebase queue or a failed job processor means stacks silently go out of sync.

Use a cron heartbeat to verify the rebase job queue is being processed:

1. In Vigilmon, click Add Monitor → Cron Heartbeat.
2. Set the expected ping interval to 30 minutes.
3. Copy the heartbeat URL.

Add a monitoring script that checks Graphite's rebase queue depth:

#!/bin/bash
# /etc/cron.d/graphite-rebase-heartbeat — runs every 10 minutes
GRAPHITE_API="https://graphite.yourdomain.com"
HEARTBEAT="https://vigilmon.online/heartbeat/rebase-abc123"

# Check that the rebase job API endpoint is responsive
STATUS=$(curl -s -o /dev/null -w "%{http_code}" \
  "$GRAPHITE_API/api/rebase/queue/depth")

if [ "$STATUS" = "200" ]; then
  curl -s "$HEARTBEAT"
fi

If Graphite's rebase service stops responding or crashes, the heartbeat goes silent.

Step 5: Monitor Webhook Event Delivery

GitHub delivers PR and push events to Graphite's webhook endpoint. Missed events mean Graphite doesn't know about new commits and won't trigger stack rebase jobs. Unlike API calls that can be retried, missed webhook events require manual re-triggering.

1. Click Add Monitor → HTTP / HTTPS.
2. URL: https://graphite.yourdomain.com/webhook (or your configured webhook path — check your GitHub App settings).
3. Set Check interval to 2 minutes.
4. Set Expected HTTP status to 200 (or 405 if the endpoint rejects unauthenticated GET requests — verify first with curl).
5. Click Save.

The webhook endpoint being reachable confirms that GitHub can successfully deliver events. Combine this with an SSL certificate alert (Step 6) since an expired certificate causes webhook delivery to fail entirely.

Step 6: SSL Certificate Expiry Alert

1. Open the HTTP monitor for the Graphite web application (Step 1).
2. Enable Monitor SSL certificate under the SSL section.
3. Set Alert when certificate expires in less than 21 days.
4. Click Save.

An expired certificate not only breaks HTTPS access for developers using the dashboard, it also causes GitHub's webhook deliveries to fail with a TLS error — stopping all stack rebase automation.

Step 7: Configure Alert Channels

1. Go to Alert Channels in Vigilmon and add Slack, email, or PagerDuty.
2. For the web application monitor: set Consecutive failures before alert to 2. A single blip during a deploy is normal; two consecutive failures means Graphite is genuinely down.
3. For the PostgreSQL TCP monitor: set Consecutive failures before alert to 1. A database that can't be reached is a critical failure.
4. For the GitHub API monitor: set Consecutive failures before alert to 3. Transient GitHub blips are common.
5. For the rebase automation heartbeat: leave the default of 1 missed heartbeat.

Summary

| Monitor | Target | What It Catches | |---|---|---| | Web app HTTP | https://graphite.yourdomain.com/ | Service crash, dashboard unavailable | | TCP port | :3000 | Process down, port not listening | | PostgreSQL TCP | :5432 | Database unreachable | | PostgreSQL heartbeat | Heartbeat URL | Query failure, data corruption | | SSL certificate | Graphite domain | Certificate expiry, webhook TLS failure | | GitHub API | api.github.com/zen | GitHub API reachability | | Rebase automation | Heartbeat URL | Rebase queue stuck, job processor down | | Webhook endpoint | /webhook | Webhook delivery point unreachable |

Graphite makes the stacked PR workflow viable by automating the tedious parts — keeping stacks rebased, posting stack diagrams, routing reviews. When the automation breaks, the stacked PR workflow breaks with it, and teams fall back to manually managing branch dependencies. With Vigilmon watching Graphite's web application, database, GitHub API connectivity, and rebase automation, you get the signal that keeps your stack-based development workflow running smoothly.