tutorial

Monitoring JetBrains TeamCity with Vigilmon

TeamCity is a powerful enterprise CI/CD server — but when its server goes unresponsive, build agents disconnect, or the build queue grows unchecked, your entire CI pipeline stalls without any external alert. Here's how to monitor TeamCity availability, agent health, build queue depth, and build execution with Vigilmon.

JetBrains TeamCity is a mature, enterprise-grade CI/CD server widely used for Java, .NET, and polyglot build pipelines. It supports a powerful agent-based architecture where a central server distributes builds to connected build agents. When the TeamCity server itself becomes unresponsive — database connectivity loss, JVM heap exhaustion, long GC pauses — all agents disconnect and builds queue indefinitely. When individual agents disconnect due to network issues, OOM kills, or misconfigured JVM parameters, build capacity silently degrades without the server raising an alert. Vigilmon adds external monitoring for TeamCity's REST API, build agent availability, queue depth, and long-running builds so you catch CI infrastructure failures before your team starts filing "why is my build stuck?" tickets.

What You'll Set Up

  • TeamCity server REST API health monitoring
  • Build agent availability tracking
  • Build queue depth monitoring
  • Long-running and hung build detection
  • SSL certificate monitoring for your TeamCity instance

Prerequisites

  • TeamCity 2023.x+ (on-premise or TeamCity Cloud)
  • TeamCity user account with Project Viewer role for API access
  • Access token for REST API authentication (created under Profile → Access Tokens)
  • A free Vigilmon account

Step 1: Monitor TeamCity Server Availability

TeamCity's web UI and REST API share the same JVM process. When the server runs out of heap space or the database connection pool is exhausted, both the UI and API become unresponsive simultaneously. Add HTTP monitoring as your primary health signal:

  1. In Vigilmon, click Add Monitor → HTTP/HTTPS.
  2. Set the URL to https://teamcity.your-org.internal/app/rest/server.
  3. Set Check interval to 1 minute.
  4. Under Request headers, add Authorization: Bearer YOUR_TEAMCITY_TOKEN.
  5. Under Expected response, add keyword version — the server endpoint always returns version info when healthy.
  6. Set Timeout to 15 seconds.

The /app/rest/server endpoint returns TeamCity version, build number, and node role information. A slow or absent response from this endpoint is the earliest indicator of JVM pressure before the server becomes completely unresponsive.

For environments without HTTPS agent tokens, the login page is a reliable unauthenticated check:

  1. Add an additional HTTP monitor for https://teamcity.your-org.internal/login.html.
  2. Add keyword TeamCity — present in the title of every TeamCity login page.
  3. If the login page fails while the REST API returns 401, the web layer is degraded while the API is still running.

Step 2: Monitor Build Agent Availability

TeamCity distributes builds to connected build agents. When agents disconnect — whether due to network issues, agent JVM crashes, or the agent machine being rebooted — builds queue waiting for compatible agents. In a fleet of 5 agents, losing 2 silently halves your build throughput. Monitor agent availability via the API:

  1. In Vigilmon, click Add Monitor → Cron Heartbeat.
  2. Set Expected ping interval to 5 minutes.
  3. Copy the heartbeat URL (e.g., https://vigilmon.online/heartbeat/abc123).

Create the agent health check script:

#!/bin/bash
# /usr/local/bin/teamcity-agents-check.sh

HEARTBEAT_URL="https://vigilmon.online/heartbeat/abc123"
TC_URL="https://teamcity.your-org.internal"
TC_TOKEN="your-teamcity-access-token"
MIN_AGENTS=2    # Alert if fewer than this many agents are connected and enabled
TIMEOUT=15

# Query TeamCity for connected, enabled build agents
AGENTS_RESP=$(curl -sf \
  --max-time "$TIMEOUT" \
  -H "Authorization: Bearer ${TC_TOKEN}" \
  -H "Accept: application/json" \
  "${TC_URL}/app/rest/agents?locator=connected:true,enabled:true&fields=count,agent(id,name,connected,enabled)" 2>/dev/null)

if [ -z "$AGENTS_RESP" ]; then
  echo "TeamCity API unreachable — cannot check agent count"
  exit 1
fi

AGENT_COUNT=$(echo "$AGENTS_RESP" | python3 -c "
import sys, json
data = json.load(sys.stdin)
print(data.get('count', 0))
" 2>/dev/null || echo 0)

if [ "${AGENT_COUNT:-0}" -ge "$MIN_AGENTS" ]; then
  echo "TeamCity agents OK: ${AGENT_COUNT} connected and enabled"
  curl -s "$HEARTBEAT_URL"
else
  echo "TeamCity agents degraded: only ${AGENT_COUNT} agents available (minimum: ${MIN_AGENTS})"
  exit 1
fi

Install the cron job:

chmod +x /usr/local/bin/teamcity-agents-check.sh
(crontab -l 2>/dev/null; echo "*/5 * * * * /usr/local/bin/teamcity-agents-check.sh >> /var/log/teamcity-health.log 2>&1") | crontab -

TeamCity's own agent connection monitor shows disconnected agents in the UI, but only alerts if you have email notifications configured and are watching the right project. Vigilmon's external monitor catches agent loss regardless of notification configuration.

Step 3: Monitor Build Queue Depth

TeamCity's build queue fills when there are more triggered builds than available agents. A short queue spike during peak hours is normal; a queue that grows continuously during off-peak hours indicates either all agents are disconnected, a runaway trigger (e.g., a misconfigured VCS polling interval), or builds consistently failing to start due to incompatible agent requirements.

#!/bin/bash
# /usr/local/bin/teamcity-queue-check.sh

HEARTBEAT_URL="https://vigilmon.online/heartbeat/def456"
TC_URL="https://teamcity.your-org.internal"
TC_TOKEN="your-teamcity-access-token"
MAX_QUEUE_DEPTH=20    # Alert if more than 20 builds waiting in queue
TIMEOUT=15

# Get current build queue count
QUEUE_RESP=$(curl -sf \
  --max-time "$TIMEOUT" \
  -H "Authorization: Bearer ${TC_TOKEN}" \
  -H "Accept: application/json" \
  "${TC_URL}/app/rest/buildQueue?fields=count" 2>/dev/null)

if [ -z "$QUEUE_RESP" ]; then
  echo "TeamCity API unreachable — cannot check queue"
  exit 1
fi

QUEUE_DEPTH=$(echo "$QUEUE_RESP" | python3 -c "
import sys, json
data = json.load(sys.stdin)
print(data.get('count', 0))
" 2>/dev/null || echo 0)

QUEUE_DEPTH="${QUEUE_DEPTH:-0}"

if [ "$QUEUE_DEPTH" -le "$MAX_QUEUE_DEPTH" ]; then
  echo "TeamCity queue OK: ${QUEUE_DEPTH} builds waiting"
  curl -s "$HEARTBEAT_URL"
else
  echo "TeamCity queue backed up: ${QUEUE_DEPTH} builds waiting (max: ${MAX_QUEUE_DEPTH})"
  exit 1
fi

Run this every 5 minutes. To distinguish between "queue is growing because agents are busy" versus "queue is growing because something is broken," pair this monitor with the agent availability check — if queue depth is high but agent count is normal, it's a capacity issue; if queue depth is high and agent count is low, agents are disconnected.

Step 4: Detect Hung and Long-Running Builds

TeamCity builds can hang when a test step waits indefinitely for a resource, a deployment step connects to an unreachable server, or the build script enters an infinite retry loop. TeamCity has a "build failure conditions" feature that can cancel builds after a configurable timeout — but if this is not configured per build configuration, hung builds run until someone notices.

#!/bin/bash
# /usr/local/bin/teamcity-hung-builds-check.sh

HEARTBEAT_URL="https://vigilmon.online/heartbeat/ghi789"
TC_URL="https://teamcity.your-org.internal"
TC_TOKEN="your-teamcity-access-token"
MAX_RUNNING_MINUTES=90    # Alert if any build runs longer than 90 minutes
TIMEOUT=15

# Get all currently running builds with their start times
RUNNING_BUILDS=$(curl -sf \
  --max-time "$TIMEOUT" \
  -H "Authorization: Bearer ${TC_TOKEN}" \
  -H "Accept: application/json" \
  "${TC_URL}/app/rest/builds?locator=state:running&fields=count,build(id,buildTypeId,startDate,number)" 2>/dev/null)

if [ -z "$RUNNING_BUILDS" ]; then
  echo "Could not fetch running builds — skipping check"
  curl -s "$HEARTBEAT_URL"
  exit 0
fi

LONGEST_RUNNING=$(echo "$RUNNING_BUILDS" | python3 -c "
import sys, json
from datetime import datetime, timezone

data = json.load(sys.stdin)
builds = data.get('build', [])
now = datetime.now(timezone.utc)
max_minutes = 0

for b in builds:
    start = b.get('startDate', '')
    if start:
        try:
            # TeamCity format: 20240101T120000+0000
            started_at = datetime.strptime(start, '%Y%m%dT%H%M%S%z')
            minutes = (now - started_at).total_seconds() / 60
            max_minutes = max(max_minutes, minutes)
        except:
            pass

print(int(max_minutes))
" 2>/dev/null || echo 0)

LONGEST_RUNNING="${LONGEST_RUNNING:-0}"

if [ "$LONGEST_RUNNING" -le "$MAX_RUNNING_MINUTES" ]; then
  echo "TeamCity builds OK: longest running ${LONGEST_RUNNING} min"
  curl -s "$HEARTBEAT_URL"
else
  echo "TeamCity build hung: running for ${LONGEST_RUNNING} min (max: ${MAX_RUNNING_MINUTES} min)"
  exit 1
fi

Set the heartbeat interval to MAX_RUNNING_MINUTES + 10. For teams with builds that legitimately take over an hour (large test suites, slow Docker builds), raise the threshold to reflect your actual P95 build duration plus a 20% buffer.

Step 5: Monitor SSL Certificate Expiry

TeamCity is typically accessed over HTTPS by agents, developers, and CI integrations. An expired SSL certificate causes agents to refuse to connect, webhooks from external services to fail, and the UI to become inaccessible in browsers with strict certificate policies.

  1. In Vigilmon, click Add Monitor → SSL Certificate.
  2. Set the Domain to teamcity.your-org.internal.
  3. Set Alert before expiry to 30 days.
  4. Click Save.

For a TeamCity instance behind a reverse proxy (nginx, Apache), monitor both the external domain and the internal TeamCity certificate if they differ:

#!/bin/bash
# /usr/local/bin/teamcity-ssl-check.sh

HEARTBEAT_URL="https://vigilmon.online/heartbeat/jkl012"
TC_HOST="teamcity.your-org.internal"
TC_PORT="443"
WARN_DAYS=30

EXPIRY_DATE=$(echo | openssl s_client -servername "$TC_HOST" \
  -connect "${TC_HOST}:${TC_PORT}" 2>/dev/null | \
  openssl x509 -noout -enddate 2>/dev/null | \
  cut -d= -f2)

if [ -z "$EXPIRY_DATE" ]; then
  echo "Could not retrieve SSL certificate from ${TC_HOST}:${TC_PORT}"
  exit 1
fi

DAYS_LEFT=$(python3 -c "
from datetime import datetime, timezone
expiry = datetime.strptime('${EXPIRY_DATE}', '%b %d %H:%M:%S %Y %Z').replace(tzinfo=timezone.utc)
now = datetime.now(timezone.utc)
print((expiry - now).days)
" 2>/dev/null || echo -1)

if [ "${DAYS_LEFT:-0}" -ge "$WARN_DAYS" ]; then
  echo "SSL OK: ${TC_HOST} expires in ${DAYS_LEFT} days"
  curl -s "$HEARTBEAT_URL"
else
  echo "SSL expiring: ${TC_HOST} expires in ${DAYS_LEFT} days (threshold: ${WARN_DAYS})"
  exit 1
fi

Step 6: Set Up Alert Channels

Configure Vigilmon to route TeamCity alerts to the right teams:

  1. Go to Alert Channels in Vigilmon.
  2. Add a Slack webhook to #ci-alerts — agent loss and queue backlog affect all developers.
  3. Add PagerDuty for the server availability monitor — a down TeamCity server stops all CI immediately.
  4. Add email for SSL certificate expiry — gives you weeks of lead time.
  5. Set Consecutive failures before alert to 2 for the queue depth monitor — brief spikes during busy commit periods are normal.

Send context-rich Slack alerts:

TC_DASHBOARD="https://teamcity.your-org.internal/agents.html"
ALERT_MSG=":warning: TeamCity: only *${AGENT_COUNT}* build agent(s) connected (minimum: ${MIN_AGENTS}). Check: ${TC_DASHBOARD}"

curl -X POST https://hooks.slack.com/services/YOUR/WEBHOOK/URL \
  -H 'Content-type: application/json' \
  --data "{\"text\": \"$ALERT_MSG\", \"channel\": \"#ci-alerts\"}"

Add maintenance windows during TeamCity upgrades:

curl -X POST https://vigilmon.online/api/maintenance \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -d '{
    "monitor_id": "TEAMCITY_SERVER_MONITOR_ID",
    "duration_minutes": 30,
    "reason": "TeamCity server upgrade"
  }'

Summary

| Monitor | Target | What It Catches | |---|---|---| | HTTP monitor (server API) | /app/rest/server keyword check | JVM heap exhaustion, DB connection loss, server crash | | HTTP monitor (login page) | /login.html keyword check | Web layer failures, plugin initialization errors | | Cron heartbeat (agents) | Connected/enabled agent count | Agent disconnection, OOM kills, network partition | | Cron heartbeat (queue depth) | /app/rest/buildQueue count | Queue backlog, runaway triggers, capacity exhaustion | | Cron heartbeat (hung builds) | Running build duration | Hung test steps, infinite retry loops, deployment timeouts | | SSL certificate monitor | TeamCity domain | Certificate expiry before agents and browsers reject it |

TeamCity's own monitoring features — server health page, agent status panel, email notifications — are helpful but require the TeamCity server itself to be operational to send them. Vigilmon's external monitors catch the cases TeamCity can't self-report: server crashes, database outages, and network-level failures that prevent TeamCity from sending notifications in the first place.

Start monitoring for free at vigilmon.online.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles