tutorial

How to Monitor Open Match Game Matchmaking Infrastructure with Vigilmon

Open Match is an open source, CNCF-hosted game matchmaking framework developed by Google. It decouples matchmaking infrastructure (ticket queuing, Redis stat...

Open Match is an open source, CNCF-hosted game matchmaking framework developed by Google. It decouples matchmaking infrastructure (ticket queuing, Redis state management, match proposal lifecycle) from matchmaking logic (which you write as custom Match Functions) so your game team can iterate on matchmaking rules without touching the core infrastructure.

In production, Open Match is a distributed system with multiple gRPC services — Frontend, Backend, Synchronizer — backed by Redis and sitting in front of your fleet orchestrator (typically Agones). When any component degrades, players either can't enter the matchmaking queue, wait indefinitely for matches, or get matched but never allocated a game server. None of these failures are visible to Kubernetes internal health probes. External monitoring is what catches them.

This tutorial walks you through setting up Vigilmon monitoring for Open Match so you catch Frontend crashes, Redis connectivity loss, Synchronizer failures, and Director allocation bottlenecks before they impact player wait times.

Why Open Match needs external monitoring

Open Match's distributed architecture creates multiple independent failure surfaces:

  • Frontend crash — players can't submit match tickets; queue appears to work from the game client's perspective until it times out
  • Backend crash — the Director can't request matches; existing tickets accumulate in Redis indefinitely
  • Synchronizer failure — Match Functions stop being invoked; tickets age in the queue past SLO without being matched
  • Redis connectivity loss — all ticket operations fail; Frontend returns errors; tickets submitted before the outage are lost
  • Director allocation failure — players are matched but no game server is allocated; players sit on the "found a match" screen forever
  • Ticket expiry spike — MMF criteria too strict or fleet capacity exhausted; tickets expire before being matched, directly impacting player experience

Standard Kubernetes health probes report each pod as alive but can't observe these cross-service failure modes. External monitoring with Vigilmon provides the single source of truth for matchmaking system health.

What you'll need

  • A Kubernetes cluster running Open Match (typically the open-match namespace)
  • Open Match Frontend and Backend gRPC services with HTTP health probes exposed
  • A free Vigilmon account

Step 1: Expose Open Match service health endpoints

Open Match gRPC services expose HTTP health endpoints. Expose them externally via Kubernetes Services:

# open-match-health-services.yaml
apiVersion: v1
kind: Service
metadata:
  name: om-frontend-health
  namespace: open-match
spec:
  type: LoadBalancer
  selector:
    app: open-match-frontend
  ports:
    - name: http
      port: 51500
      targetPort: 51500
      protocol: TCP
---
apiVersion: v1
kind: Service
metadata:
  name: om-backend-health
  namespace: open-match
spec:
  type: LoadBalancer
  selector:
    app: open-match-backend
  ports:
    - name: http
      port: 51505
      targetPort: 51505
      protocol: TCP
---
apiVersion: v1
kind: Service
metadata:
  name: om-synchronizer-health
  namespace: open-match
spec:
  type: LoadBalancer
  selector:
    app: open-match-synchronizer
  ports:
    - name: http
      port: 51506
      targetPort: 51506
      protocol: TCP

kubectl apply -f open-match-health-services.yaml
kubectl get svc -n open-match | grep health
# om-frontend-health       LoadBalancer   10.96.0.10   203.0.113.20   51500:30500/TCP   2m
# om-backend-health        LoadBalancer   10.96.0.11   203.0.113.21   51505:30505/TCP   2m
# om-synchronizer-health   LoadBalancer   10.96.0.12   203.0.113.22   51506:30506/TCP   2m

Verify each service responds:

curl http://203.0.113.20:51500/healthz
# {"status":"SERVING"}
curl http://203.0.113.21:51505/healthz
# {"status":"SERVING"}
curl http://203.0.113.22:51506/healthz
# {"status":"SERVING"}

Step 2: Add Open Match monitors in Vigilmon

Add a monitor for each core Open Match service:

Frontend monitor

  1. Log in to vigilmon.online and go to Monitors → New Monitor
  2. Choose HTTP / HTTPS
  3. URL: http://203.0.113.20:51500/healthz
  4. Name: Open Match Frontend
  5. Interval: 1 minute
  6. Expected status: 200
  7. Response body contains: SERVING
  8. Save

Backend monitor

  1. Monitors → New Monitor → HTTP / HTTPS
  2. URL: http://203.0.113.21:51505/healthz
  3. Name: Open Match Backend
  4. Interval: 1 minute
  5. Expected status: 200, body contains: SERVING
  6. Save

Synchronizer monitor

  1. Monitors → New Monitor → HTTP / HTTPS
  2. URL: http://203.0.113.22:51506/healthz
  3. Name: Open Match Synchronizer
  4. Interval: 1 minute
  5. Expected status: 200, body contains: SERVING
  6. Save

Vigilmon uses multi-region consensus to confirm failures: multiple probes from different geographic regions must all see the failure before an alert fires, eliminating false positives from transient network blips.

Step 3: Monitor Redis connectivity

Open Match stores all ticket state in Redis. Redis connectivity loss is a total matchmaking failure. Add a TCP monitor for Redis so you know about connectivity issues before they cascade into ticket operation failures:

# Expose Redis via NodePort for monitoring
kubectl expose deployment open-match-redis \
  --type=NodePort \
  --name=redis-monitor \
  --port=6379 \
  -n open-match

Or if Redis is already accessible from a node IP:

  1. Monitors → New Monitor → TCP Port
  2. Host: your node IP or Redis LoadBalancer IP
  3. Port: 6379
  4. Name: Open Match Redis
  5. Interval: 30 seconds
  6. Save

For Redis clusters behind a password, add a lightweight Redis health-check proxy:

# redis-health-proxy.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: redis-health-proxy
  namespace: open-match
spec:
  replicas: 1
  selector:
    matchLabels:
      app: redis-health-proxy
  template:
    metadata:
      labels:
        app: redis-health-proxy
    spec:
      containers:
        - name: proxy
          image: redis:7-alpine
          command:
            - /bin/sh
            - -c
            - |
              while true; do
                if redis-cli -h open-match-redis -a "$REDIS_PASSWORD" ping | grep -q PONG; then
                  echo '{"redis":"ok"}' > /tmp/health.json
                else
                  echo '{"redis":"error"}' > /tmp/health.json
                fi
                sleep 10
              done &
              python3 -m http.server 8080 --directory /tmp
          env:
            - name: REDIS_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: open-match-redis-secret
                  key: password
          ports:
            - containerPort: 8080

Then monitor this HTTP endpoint:

  1. Monitors → New Monitor → HTTP / HTTPS
  2. URL: http://<proxy-ip>:8080/health.json
  3. Name: Open Match Redis Connectivity
  4. Response body contains: "redis":"ok"

Step 4: Monitor ticket queue health with a synthetic probe

Ticket queue health — whether the matchmaking pipeline is actually processing tickets end-to-end — requires a synthetic probe that submits a test ticket and measures its age. Add this as a periodic health-check job:

# om-synthetic-probe-cronjob.yaml
apiVersion: batch/v1
kind: CronJob
metadata:
  name: om-synthetic-probe
  namespace: open-match
spec:
  schedule: "*/2 * * * *"
  jobTemplate:
    spec:
      template:
        spec:
          restartPolicy: Never
          containers:
            - name: probe
              image: curlimages/curl:latest
              command:
                - /bin/sh
                - -c
                - |
                  TICKET=$(curl -s -X POST http://open-match-frontend:51504/v1/frontendservice/tickets \
                    -H 'Content-Type: application/json' \
                    -d '{"search_fields":{"tags":["synthetic-probe"]}}')
                  echo "Ticket submitted: $TICKET"
                  TICKET_ID=$(echo $TICKET | grep -o '"id":"[^"]*"' | cut -d'"' -f4)
                  sleep 5
                  curl -s -X DELETE http://open-match-frontend:51504/v1/frontendservice/tickets/$TICKET_ID
                  echo "Probe complete"

Expose the probe result as an HTTP endpoint and monitor it with Vigilmon. This validates the Frontend API is accepting and processing ticket creation requests end-to-end.

Step 5: Configure alert channels

Email alerts

  1. In Vigilmon, go to Alert Channels → Add Channel → Email
  2. Enter your backend/infrastructure team email
  3. Assign to all Open Match monitors

Webhook for Slack or PagerDuty

  1. Alert Channels → Add Channel → Webhook
  2. Enter your Slack webhook URL or PagerDuty Events API endpoint
  3. Assign to all Open Match monitors

When the Open Match Synchronizer goes down, Vigilmon sends:

{
  "monitor_name": "Open Match Synchronizer",
  "status": "down",
  "url": "http://203.0.113.22:51506/healthz",
  "started_at": "2024-06-01T14:10:00Z",
  "duration_seconds": 65
}

Use this to trigger immediate investigation:

# Check Synchronizer pod status
kubectl get pods -n open-match -l app=open-match-synchronizer

# Check events for recent errors
kubectl get events -n open-match --sort-by='.lastTimestamp' | tail -20

# Check Redis is still accessible
kubectl exec -it deploy/open-match-redis -n open-match -- redis-cli ping

# Count tickets stuck in queue
kubectl exec -it deploy/open-match-redis -n open-match -- redis-cli keys "om-ticket:*" | wc -l

Key alert thresholds for Open Match

| What to monitor | Alert condition | |---|---| | Frontend health | Any non-200 / non-SERVING response | | Backend health | Any non-200 / non-SERVING response | | Synchronizer health | Any non-200 / non-SERVING response | | Redis TCP port | Connection refused or timeout | | Redis memory usage | >80% of maxmemory | | Ticket P95 wait time | >60s for fast-paced games, >120s for strategy games | | MMF proposal success rate | <95% (error rate spike) | | Director allocation success | <99% (Agones fleet capacity issue) | | Ticket expiry rate | >10% (fleet capacity or MMF criteria too strict) | | Frontend / Backend CPU | >80% sustained for >5 minutes |

Conclusion

Open Match makes it possible to build scalable, custom matchmaking at game-studio scale without owning the infrastructure. But as a distributed system with multiple gRPC services and a Redis backend, it has several independent failure surfaces — any one of which can silently break the matchmaking pipeline while Kubernetes reports all pods as healthy.

Vigilmon gives you external health monitoring across Frontend, Backend, Synchronizer, and Redis with zero infrastructure overhead. You get alerts in seconds when any component fails, before players start complaining about matchmaking being broken.

Sign up for Vigilmon free — no credit card required, first monitors live in under five minutes.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles