How to Monitor Google Cloud Firestore Uptime and Health with Vigilmon

Google Cloud Firestore is a serverless, horizontally scaling document database — but "serverless" means the platform manages availability, not that monitoring becomes optional. Firestore imposes per-database write limits (1 write/second per document for sustained throughput), soft quota caps on concurrent listener connections, and region-level outage windows that GCP's own health dashboard has historically under-reported. A Firestore Security Rules misconfiguration can cause your application to receive PERMISSION_DENIED on every write without surfacing in GCP Monitoring. A scheduled Cloud Functions job that exports Firestore collections to Cloud Storage fails silently when the export service account loses a Storage IAM role. And real-time listener connections established via the Firestore SDK drop and reconnect under sustained load in ways that look healthy in the Firebase console but result in stale reads in your application.

Vigilmon gives you external visibility into Firestore health through HTTP probe monitoring and heartbeat monitoring for scheduled export jobs and background sync functions. This tutorial walks through both.

Why Google Cloud Firestore Monitoring Needs More Than GCP Dashboards

GCP Monitoring, the Firebase console, and the Firestore usage dashboard tell you quota consumption and raw read/write counts. They cannot tell you:

• Whether Firestore is reachable from your application servers with a valid service account credential and is accepting authenticated reads and writes
• Whether a Security Rules deployment that went out thirty minutes ago is silently blocking all client writes with PERMISSION_DENIED errors
• Whether the listener connection limit for your project has been reached and new SDK clients are failing to establish real-time sync
• Whether a scheduled Firestore export job to Cloud Storage has stalled due to a revoked IAM role or a Storage bucket permissions error
• Whether a background sync Cloud Function that copies Firestore documents to BigQuery for analytics has stopped running
• Whether a composite index that your critical queries depend on is still active or was deleted by a team member via the Firebase console

These failure modes produce data staleness, silent write drops, and quota-exhaustion errors that appear in application logs but not in GCP health dashboards. External monitoring through Vigilmon catches them by probing the actual read/write paths your application depends on.

Step 1: Build a Google Cloud Firestore Health Endpoint

Firestore is not directly accessible via a raw TCP probe — it speaks gRPC over HTTPS and requires authentication. Deploy a Cloud Run service, Cloud Function, or App Engine endpoint that uses the Firestore Admin SDK to perform an authenticated probe and exposes the result as an HTTP endpoint.

Node.js / Cloud Run Example

// health/firestore.js — deployed as a Cloud Run service
const { Firestore } = require('@google-cloud/firestore');
const express = require('express');

const app = express();
const db = new Firestore({
  projectId: process.env.GCP_PROJECT_ID,
  // When running on Cloud Run, Application Default Credentials are used automatically.
  // For local testing, set GOOGLE_APPLICATION_CREDENTIALS to a service account JSON key.
});

const HEALTH_DOC_PATH = process.env.FIRESTORE_HEALTH_DOC || '_vigilmon/health';

app.get('/health/firestore', async (req, res) => {
  try {
    const docRef = db.doc(HEALTH_DOC_PATH);

    // Write a sentinel document — verifies write path and Security Rules
    const writeStart = Date.now();
    await docRef.set({ checked_at: Firestore.Timestamp.now(), probe: 'vigilmon' });
    const writeMs = Date.now() - writeStart;

    // Read it back — verifies read path and replication consistency
    const readStart = Date.now();
    const snap = await docRef.get();
    const readMs = Date.now() - readStart;

    if (!snap.exists) {
      return res.status(503).json({
        status: 'degraded',
        reason: 'sentinel_document_not_found',
        write_ms: writeMs,
        read_ms: readMs,
      });
    }

    return res.status(200).json({
      status: 'ok',
      write_ms: writeMs,
      read_ms: readMs,
    });
  } catch (err) {
    const code = err.code || 'UNKNOWN';
    // PERMISSION_DENIED means Security Rules are blocking writes
    // UNAVAILABLE means Firestore is experiencing a regional outage
    return res.status(503).json({
      status: 'down',
      error: err.message,
      grpc_code: code,
    });
  }
});

const PORT = process.env.PORT || 8080;
app.listen(PORT, () => console.log(`Firestore health service on :${PORT}`));

Python / Cloud Functions Example

# main.py — deployed as a Cloud Function (HTTP trigger)
import os
import time
import functions_framework
from google.cloud import firestore
from google.api_core.exceptions import GoogleAPIError, PermissionDenied, Unavailable

db = firestore.Client(project=os.environ.get("GCP_PROJECT_ID"))
HEALTH_DOC = os.environ.get("FIRESTORE_HEALTH_DOC", "_vigilmon/health")

@functions_framework.http
def firestore_health(request):
    try:
        doc_ref = db.document(HEALTH_DOC)

        # Write sentinel
        write_start = time.monotonic()
        doc_ref.set({"checked_at": firestore.SERVER_TIMESTAMP, "probe": "vigilmon"})
        write_ms = int((time.monotonic() - write_start) * 1000)

        # Read back
        read_start = time.monotonic()
        snap = doc_ref.get()
        read_ms = int((time.monotonic() - read_start) * 1000)

        if not snap.exists:
            return {"status": "degraded", "reason": "sentinel_not_found",
                    "write_ms": write_ms, "read_ms": read_ms}, 503

        return {"status": "ok", "write_ms": write_ms, "read_ms": read_ms}, 200

    except PermissionDenied as e:
        return {"status": "down", "error": str(e), "reason": "security_rules_blocking"}, 503
    except Unavailable as e:
        return {"status": "down", "error": str(e), "reason": "firestore_regional_outage"}, 503
    except GoogleAPIError as e:
        return {"status": "down", "error": str(e)}, 503

Ensure your Cloud Run service account or Cloud Function identity has the roles/datastore.user IAM role on the project. Verify the endpoint before wiring up Vigilmon:

curl -i https://your-cloud-run-service.a.run.app/health/firestore
# HTTP/1.1 200 OK
# {"status":"ok","write_ms":42,"read_ms":18}

Step 2: Configure Vigilmon HTTP Monitor for Google Cloud Firestore

1. Log in to vigilmon.online and go to Monitors → New Monitor
2. Choose HTTP / HTTPS
3. Set the URL to your health endpoint: https://your-cloud-run-service.a.run.app/health/firestore
4. Set the check interval to 1 minute
5. Under Expected response, configure:
   • Status code: 200
   • Response body contains: "status":"ok"
   • Response time threshold: 5000ms
6. Under Alert channels, assign your Slack or PagerDuty channel
7. Save the monitor

Vigilmon probes from multiple geographic regions simultaneously, requiring multi-region consensus before opening an incident. This eliminates false alarms from transient single-probe network blips and distinguishes genuine Firestore regional outages from local connectivity issues.

Monitoring Write Latency as an Early Warning Signal

Firestore write latency inside the same GCP region should complete in under 100ms under normal load. Elevated write latency is the first observable signal of approaching quota limits or a regional capacity event. Configure a second Vigilmon monitor against the same endpoint with a tighter response time threshold:

• [firestore-primary] /health/firestore — alert at 2000ms, page at 10000ms
• Use Vigilmon's status page grouping to surface both monitors in a single Firestore health panel

Step 3: Heartbeat Monitoring for Firestore Export Jobs and Background Functions

Scheduled Firestore managed exports (gcloud firestore export) that feed your analytics pipeline into BigQuery or Cloud Storage are asynchronous operations initiated via the Firestore Admin API. When the export service account loses the roles/datastore.importExportAdmin role, the export job returns PERMISSION_DENIED on creation and the nightly export silently stops. Background Cloud Functions that sync Firestore documents to other systems fail when the underlying Function invocation times out at the 9-minute Cloud Functions Gen 1 limit without leaving a user-visible trace.

Vigilmon heartbeat monitors detect these silent stalls: your export job or sync function pings Vigilmon on each successful completion. If pings stop arriving within the expected window, Vigilmon fires an alert.

Set Up the Heartbeat Monitor

1. In Vigilmon, go to Monitors → New Monitor → Heartbeat
2. Set the name: firestore-nightly-export (or firestore-bigquery-sync)
3. Set the expected interval to match your job schedule (e.g. 24 hours for a nightly export)
4. Set the grace period: 2 hours
5. Save — copy the unique heartbeat URL, e.g. https://vigilmon.online/heartbeat/abc123xyz

Python Heartbeat Example for Firestore Export Jobs

# firestore_export.py — run as a Cloud Scheduler → Cloud Run job or Cloud Function
import os
import time
import requests
from google.cloud import firestore_admin_v1
from google.api_core.exceptions import GoogleAPIError

PROJECT_ID = os.environ["GCP_PROJECT_ID"]
EXPORT_BUCKET = os.environ["FIRESTORE_EXPORT_BUCKET"]
# e.g. gs://my-firestore-exports
HEARTBEAT_URL = os.environ["VIGILMON_HEARTBEAT_URL"]

def run_firestore_export():
    client = firestore_admin_v1.FirestoreAdminClient()
    database_name = f"projects/{PROJECT_ID}/databases/(default)"

    print(f"Starting Firestore export to {EXPORT_BUCKET}")
    operation = client.export_documents(
        request={
            "name": database_name,
            "output_uri_prefix": EXPORT_BUCKET,
        }
    )

    # Wait for the long-running operation to complete
    print("Waiting for export operation to complete...")
    result = operation.result(timeout=3600)  # 1-hour timeout
    print(f"Export complete: {result.output_uri_prefix}")

    # Ping Vigilmon — export completed successfully
    requests.get(HEARTBEAT_URL, timeout=5)
    print("Heartbeat sent to Vigilmon")

if __name__ == "__main__":
    try:
        run_firestore_export()
    except GoogleAPIError as e:
        print(f"Firestore export failed: {e}")
        # No heartbeat ping — Vigilmon will alert after grace period
        raise

Node.js Heartbeat Example for Firestore → BigQuery Sync Functions

// firestore-bq-sync/index.js — Cloud Function triggered by Cloud Scheduler
const { BigQuery } = require('@google-cloud/bigquery');
const { Firestore } = require('@google-cloud/firestore');
const axios = require('axios');

const db = new Firestore();
const bq = new BigQuery();
const HEARTBEAT_URL = process.env.VIGILMON_HEARTBEAT_URL;
const BQ_DATASET = process.env.BQ_DATASET;
const BQ_TABLE = process.env.BQ_TABLE;

exports.syncFirestoreToBigQuery = async (req, res) => {
  try {
    const snapshot = await db.collection('orders')
      .where('synced_to_bq', '==', false)
      .limit(1000)
      .get();

    if (snapshot.empty) {
      console.log('No documents to sync');
      // Still ping — the job ran successfully even if there was nothing to do
      await axios.get(HEARTBEAT_URL, { timeout: 5000 });
      return res.status(200).json({ status: 'ok', synced: 0 });
    }

    const rows = snapshot.docs.map(doc => ({
      id: doc.id,
      ...doc.data(),
      synced_at: new Date().toISOString(),
    }));

    await bq.dataset(BQ_DATASET).table(BQ_TABLE).insert(rows);
    console.log(`Inserted ${rows.length} rows into BigQuery`);

    // Mark synced in Firestore (batched writes)
    const batch = db.batch();
    snapshot.docs.forEach(doc => batch.update(doc.ref, { synced_to_bq: true }));
    await batch.commit();

    // Ping Vigilmon — sync completed successfully
    await axios.get(HEARTBEAT_URL, { timeout: 5000 });
    return res.status(200).json({ status: 'ok', synced: rows.length });
  } catch (err) {
    console.error('Firestore → BigQuery sync failed:', err);
    // No heartbeat — Vigilmon alerts after grace period expires
    return res.status(500).json({ status: 'error', error: err.message });
  }
};

Step 4: Alert Routing and Response Time Thresholds

Configure alert routing in Vigilmon to match the severity of each Firestore failure mode:

| Monitor | Alert Channel | Priority | |---|---|---| | Firestore health /health/firestore | Slack + PagerDuty | P1 | | Firestore write latency alert | Slack | P2 | | Heartbeat: nightly Firestore export | Slack + email | P2 | | Heartbeat: Firestore → BigQuery sync | Email | P3 |

Set response time thresholds as early warnings before full availability failures:

• Alert at 2000ms for the health endpoint — an authenticated Firestore write + read round-trip in the same region should complete in under 200ms; latency above 2 seconds indicates quota pressure or a regional capacity event
• Alert at 8000ms as a second threshold — signals active quota throttling or a full regional degradation
• Set a separate P1 threshold at 15000ms to page on-call before Vigilmon marks the monitor fully down — gives the team lead-time before users report stale data or failed writes

Summary

Firestore failures surface in subtle ways — Security Rules blocking writes, quota exhaustion throttling, silent export failures, and listener connection drops — long before GCP dashboards show anything actionable. Vigilmon gives you external visibility across the full failure surface:

| Monitor Type | What It Covers | |---|---| | HTTP monitor on /health/firestore | Authenticated write/read round-trip, Security Rules validity, credential health | | Response time thresholds | Quota throttling and regional capacity events as early-warning signals | | Heartbeat monitor: nightly export | Silent export failures due to IAM role changes or quota overruns | | Heartbeat monitor: BigQuery sync | Background sync function timeouts and failed batch writes |

Get started free at vigilmon.online — your first Firestore monitor is running in under two minutes.