Backup systems fail silently. A backup job that exits without producing a complete snapshot, a retention policy that stops pruning and fills the storage volume, an offsite replication schedule that drifts by three days — none of these announce themselves. They are discovered weeks or months later, at the worst possible time: during a restore operation after data loss, or a ransomware incident when you need that clean copy right now.
This guide covers why backup monitoring is a distinct discipline from backup configuration, how to use Vigilmon heartbeat monitoring to detect silent backup job failures, and what else belongs in a complete data protection observability setup.
Why Backup Monitoring Matters
Most engineering teams treat backup as a completed task. They configure the backup job, watch it run a few times, and move on. The assumption is that the job will keep running.
The reality is different:
Silent failures are common. Backup jobs fail in ways that produce no external notification unless you explicitly instrument for failure. A disk quota exceeded partway through a backup produces a partial archive with no error email. A credential rotation that breaks the backup agent's storage access produces a non-zero exit code that nobody is watching. A backup job that depends on a database connection that times out occasionally never completes cleanly but also never triggers an alert because the error threshold isn't configured.
Restore failures are discovered too late. The backup file exists. The restore fails because the archive is corrupted, incomplete, or incompatible with the current software version. This is not an edge case — it is common enough that "verify your backups restore successfully" is standard advice that most teams do not follow consistently.
Ransomware recovery depends on backup integrity. During a ransomware incident, every hour without a verified clean restore point has direct financial consequences. A backup monitoring gap that reveals itself during an incident is not a configuration oversight — it is a business continuity failure.
Regulatory and compliance requirements. Many data protection frameworks (SOC 2, ISO 27001, HIPAA, GDPR) require documented evidence that backup processes run on schedule and complete successfully. Without monitoring and alerting, producing that evidence is a manual process that rarely gets done.
Heartbeat Monitoring for Backup Jobs
The most reliable way to monitor a backup job is to instrument the job itself to signal success, and alert when that signal stops arriving. This is called heartbeat monitoring, and it is exactly what Vigilmon supports.
The mechanism is simple:
- At the end of your backup script, add a single HTTP GET or POST request to your Vigilmon heartbeat URL
- Vigilmon monitors for that ping on the expected schedule
- If the ping does not arrive within the configured window, Vigilmon fires an alert
This inverts the detection model. Instead of checking whether a backup file exists (which is hard to automate without access to the backup destination), the backup job itself reports whether it succeeded.
Example: cron backup job with heartbeat ping
#!/bin/bash
# Nightly database backup with heartbeat monitoring
set -e
TIMESTAMP=$(date +%Y%m%d_%H%M%S)
BACKUP_FILE="/backups/db_${TIMESTAMP}.sql.gz"
# Run the backup
pg_dump myapp_production | gzip > "$BACKUP_FILE"
# Verify the file is non-empty
if [ ! -s "$BACKUP_FILE" ]; then
echo "Backup file is empty — aborting heartbeat ping" >&2
exit 1
fi
# Sync to offsite storage
aws s3 cp "$BACKUP_FILE" "s3://myapp-backups/daily/"
# Signal success to Vigilmon
curl -fsS "https://vigilmon.online/heartbeat/YOUR_MONITOR_TOKEN" > /dev/null
echo "Backup complete: $BACKUP_FILE"
The heartbeat ping only fires if all preceding steps complete without error (set -e exits on any failure before reaching the curl line). If the backup fails at any point — database connection, disk write, S3 upload — the heartbeat is not sent, and Vigilmon alerts within minutes.
Setting the alert window. Configure your Vigilmon heartbeat monitor's expected interval to match your backup schedule with a reasonable buffer. For a nightly backup that typically finishes in 20 minutes, set the heartbeat interval to 24 hours with a 1-hour alert buffer. This gives the job time to complete while catching jobs that fail to run at all.
What Else to Monitor in a Backup Stack
Heartbeat monitoring on the backup job itself is the most important check. But a complete backup monitoring posture includes several additional layers.
Backup storage endpoint availability. If the storage target (S3 bucket, NAS endpoint, backup server) is unreachable, backup jobs may fail silently (depending on error handling) or queue indefinitely. Monitor the storage endpoint URL via HTTP or TCP to get ahead of storage failures before the next backup window.
Restore endpoint or restore API. If your application has a restore workflow — a management API endpoint, a restore service, an admin panel — monitor it independently. A restore endpoint that is down during an incident adds operational latency when you can least afford it.
Offsite replication lag. For tiered backup architectures (local backup + offsite replication), the replication process is a second failure domain. If local backup succeeds but offsite replication stalls, your disaster recovery posture has silently degraded. Add a second heartbeat monitor tied to the replication job completion.
Backup storage volume health. If your backup storage can emit an HTTP endpoint (most NAS devices and cloud storage solutions can be fronted with a simple health check), monitor it for response time and availability. A storage endpoint that is responding slowly is more useful information than learning it's full when the backup job fails.
SSL certificate expiry on backup endpoints. Backup agents that authenticate over HTTPS will fail if the server's certificate expires. Monitor SSL expiry on any backup-related HTTPS endpoints the same way you would for production infrastructure.
Setting Up Vigilmon for Backup Monitoring
Backup monitoring in Vigilmon uses two monitor types:
Heartbeat monitors — for job completion detection. The backup script pings Vigilmon on success; Vigilmon alerts if the ping stops arriving within the expected window. No credentials, no storage access, no external agent required.
HTTP/TCP monitors — for endpoint availability. Point these at your backup storage endpoint, restore API, or offsite replication target. Vigilmon checks these on a 1-minute interval from multiple geographic regions and alerts via multi-region consensus.
Both monitor types are available on the free tier (5 monitors total). A typical backup stack can be covered with 3–4 monitors: one heartbeat for the backup job, one TCP or HTTP check for backup storage, and one for the restore endpoint.
Alerting Strategy for Backup Monitoring
Backup monitoring alerts have a different urgency profile than production uptime alerts. Some recommendations:
Heartbeat miss: high urgency. A backup that fails to complete on schedule is a data protection gap that accumulates over time. Alert immediately and route to someone who can investigate and re-run the job.
Storage endpoint degraded: medium urgency. A slow or intermittently unavailable backup storage endpoint should trigger investigation during business hours, not a 2am page. Configure alert escalation accordingly.
Restore endpoint down: context-dependent. If the restore endpoint is only used during incidents, monitoring it continuously adds assurance. Alert on a non-critical channel unless you are in an active incident.
Offsite replication heartbeat miss: high urgency with delay. Give replication jobs a wider alert window (replication may run on a different schedule than backup) but treat a missed replication heartbeat seriously — it degrades your disaster recovery posture without being immediately visible.
Conclusion
Silent backup failure is one of the highest-consequence, lowest-visibility failure modes in modern infrastructure. The fix is not better backup software — it is monitoring that detects when backup jobs stop completing, storage endpoints become unavailable, and replication lags beyond acceptable windows.
Vigilmon's heartbeat monitoring model is purpose-built for this: your backup script signals success on completion, and Vigilmon alerts if the signal stops. Combined with HTTP/TCP monitors on the backup storage and restore endpoints, you can build complete backup observability with three to four monitors.
Start monitoring your backup jobs for free at vigilmon.online — heartbeat monitors, HTTP/TCP checks, multi-region consensus alerting, Slack notifications. No agent required. No credit card.
Tags: #monitoring #devops #backups #dataprotection #uptime #sre #heartbeat #disasterrecovery