Monitoring backup success rates with custom scripts and alerts

A backup job can report success just before the protected data proves unusable. Monitoring backup success rates closes that gap by checking the job result, the age of the recovery point and the evidence required for restoration. It turns a quiet failure into an event with an owner and a deadline.

The practical approach is a monitoring layer that reads scheduler events, backup logs and cloud APIs, then converts inconsistent results into predictable statuses. Scripts should separate completed, degraded, missed and failed runs, calculate rates over useful windows, and raise an alert when a defined recovery service level is at risk.

Why a green job can still hide risk

Most schedulers provide a binary answer: the process finished and returned exit code zero. Backup products may also report warnings for an expired repository, full destination, retryable object failure or zero files processed. A monitor should preserve those signals rather than flatten them into green.

Completion says little about restorability. Pair the job status with newest restore-point age, protected object count, transferred bytes, retention health and the reachability of an off-site copy. A job can update metadata while its encrypted payload or immutable snapshot fails.

Recovery objectives provide the threshold. RPO sets the acceptable age of recoverable data; RTO sets the maximum restoration time and must be proved by testing. For workloads hosted in AWS regions in Sydney and Melbourne, also verify regional API health and intended data-residency boundaries. Cross-region replication improves resilience but adds latency and cost.

Turn logs into a predictable result contract

Define a small contract before writing checks. Every run should emit its workload, job ID, start and finish times in UTC, scheduler status, product status, exit code, files and bytes processed, newest restore point, retention state and repository identifier. Mark a run completed only when every required stage succeeds.

The script can combine product APIs, exported logs and object-store inventory. Prefer structured JSON over scraping human-readable logs, then save the raw response alongside a normalised result and an evidence hash. That gives an alert a durable audit trail and lets a later restore test be tied to the exact run.

Keep unknown as a real state. A timed-out API or missing log means the backup outcome cannot be verified, which is different from success. Synchronise clocks with NTP, store timestamps in UTC and render them locally, including the shift between Australian Eastern Standard Time and Australian Eastern Daylight Time.

Calculate rates that reflect recovery risk

A simple success rate is completed eligible runs divided by expected eligible runs, multiplied by one hundred. Calculate it over 15-minute, daily and weekly windows while counting unobserved jobs as unknown rather than quietly removing them. Generate expected run slots from each workload’s backup policy, not from the jobs that happened to appear.

One overall percentage can still be dangerous. Five successful file servers can conceal a failing database, so report rates by workload and criticality. Pair the percentage with RPO breach age, consecutive failures, duration drift and changes in protected data volume. A 96 per cent rate matters little if the latest good recovery point is already too old.

Dashboards should show current state beside history: last good run, newest restore point, run duration, backup size, repository capacity, failed stage and latest restore-test result. If the monitoring system cannot determine whether a run was expected, publish that gap. An inaccurate denominator creates a comforting metric with no operational value.

Make alerts calm, specific and actionable

Reserve urgent pages for an RPO breach, loss of every immutable copy or repeated failure of a critical workload. Send a warning for one missed window, low repository capacity or a failed off-site verification. Include the workload, region, failed stage, last good restore point, evidence link and a runbook URL.

Group repeated failures into one incident and send a recovery notice when checks pass. Deduplication prevents a scheduler retry from generating five messages, while dependency rules stop a dead repository from paging every workload attached to it. Quiet hours can delay non-urgent warnings, but they should never mute a critical recovery alert.

For Australian managed-service buyers, mixed Office 365, cloud and on-premises estates are common, and monitoring seats may be bundled into contracts priced in Australian dollars. Normalise vendor-specific statuses before routing them. For an iPhone acting as a Perth wallboard during the arvo handover, set brightness manually so afternoon sun cannot hide a critical backup banner; presentation is part of alerting.

Run monitoring scripts like production software

Give each script a dedicated service account with read-only access wherever possible, and keep credentials in a secret store rather than source control. Add timeouts, bounded retries, backoff and a lock file for scheduled runs. Exit non-zero when the monitor cannot collect evidence, so a monitoring outage never presents as a healthy backup.

Store scripts and configuration in version control, review changes and test against a synthetic workload before deployment. After release, run hourly probes against a disposable backup set and compare expected files, object count, restore-point timestamp and alert path. Retain raw results with the script version to make investigations repeatable.

Treat repository capacity, retention changes and network throughput as monitored conditions. A job may finish within its scheduler window yet breach the recovery objective as transferred data grows. Record duration and bytes processed so slowing performance becomes visible before the newest recovery point passes its target.

Verify the whole backup control loop

A final rehearsal should follow one critical workload from scheduled job to verified restoration. I use the same evidence-first approach in independent infrastructure planning, systems architecture and cloud migration work, described on karlkatzke.com.

The job and recovery evidence should include:

• A result for every expected run, with unknown states alerted rather than omitted
• The newest restore point remains within the workload’s RPO
• A tested restoration completes within the defined RTO
• File counts, transferred bytes, retention and repository status agree across independent sources
• An off-site immutable or offline copy remains reachable using separate credentials

The notification path should include:

• Severity based on recovery impact, not merely the backup product’s error code
• A named primary, secondary and after-hours owner for each critical workload
• Incident grouping, deduplication, escalation timing and a clear all-clear notice
• A runbook containing rollback, retry and manual restore steps
• Periodic reporting of success rates, missed runs, RPO breaches and restore-test outcomes

Choose the most critical dataset, wire one custom check into its existing dashboard and alert route today, then rehearse the restoration. Repeating that closed loop for each tiered workload turns backup monitoring from a green indicator into tested recovery confidence.