Automating backups to AWS S3 with rclone and cron

A scheduled rclone-to-S3 workflow turns a fragile reminder into a repeatable control. The job should capture defined data, protect it with least-privilege credentials, report every outcome, and be tested by restoring an earlier copy. Success means an operator can answer three questions quickly: what ran, what reached S3, and whether the data can be recovered.

This matters in Australia, where many small managed service providers and in-house IT teams support demanding clients without a dedicated backup engineer. Australian teams often value plain, dependable runbooks—fair enough to call a boring nightly job “no dramas”. Workloads may sit in Melbourne, Brisbane, Canberra, or a regional office, yet the operating standard should remain the same. Local habits change the timetable, not the need for recoverable data.

AWS S3 separates backup storage from the source host, but replication alone does not create a sound backup strategy. Rclone must copy consistent data, while S3 versioning, encryption, lifecycle rules, and IAM permissions provide the surrounding controls. Cron supplies the schedule; monitoring and restoration tests supply confidence.

A workable design is small enough to explain in a handover note and strict enough to leave little room for guesswork. This approach suits a Linux server, NAS, or small virtual machine, and it can accommodate Australian data-residency needs by selecting a nearby AWS region such as Sydney or Melbourne.

Establish a safe S3 destination

Create a dedicated bucket rather than sharing a general-purpose prefix. Choose Sydney (ap-southeast-2) or Melbourne (ap-southeast-4) when proximity, latency, and data location favour it; a Brisbane workload may still fit best in Sydney. Enable versioning and server-side encryption, then give each source host its own path. Object Lock can add tamper resistance, but its retention settings require deliberate compliance and recovery planning.

Prefer an EC2 instance profile or another workload identity over long-lived access keys. Scope the IAM policy to the required bucket and prefix, including only the S3 actions the transfer process needs. On a NAS or home server, use a restricted IAM user with rotated keys if a role is impractical. Rclone’s configuration can contain secrets, so encrypt it with rclone config password and keep the file readable only by the backup account.

Turn on public-access blocks, require TLS, and use lifecycle rules to expire old versions and abandoned multipart uploads. Review storage, request, and data-transfer charges as part of the runbook. Where personal information is involved, check that the selected location and any replication arrangement align with the Australian Privacy Principles rather than assuming cloud storage is automatically compliant.

Configure rclone for predictable copies

Install a current rclone release and run rclone config to create a remote named s3backup. Select the Amazon S3 provider, set the matching region, and let the AWS credential chain use the attached role. For static credentials, store them in the encrypted rclone configuration rather than directly in a cron command. Test the remote with rclone lsd s3backup: before scheduling any transfer.

A basic source-to-S3 copy might look like this:

rclone copy /srv/app-data s3backup:company-backups/app-data \
  --transfers 4 --checkers 8 --log-level INFO

Prefer copy to sync for an initial backup. sync can remove destination objects when they are absent from the source, which is undesirable if an accidental deletion or ransomware event has already reached the machine being protected. Use --dry-run first, review the proposed transfers, and inspect errors rather than assuming a zero exit status on every schedule. For databases, create a transaction-consistent dump or snapshot before rclone reads it; copying live database files can produce a backup that cannot restore cleanly.

Schedule the job without timing surprises

Put the workflow in a script under /usr/local/sbin, use absolute paths, and make the script fail when rclone fails. A lock prevents a slow transfer from overlapping the next run. A typical root crontab entry is:

15 02 * * * /usr/bin/flock -n /var/lock/s3-backup.lock /usr/local/sbin/backup-to-s3 >> /var/log/s3-backup-cron.log 2>&1

Cron follows the host’s configured timezone. Sydney, Melbourne, Canberra, Hobart, and Adelaide change their clocks at daylight-saving boundaries; Brisbane does not, and Perth uses Western Standard Time without daylight saving. Test the chosen wall-clock time around the annual clock change because cron implementations can handle a skipped or repeated hour differently. If exact UTC scheduling is essential, use a systemd timer with an explicit UTC calendar instead.

Remote teams, including K9 search-and-rescue volunteers, should also plan for connectivity loss without treating it as a successful backup. The practical field operations guidance offers a useful reminder to rehearse procedures where power and networks are unreliable. For rclone, that means retrying safely, retaining logs locally, and sending a clear failure signal rather than repeatedly hammering S3 without limit.

Prove that backups can be restored

Send rclone logs and the script’s exit code to a monitored destination, such as CloudWatch Logs, a syslog collector, or the monitoring system already used by the organisation. Alert on a missing run as well as an explicit error; silence often means cron never started the script. Keep logs long enough to investigate a weekly cycle, and include the source path, destination prefix, transfer count, and completion time in each run record.

Reliable infrastructure matters when the backup target sits at the end of a field link. My notes on a portable solar charging station show how power, communications, dust, and Australian heat can affect operations beyond an office. The same discipline applies to backups: test them on the actual service path, with realistic bandwidth and outages, rather than only from the server rack.

Treat restoration as the acceptance test. On a schedule, restore a representative file, configuration set, and database export into a clean environment, then compare checksums and application behaviour. Keep earlier S3 versions long enough to recover from delayed or unnoticed corruption. A green “backup completed” message is evidence of transfer, not evidence of recoverability.

Operational checklist for dependable backups

Use this checklist when implementing or reviewing the workflow:

  • Choose an AWS region that balances Australian latency, data residency, request cost, and the location of the protected system.
  • Grant the workload only the IAM permissions and S3 prefix required for its backups, and prefer temporary role credentials.
  • Encrypt the rclone configuration, run a dry-run transfer, and capture any errors before enabling the schedule.
  • Back up application-consistent data, especially database exports, rather than live files that may change during transfer.
  • Run cron with absolute paths, a non-overlapping lock, local logs, and an alert for missed or failed runs.
  • Review S3 versioning, lifecycle expiration, storage costs, and transfer charges so retention does not create unexpected bills.
  • Restore and verify sample data regularly, recording the operator, timestamp, source version, and result.

For a broader infrastructure assessment, my information technology consulting work covers the architecture decisions around cloud migration, backup targets, monitoring, and operational ownership. The practical result should fit the client’s existing support model rather than create another specialist system that only one administrator understands.

Create the bucket, identity, and rclone remote in a non-production account today. Then run one supervised transfer, restore a sample, and enable the scheduled job only after those checks pass.

Experience

Information Technology Consulting

Independent Practice

Provides IT consulting services focused on infrastructure planning, cloud migration strategy, and systems architecture. Engagements draw on years of hands-on sysadmin and development experience across Linux, Windows, and hybrid environments.

K9 Search & Rescue Volunteer

Ongoing

Active participant in K9 Search & Rescue operations, combining technical logistics skills with field support for canine search teams.

Karl Katzke's Blog

October 2006 – May 2014

Published a long-running personal technology blog covering cloud vs. in-house infrastructure, F# and Mono on OSX, hardware vendor critiques, RAID card performance analysis, and sysadmin storytelling. Notable posts include "When Sysadmins Ruled the Earth" (May 15, 2014) and "Getting Started with F# and Mono on OSX" (December 22, 2012).

Credentials

A small badge icon with a shield shape in muted blue tones on a light background

Systems Administration

Deep experience with Linux (RHEL, SLES, CentOS), high-availability clusters, and STONITH configurations.

A small badge icon with a gear shape in muted blue tones on a light background

Cloud Infrastructure

Practical knowledge of AWS EC2, reserved instances, and cost analysis for cloud vs. on-premises deployments.

A small badge icon with a code symbol in muted blue tones on a light background

Development

Proficient in F#, PHP (Symfony), and cross-platform tooling including Mono and MonoDevelop on OSX.

Studies

F# & Functional Programming

Self-directed, 2012

Explored strongly typed functional programming with F# on OSX using the Mono runtime. Published a detailed getting-started guide covering toolchain setup and cross-platform game development research.

High-Availability & Cluster Management

Professional Development, 2009

Configured and documented crm_mon email alerting for STONITH events on SLES11-HAE clusters, integrating with Nagios monitoring for production environments.

Hardware & Storage Performance

Ongoing

Conducted hands-on benchmarking of SATA/SAS RAID controllers including HighPoint RocketRaid 2740 and LSI/SuperMicro AOC-USASLP2-H8iR, comparing against software RAID configurations.

Skills

A small icon representing a server with clean geometric lines in slate blue

Linux Administration

RHEL, SLES, CentOS — package management, kernel tuning, HA clustering, and monitoring integration.

A small icon representing a cloud shape with clean geometric lines in slate blue

Cloud Architecture

AWS EC2, reserved-instance planning, cost modeling, and hybrid infrastructure strategy.

A small icon representing code brackets with clean geometric lines in slate blue

F# & .NET/Mono

Functional programming on OSX, MonoDevelop toolchain, and cross-platform game-dev exploration.

A small icon representing a database cylinder with clean geometric lines in slate blue

PHP & Symfony

Web application development with the Symfony framework and the broader PHP ecosystem.

A small icon representing a storage drive with clean geometric lines in slate blue

Storage & RAID

SATA/SAS controller evaluation, md RAID configuration, and performance benchmarking.

A small icon representing a shield with clean geometric lines in slate blue

High Availability

Pacemaker, STONITH, crm_mon alerting, and Nagios integration for production cluster monitoring.