Posted inLinux

Master Rsync on Linux: SSH Sync, Bandwidth Tuning, and Incremental Backups

Linux tutorial - IT technology blog
Linux tutorial - IT technology blog

Quick Start (5 Min)

Transferring tens of gigabytes between servers with standard utilities like cp or scp is a gamble. A dropped SSH connection forces you to restart from scratch. Worse, they blindly rewrite every byte of a 40 GB database dump even if only 2 KB changed. That is why rsync (remote sync) remains essential for sysadmins and DevOps teams: its delta-transfer algorithm inspects files at both ends and pushes only the differences across the wire.

First, verify rsync is present on both your local host and the target machine:

# Debian / Ubuntu
sudo apt update && sudo apt install -y rsync

# RHEL / Rocky Linux / AlmaLinux
sudo dnf install -y rsync

The fundamental syntax is simple:

rsync [OPTIONS] SOURCE DESTINATION

The Golden Trailing Slash Rule

Watch your trailing slashes. Missing or adding a single / is the single most common cause of accidental directory nesting on Linux:

# Without trailing slash: copies 'mydata' directory into '/backup'
# Result: /backup/mydata/file.txt
rsync -av /var/www/mydata /backup

# With trailing slash: copies the *contents* of 'mydata' directly into '/backup'
# Result: /backup/file.txt
rsync -av /var/www/mydata/ /backup

Your First Remote Sync over SSH

To securely pull log directories from a production host, invoke standard archive flags with SSH transport:

rsync -avz -e ssh [email protected]:/var/log/nginx/ /opt/logs/remote-nginx/

Each flag performs a distinct task: -a preserves file modes, timestamps, symlinks, and ownership; -v prints verbose output; and -z compresses packet payloads in flight to save network throughput.

Deep Dive: Core Flags and Secure Remote Transfers

Running syncs against production volumes requires knowing how flags alter filesystem state.

Essential Operational Flags

  • -a (Archive mode): Bundles -rlptgoD. It recursively syncs files, preserves symlinks, and keeps permissions, timestamps, owner, group, and device nodes intact.
  • -P (Progress and Partial): Combines --progress and --partial. You get an active progress bar with transfer speeds and ETAs. If the connection breaks halfway through an 8 GB ISO transfer, rsync saves the partial file and resumes at that exact byte offset next time.
  • --delete: Prunes files from the destination path if they have disappeared from the source. It ensures a 1:1 replica, but double-check your target paths before running it.
  • -h (Human-readable): Translates raw byte counters into readable units like 512 MB or 3.4 GB.

Safe Execution with Dry Runs

Never run an untested sync with --delete against real data. Test your path logic first using -n (dry run) alongside -i (itemize changes):

rsync -avzPhn --delete /data/project/ [email protected]:/storage/project/

The dry run parses every inclusion and exclusion pattern and prints out planned actions without touching a single file on disk.

Custom SSH Configurations

Hardened bastion hosts often listen on non-standard ports or demand dedicated private keys. Pass custom connection arguments straight into -e:

rsync -avzP -e "ssh -p 2222 -i /home/deploy/.ssh/backup_rsa" /etc/app/ backup-node:/remote-backups/app/

Advanced Usage: Bandwidth Tuning and Incremental Backups

Moving a 120 GB database dump across a shared WAN link can easily saturate uplinks, drop VoIP packets, and slow down user requests. You can prevent network congestion with bandwidth caps and save disk space with filesystem hard links.

Bandwidth Throttling

The --bwlimit option sets a transfer rate ceiling in kilobytes per second. To cap an offsite sync to roughly 5 MB/s (40 Mbps):

# Cap throughput at 5,000 KB/s (~5 MB/s)
rsync -avzP --bwlimit=5000 /var/backups/db.sql offsite-storage:/backups/

This keeps bulk transfers from crowding out live application traffic during business hours.

Smart Incremental Backups with Hard Links

Running full backups every night exhausts disk arrays quickly. Traditional incremental schemes save space, but restoring files from deep snapshot chains is slow and error-prone. Rsync provides an elegant middle ground with --link-dest.

When generating a daily snapshot, rsync checks the previous day’s directory. If a file has not changed in size or timestamp, rsync creates an inode hard link to the existing file rather than writing new data blocks. Unchanged files consume zero extra storage blocks on disk.

In a test on an Ubuntu 22.04 server tracking a 50 GB directory with a daily change rate of 3%, daily backup runs completed in under 45 seconds. Each snapshot appears as a fully populated standalone folder, making restorations as simple as a standard cp or tar command.

Production Incremental Backup Script

You can deploy this lightweight automated backup script directly into cron:

#!/usr/bin/env bash
set -euo pipefail

# Configuration
SOURCE_DIR="/var/www/html/"
BACKUP_ROOT="/mnt/backups/web-app"
DATE_STAMP="$(date +%Y-%m-%d_%H%M%S)"
CURRENT_BACKUP="${BACKUP_ROOT}/${DATE_STAMP}"
LATEST_LINK="${BACKUP_ROOT}/latest"

mkdir -p "${BACKUP_ROOT}"

# Check if an earlier backup exists to link against
LINK_DEST_OPTION=""
if [ -d "${LATEST_LINK}" ]; then
    LINK_DEST_OPTION="--link-dest=${LATEST_LINK}"
fi

# Run incremental sync
rsync -a --delete \
    ${LINK_DEST_OPTION} \
    --exclude="*.log" \
    --exclude="cache/" \
    "${SOURCE_DIR}" "${CURRENT_BACKUP}"

# Atomically point 'latest' symlink to the new snapshot
rm -f "${LATEST_LINK}"
ln -s "${CURRENT_BACKUP}" "${LATEST_LINK}"

echo "Backup completed successfully: ${CURRENT_BACKUP}"

Make the script executable and schedule it to run every night at 2:00 AM:

chmod +x /usr/local/bin/backup-app.sh

# Open crontab editor
sudo crontab -e

# Add the schedule line:
0 2 * * * /usr/local/bin/backup-app.sh >> /var/log/backup-app.log 2>&1

Production Hardening and Performance Tips

When running rsync against large production filesystems, keep these operational details in mind:

1. Organize Exclusions in Dedicated Files

Long chains of inline --exclude flags turn shell scripts into unreadable messes. Put patterns in a dedicated text file instead:

# /etc/rsync-excludes.txt
.git/
*.tmp
node_modules/
storage/logs/*
cache/*

Load the list using --exclude-from:

rsync -avz --exclude-from="/etc/rsync-excludes.txt" /var/www/app/ /storage/app/

2. Protect System I/O with Nice and Ionice

Walking millions of file inodes and generating checksums generates heavy disk contention. On a loaded web or database host, a sudden backup job can spike I/O wait times and cause HTTP 504 timeouts. Dial back CPU and block I/O priorities with nice and ionice:

# Lower CPU scheduling priority (nice 19) and grant idle-only disk priority (ionice -c 3)
nice -n 19 ionice -c 3 rsync -a /heavy-data/ /backup/heavy-data/

3. Relieve CPU Bottlenecks on High-Speed LANs

On a 10GbE network connection, SSH encryption often hits CPU limits before the network cable saturates. Notice that -z compression also wastes CPU cycles on fast local networks with already compressed images or videos. If you are syncing across a trusted private VLAN, switch to an efficient stream cipher like [email protected] or [email protected] and drop -z:

rsync -av -e "ssh -c [email protected]" /srv/media/ remote-box:/srv/media/

These adjustments keep transfers fast, preserve storage capacity, and ensure predictable backups without degrading your production services.

Share:
Tags: