Transfers

Transfers#

Moving bytes from one place to another, on the same host or across the network, with the right metadata and the right verification. For the operator, “transfer” covers the local intra-host copy of a config, the rename in place, the cross-host pull of a target’s home directory, the offline image of a disk, and the daily sync of an engagement directory to a redirector or a NAS.

Pick the tool by the trade-off it bites on. cp and mv for intra-host moves where the cost is metadata fidelity. rsync for anything resumable or repeated; the delta algorithm saves an order of magnitude of bandwidth on the second pass. scp and sftp for one-shot SSH transfers without infrastructure. tar piped through SSH when the operator needs a streaming single-pass copy of a whole tree, often with compression and encryption in the same pipeline. dd when the operator wants the bytes underneath the filesystem.

The sections below walk the anatomy, the tools in priority order, the constraints that bite, and the muscle-memory tasks the operator runs against this surface.

A file transfer is a read on the source, an optional transform (compress, encrypt, hash), a transport (kernel page cache, socket, or device), and a write on the destination with the metadata the operator chose to preserve.

        flowchart LR
    SRC[Source<br/>file / tree / device] --> READ[read]
    READ --> XFORM["transform<br/>(compress / encrypt / hash)"]
    XFORM --> TRANSPORT[transport]
    TRANSPORT --> WRITE[write]
    WRITE --> DST[Destination<br/>file / tree / device]

    TRANSPORT -. "local pipe" .-> TLOCAL[same host]
    TRANSPORT -. "SSH / TLS" .-> TNET[remote host]

Element	Description
`read`	Source I/O. Filesystem on a local disk, a stream from another tool, or a block device.
`transform`	Optional. Compression (`zstd`, `gzip`), encryption (`age`, `gpg`), or hashing for verify (see Hashing).
`transport`	Local pipe, loopback, SSH, TLS, or a kernel socket. Bandwidth, latency, and integrity guarantees live here.
`write`	Destination I/O. New file, overwritten file, append to existing, or device write.
metadata	Mode, owner, group, atime/mtime, ACLs, xattrs, hardlinks, sparseness. Each tool decides what to carry.

A useful shorthand. Local intra-host transfer is read+write with no transport. Pull from a remote is read on the far side, transport over SSH/TLS, write locally. Push is the inverse. Sync is both sides comparing first and then transferring only the delta.

cp and mv#

cp and mv are the local-only workhorses. cp copies bytes plus chosen metadata; mv renames (cheap, no copy) when source and destination are on the same filesystem, or copies + deletes (expensive) when they cross filesystems.

Command	Effect
`cp SRC DST`	Copy one file.
`cp -a SRC DST`	Archive mode. Preserves mode, owner, timestamps, links, ACLs, xattrs; recurses.
`cp -r SRC/ DST/`	Recursive copy without the full metadata preservation of `-a`.
`cp --reflink=auto SRC DST`	Copy-on-write on Btrfs / XFS / Bcachefs (instant, zero data movement).
`cp --sparse=always SRC DST`	Detect runs of zeros, write as sparse holes.
`cp -i SRC DST`	Prompt before overwrite. `-n` to never overwrite.
`mv SRC DST`	Rename or move. Atomic within one filesystem.
`install -m 0644 -o root -g root SRC DST`	Copy with explicit mode and ownership in one shot.

Copy a config and keep its permissions and timestamps.

$ cp -a /etc/ssh/sshd_config /tmp/sshd.bak

Move a tree across filesystems (becomes copy + delete; verify before the delete completes).

$ mv -v /srv/data /mnt/new-pool/data

Reflink-copy a 10 GB VM image instantly on Btrfs.

$ cp --reflink=always disk.qcow2 snapshot.qcow2

Closing pointer. cp -a is the operator’s default for anything that needs to round-trip through tooling later (logs, configs, collected artifacts). For tree-to-tree work with a network or a repeat, jump to rsync.

rsync#

rsync is the daily-driver delta-sync tool. The rolling-checksum algorithm transfers only the changed blocks of changed files, so the second-and-later run over the same source costs a fraction of the first. Works locally, over SSH, against an rsync daemon, or in either direction across both. The operator uses it for backups, mirroring engagement data to a redirector, syncing configurations across estates, and resuming interrupted copies.

Command	Effect
`rsync -avh SRC/ DST/`	Verbose, archive mode (`-a` = recursive + perms + times + symlinks + owner + group + devices).
`rsync -aHAX SRC/ DST/`	Add hardlinks, ACLs, and xattrs to `-a`.
`rsync -avzP SRC/ user@host:/DST/`	Compress (`-z`), show progress + allow resume (`-P`), push to remote.
`rsync -avn SRC/ DST/`	Dry run; list what would change.
`rsync -av --delete SRC/ DST/`	Make destination an exact mirror; remove files absent from source.
`rsync -av --checksum SRC/ DST/`	Decide by content hash, not mtime + size (slower, exact).
`rsync -av --bwlimit=10M SRC/ DST/`	Cap throughput at 10 MB/s.
`rsync -av -e 'ssh -p 2222 -i ~/.ssh/key' SRC/ host:/DST/`	Choose the SSH command (custom port, key, jump host).
`rsync -av --exclude '.git/' --exclude 'node_modules/' SRC/ DST/`	Skip noisy directories.
`rsync -av --link-dest=/snap/prev SRC/ /snap/today`	Hardlink unchanged files from a previous snapshot, save space.

Mirror a directory exactly, with deletion, dry-run first.

$ rsync -avhn --delete /home/user/proj/ /mnt/backup/proj/
$ rsync -avh  --delete /home/user/proj/ /mnt/backup/proj/

Pull a remote tree over SSH, resumable, with progress.

$ rsync -avzP user@target:/var/log/ ./pulled-logs/

Daily incremental backup with hardlink dedup against yesterday.

$ rsync -aHAX --delete \
    --link-dest=/backup/$(date -d 'yesterday' +%F) \
    /srv/ /backup/$(date +%F)/

Trailing slash matters. SRC/ (with slash) means “the contents of SRC”; SRC (no slash) means “the directory SRC itself”. This is the most common rsync foot-gun.

scp and sftp#

scp and sftp ride SSH. No daemon to set up beyond sshd, no agent, no infra. The trade-off is bare-functionality compared to rsync; for a one-shot copy that won’t be repeated, they’re the shortest path.

scp since OpenSSH 9.0 uses the SFTP protocol underneath by default (-O reverts to the legacy SCP protocol). Treat scp and sftp as the same tool with two different CLIs.

Command	Effect
`scp file user@host:/path/`	Push a file.
`scp user@host:/path/file ./`	Pull a file.
`scp -r dir/ user@host:/path/`	Recursive push.
`scp -P 2222 -i key file user@host:/path/`	Custom port and key.
`scp -3 user@a:/file user@b:/file`	Two-host copy proxied through the operator’s host.
`scp -J jump@bastion user@host:/file ./`	Through a jump host.
`sftp user@host`	Interactive session (`get`, `put`, `ls`, `cd`, `lcd`).
`sftp -b script user@host`	Batch mode; commands from a script.

Push a config to ten hosts in parallel (with parallel and SSH multiplexing).

$ parallel -j 10 scp sshd_config {}:/tmp/ ::: $(< hosts.txt)

Pull a remote tree without ``rsync`` on the far side.

$ scp -r user@host:/var/log/ ./logs/

Interactive session for inspect-then-grab.

$ sftp -P 2222 user@host
sftp> ls -l /var/log
sftp> get -r /var/log/audit/

Closing pointer. scp is fine for a single small transfer. For anything large, anything repeated, anything that may need a resume, switch to rsync -avzP -e ssh.

tar over SSH#

When rsync is unavailable on one side or the operator wants a single streaming copy with no on-disk intermediate, tar piped through SSH is the standard pattern. The pipeline reads, archives, optionally compresses and encrypts, transports, and writes in one continuous flow.

Command	Effect
`tar -c -C src . \\| ssh host 'tar -x -C /dst'`	Stream a tree to a remote target.
`ssh host 'tar -c -C /src .' \\| tar -x -C ./dst`	Pull a remote tree.
`tar --zstd -c -C src . \\| ssh host 'tar --zstd -x -C /dst'`	Compress in flight with zstd.
`tar -c dir \\| age -r KEY \\| ssh host 'cat > tree.tar.age'`	Compose with age for on-the-wire encryption.
`tar -c dir \\| pv \\| ssh host 'cat > tree.tar'`	Add a throughput meter with `pv`.

Stream a whole tree to a remote host, preserving owner / perms.

$ tar --numeric-owner -c -C /srv/data . | \
    ssh redirector 'tar --numeric-owner -x -C /pull/data'

Encrypted streaming pull, no plaintext touches the wire or either disk except inside the operator’s intended endpoints.

$ ssh target 'tar -c -C /home/user/proj . | age -r $RECIPIENT_KEY' \
    > proj.tar.age

For per-archive vs at-rest vs per-file encryption choices that plug into these pipelines, see Encryption. For hashing the result and verifying on receipt, see Hashing.

dd#

dd reads and writes blocks. Operator territory is disk imaging, device wipes, MBR / GPT byte poking, and any case where the operator needs to operate underneath the filesystem.

Command	Effect
`dd if=/dev/sdb of=disk.img bs=4M status=progress conv=noerror,sync`	Image a disk, continue past read errors.
`dd if=disk.img of=/dev/sdb bs=4M status=progress`	Restore an image.
`dd if=/dev/urandom of=/dev/sdb bs=1M status=progress`	Overwrite a device with random bytes.
`dd if=/dev/zero of=blob bs=1M count=1024`	Create a 1 GiB zero-filled file.

For most imaging the operator wants ddrescue (from gddrescue) instead; it tracks bad blocks, retries them, and resumes. See Disks for block-device, partition, and imaging detail.

Constraints#

Trailing slash on rsync source paths changes meaning. SRC/ copies the contents of SRC; SRC copies the directory SRC itself into the destination.
cp -r is not the same as cp -a. -r recurses but drops timestamps, ownership variation, ACLs, and xattrs unless the operator adds the flags. Use -a for fidelity.
mv is cheap within a filesystem and expensive across one. Crossing filesystems falls back to copy + unlink; an interrupt in the middle leaves both source and destination.
scp since OpenSSH 9.0 speaks SFTP by default. The legacy protocol (-O) is going away; do not rely on its quirks.
Reflink copies (cp --reflink) need Btrfs, XFS (with reflink=1), Bcachefs, or ZFS-on-Linux. ext4 will fail.
Hardlinks across filesystems are impossible at the syscall level. rsync --link-dest therefore must point inside the same filesystem as the destination.
Sparse files may de-sparsify in transit. cp --sparse=always and rsync --sparse preserve holes; tar needs --sparse.
SSH agent forwarding is convenient and dangerous in equal measure. Prefer -J jump hosts over -A forwarding.
Always verify with a hash on the far side (see Hashing). Network corruption is rare on modern paths; the operator’s cp errors are common.

Common Tasks#

Copy a directory while preserving every piece of metadata.

$ cp -a /etc /backup/etc.$(date +%F)

Mirror a tree to a backup drive, exactly, with dry-run first.

$ rsync -aHAXn --delete /home/user/ /mnt/backup/user/
$ rsync -aHAX  --delete /home/user/ /mnt/backup/user/

Pull a target’s home directory over SSH, with progress and resume.

$ rsync -avzP -e 'ssh -p 2222 -i ~/.ssh/op_key' \
    user@target:/home/user/ ./pulled/

Push the same payload to many hosts.

$ for h in $(< hosts.txt); do
    scp -i ~/.ssh/op_key payload.bin "user@$h:/tmp/" &
  done; wait

Stream a tree across SSH with no intermediate file, compressed and encrypted.

$ ssh target 'tar --zstd -c -C /srv/data .' | \
    age -r $RECIPIENT_KEY > data.tar.zst.age

Resume an interrupted big copy (rsync’s -P is partial+progress).

$ rsync -avP big.iso user@host:/srv/iso/

Verify a transferred tree by content hash on both sides.

$ find /srv/data -type f -print0 | sort -z | xargs -0 sha256sum > local.sums
$ ssh host 'find /srv/data -type f -print0 | sort -z | xargs -0 sha256sum' \
    > remote.sums
$ diff -u local.sums remote.sums

Move a file atomically into place (write to .tmp, then mv).

$ install -m 0644 -o root -g root new.conf /etc/svc/svc.conf.tmp
$ mv /etc/svc/svc.conf.tmp /etc/svc/svc.conf

Copy with throughput cap so the transfer doesn’t saturate the link.

$ rsync -avh --bwlimit=10M big-tree/ host:/srv/big-tree/

Image a remote disk to the operator’s box.

$ ssh target 'sudo dd if=/dev/sda bs=4M status=progress' \
    | dd of=target-sda.img bs=4M

Copy a tree through a jump host without staging.

$ rsync -av -e 'ssh -J jump@bastion' user@inner:/var/log/ ./logs/

Resume a partial download with ``curl`` or ``wget``.

$ curl -L -C - -o file.bin https://example.org/big.bin
$ wget -c https://example.org/big.bin

References#

man 1 cp, man 1 mv, man 1 install (local copy, move, install with permissions).
man 1 rsync, man 5 rsyncd.conf (incremental sync, daemon mode).
man 1 scp, man 1 sftp, man 1 ssh (SSH-based transfer; jump hosts; multiplexing).
man 1 tar, man 1 cpio (archive-based streaming copy).
man 1 dd, man 1 ddrescue (block-level imaging).
Files for the underlying file abstraction.
Reading and Writing for the read/write surface the transfer tools stand on.
Hashing for content-hash verification of the result.
Encryption for at-rest, per-file, and per-archive encryption.
Disks for block devices, partitions, and imaging.
Linux for the command quick-reference.
rsync algorithm (Tridgell)
OpenSSH manual