--copy-cmd option to allow use of custom copy commands about rsync

[edo1]

On other file-systems, you may still wish to compress file contents when rsync'ing for backup, copying to a mobile drive etc. You could do this using --copy-cmd and gzip (or xz or similar) to compress destination files, like-so:

Hmm… Let's have a big file (e.x. 4Gb) on the server.

1. Client runs rync server:bigfile .;

sent 43 bytes  received 4,296,015,958 bytes  8,635,208.04 bytes/sec
total size is 4,294,967,296  speedup is 1.00

2. Some changes on the server occurs;

dd if=/dev/urandom of=bigfile bs=1k count=1 seek=1000000 conv=notrunc

3. Client runs rync server:bigfile . again. Only changes (and checksums) are transferred.

sent 524,363 bytes  received 327,794 bytes  28,886.68 bytes/sec
total size is 4,294,967,296  speedup is 5,040.11

IMO such rsync behavior is incompatible with your proposal.

[Haravikk]

I'm not sure what you're trying to demonstrate here?

One of my opening lines is:

When provided, instead of copying a file using the default method (delta transfer), rsync will instead use the provided custom command.

The trade off for using the custom command is that rsync can't do checksum based transfers, but you're still gaining the full benefits of rsync's various features for finding changed files (by timestamp, and optionally size if you know it should be the same), plus filtering of transfer lists etc.

You would (and should) only use a custom copy command in cases where the benefits of doing so are well known to be better than relying on rsync's checksummed transfer algorithm, i.e- you know that changed files will need to be copied (or cloned) in their entirety.

But there are plenty of cases where the copy command itself will lack a lot of rsync's flexibility; cp as I've used in these examples doesn't have any of the finding/filtering/comparison options that rsync does, and nor do compression tools.

There could be an argument for an intermediate transfer option, e.g- for compression, the command is used to generate a file locally for transfer, so the per-block comparison can still occur with a previously compressed version of the file, but this would need to be an additional option as it won't be suitable for every command.

[WayneD]

Just so you know, this is fairly unlikely to be implemented. It would likely be limited to local copies only, so if I ever get around to doing some big changes to rsync's local copying workflow then I will be considering this idea as an additional feature.

[Haravikk]

I appreciate any consideration of this; I should stress, not all options are required, I have a tendency to overthink stuff like this, and the ability to do this with remote transfers is always something that can be done later.

In terms of local transfers the only changes that are required are:

1. --copy-cmd to take the command, once a file is identified for transfer this is executed in place of normal behaviour, disables comparison of files by size by default.
2. --copy-cmd-size to re-enable comparison by size when the copy command should preserve it.

Warnings for incompatible options (--append and --inplace) might be appropriate, but they're not required as the use of the copy command would bypass them anyway.