rsyncproject / rsync

An open source utility that provides fast incremental file transfer. It also has useful features for backup and restore operations among many other use cases.

WHAT IS RSYNC?

Rsync is a fast and extraordinarily versatile file copying tool for both remote and local files.

Rsync uses a delta-transfer algorithm which provides a very fast method for bringing remote files into sync. It does this by sending just the differences in the files across the link, without requiring that both sets of files are present at one of the ends of the link beforehand. At first glance this may seem impossible because the calculation of diffs between two files normally requires local access to both files.

A technical report describing the rsync algorithm is included with this package.

USAGE

Basically you use rsync just like scp, but rsync has many additional options. To get a complete list of supported options type:

rsync --help

See the manpage for more detailed information.

BUILDING AND INSTALLING

If you need to build rsync yourself, check out the INSTALL page for information on what libraries and packages you can use to get the maximum features in your build.

SETUP

Rsync normally uses ssh or rsh for communication with remote systems. It does not need to be setuid and requires no special privileges for installation. You must, however, have a working ssh or rsh system. Using ssh is recommended for its security features.

Alternatively, rsync can run in `daemon' mode, listening on a socket. This is generally used for public file distribution, although authentication and access control are available.

To install rsync, first run the "configure" script. This will create a Makefile and config.h appropriate for your system. Then type "make".

Note that on some systems you will have to force configure not to use gcc because gcc may not support some features (such as 64 bit file offsets) that your system may support. Set the environment variable CC to the name of your native compiler before running configure in this case.

Once built put a copy of rsync in your search path on the local and remote systems (or use "make install"). That's it!

RSYNC DAEMONS

Rsync can also talk to "rsync daemons" which can provide anonymous or authenticated rsync. See the rsyncd.conf(5) manpage for details on how to setup an rsync daemon. See the rsync(1) manpage for info on how to connect to an rsync daemon.

WEB SITE

For more information, visit the main rsync web site.

You'll find a FAQ list, downloads, resources, HTML versions of the manpages, etc.

MAILING LISTS

There is a mailing list for the discussion of rsync and its applications that is open to anyone to join. New releases are announced on this list, and there is also an announcement-only mailing list for those that want official announcements. See the mailing-list page for full details.

BUG REPORTS

The bug-tracking web page has full details on bug reporting.

That page contains links to the current bug list, and information on how to do a good job when reporting a bug. You might also like to try searching the Internet for the error message you've received, or looking in the mailing list archives.

To send a bug report, follow the instructions on the bug-tracking page of the web site.

Alternately, email your bug report to [email protected].

GIT REPOSITORY

If you want to get the very latest version of rsync direct from the source code repository, then you will need to use git. The git repo is hosted on GitHub and on Samba's site.

See the download page for full details on all the ways to grab the source.

COPYRIGHT

Rsync was originally written by Andrew Tridgell and Paul Mackerras. Many people from around the world have helped to maintain and improve it.

Rsync may be used, modified and redistributed only under the terms of the GNU General Public License, found in the file COPYING in this distribution, or at the Free Software Foundation.

rsync's Issues

rsync does not terminate when target disk is full

I'm copying a set of large files to a disk, when the disk is full, rsync does not terminate. When I stop rsync, it shows a "No space left" error. Here is an example of the output. Rsync was ran when the disk was already full from a previous run.

[root@backup]# rsync -av /data/source/ /mnt/bck/dest/
sending incremental file list
dump/
dump/2020_07_17.tgz
dump/2020_07_13.tgz
dump/2020_07_15.tgz
dump/2020_07_17.tgz
dump/2020_06_29.tgz
dump/2020_05_06.tgz
dump/2020_07_10.tgz
dump/2020_07_13.tgz
^Crsync error: received SIGINT, SIGTERM, or SIGHUP (code 20) at rsync.c(688) [sender=3.2.2]
rsync: [receiver] write failed on "/mnt/bck/dest/2020_07_17.tgz": No space left on device (28)
rsync error: error in file IO (code 11) at receiver.c(378) [receiver=3.2.2]

As you see, the error shows the first file that supposedly was copied. Afterwards, none of the listed files that were supposed to have been copied even exist on the destination disk. It has been ran several hours like this without any error until it is stopped. Also tried as non-root user with same effect.

The same issue seems to arise when copying to a destination without writing permissions. When stopping rsync:

rsync: mkstemp "/mnt/bck/dest/dump/.2020_07_17.tgz.C2GxbN" failed: Permission denied (13)

rsync 3.2.2

EDIT: Compiled latest rsync and tried again: same result.

--copy-cmd option to allow use of custom copy commands

This proposal is for the addition of a new option in the following form (name debatable):

--copy-cmd=COMMAND

When provided, instead of copying a file using the default method (delta transfer), rsync will instead use the provided custom command. The command is given as a string, similar to the --rsh option. This command only applies to file copies, other actions (metadata updates, linking etc.) occur as normal.

How it actually behaves differs depending upon the nature of the source and destination:

• Local -> Local: If both the source and destination are on the same system, then the special placeholders {src} and {dest} in the command are swapped for the (quoted and escaped) paths of the source and destination file respectively.
• Local -> Remote: If the source is local and the destination is remote, then only the {src} placeholder is used and the command should output the file data to standard output for transfer to the remote host (where the data is written to the destination).
• Remote -> Local: If the source is remote and the destination is local, then only the {dest} placeholder is used, and the command should take file data from standard input and write it out to the destination location.

Regardless of mode of operation, if the copy command returns a non zero status, rsync will treat the transfer as failed and produce an error or warning. It will also produce a warning in local -> local and remote -> local modes if no file is produced at the expected destination (i.e- command did not include the {dest} placeholder, didn't use it properly, or failed with a status of zero, which can happen if the command is several piped together).

This option would be incompatible with --append (also --inplace?). Use of this option will also disable the use of file-size for comparisons by default, as a custom command may produce a differently sized output file. An option will be needed to tell rsync to explicitly retain this behaviour, e.g- --copy-size, for when the sizes should match (when --copy-cmd is used for transparent compression, cloning etc.). Comparisons by modification time however should work as normal no matter what the copy command produces, as rsync should still be setting the time(s) on the file afterwards.

Examples

There are a few useful examples of how you could take advantage of this command:

Local File Cloning

Some filesystems support the use of clone/shadow-copy/reflink based zero-cost copying of files which functions similar to hard linking, except that each clone can be written to independently without affecting the other(s), i.e- at time of cloning they share the same data blocks on disk, but when written to they diverge, usually thanks to copy-on-write. To take advantage of this you might use --copy-cmd like so:

• macOS: rsync -a --copy-cmd='cp -c {src} {dest}' /path/to/source /path/to/destination
• Linux (with BTRFS or similar): rsync -a --copy-cmd='cp --reflink {src} {dest}' /path/to/source /path/to/destination

This is useful when you know you want to copy something for editing, but want it to be as quick and lightweight as possible, but where the plain command (cp -c or cp --reflink) doesn't offer the same flexibility that rsync does. This is also useful when you want to snapshot just a single directory, even though the full volume might support snapshots (as systems that support these commands usually do).

Invisible Compression

Some filesystems support on-demand per-file compression; for example on macOS, HFS+ and APFS both support invisible file compression. While there are patches that allow rsync to preserve this where a file is already compressed, there may be cases where you'd like to add/remove compression while copying, e.g- ensuring backups use up as little space as possible. You could do this using --copy-cmd like so:

rsync -a --copy-cmd='ditto --hfsCompression {src} {dest}' /path/to/source /path/to/destination

In this case rsync will ensure that all copied files have compression enabled where possible in the destination, even if the files were not compressed in the source.

Explicit Compression

On other file-systems, you may still wish to compress file contents when rsync'ing for backup, copying to a mobile drive etc. You could do this using --copy-cmd and gzip (or xz or similar) to compress destination files, like-so:

• Local -> Local: rsync -a --copy-cmd='gzip -9c {src} > {dest}' /path/to/source /path/to/dest
• Local -> Remote: rsync -a --copy-cmd='gzip -9c {src}' /path/to/source host:/path/to/dest
• Remote -> Local: rsync -a --copy-cmd='gzip -9 > {dest}' host:/path/to/source /path/to/dest

To decompress:

• Local -> Local: rsync -a --copy-cmd='gzip -dc {src} > {dest}' /path/to/source /path/to/dest
• Local -> Remote: rsync -a --copy-cmd='gzip -dc {src}' /path/to/source host:/path/to/dest
• Remote -> Local: rsync -a --copy-cmd='gzip -dc > {dest}' host:/path/to/source /path/to/dest

Implementation Considerations

For examples such as explicit compression, it may be useful to provide a supporting option --copy-cmd-out-ext or similar, so that files compressed using --copy-cmd can have a customised extension, for example --copy-cmd-out-ext=.gz such that rsync remains aware of the change in name, i.e- for a file with path foo/bar/baz, rsync would treat it on the destination side as foo/bar/baz.gz, but look for both versions (in-case the file was previously transferred without this extension). This would also benefit from a --copy-cmd-in-ext when reversing the direction of a copy, this would instead inform rsync to remove the extension if found on an incoming file (foo/bar/baz.gz becomes foo/bar/baz).

As a --copy-cmd may not be able to place the same guarantee on the correctness of attributes, these should be set after the copy command has been executed (does rsync already set attributes after transfer?).

Deprecated xattr header warnings

A few years ago xattr changed their headers to be provided by glibc (on Linux/glibc at least).
Right now the old/cross-platform way of checking for various xattr headers in lib/sysxattrs.h
results in the following warning:

clang -I. -I. -pipe -march=native -O3 -DHAVE_CONFIG_H -Wall -W  -c lib/sysxattrs.c -o lib/sysxattrs.o
In file included from lib/sysxattrs.c:23:
In file included from lib/sysxattrs.h:4:
/usr/include/attr/xattr.h:5:2: warning: "Please change your <attr/xattr.h> includes to <sys/xattr.h>" [-W#warnings]
#warning "Please change your <attr/xattr.h> includes to <sys/xattr.h>"
 ^
1 warning generated.

Further issue with lib/md5-asm-x86_64.s

About the fix from pull request #25 (comment), this seems to not enable dead code removal by the linker on macOS. I propose something like the following in addition, to align with the code inclusion in lib/md.5.c:

diff --git a/Makefile.in b/Makefile.in
index 31ddc43b..3dc37420 100644
--- a/Makefile.in
+++ b/Makefile.in
@@ -135,7 +135,7 @@ simd-checksum-x86_64.o: simd-checksum-x86_64.cpp
 	$(CXX) $(CXXFLAGS) $(CPPFLAGS) -c -o $@ $(srcdir)/simd-checksum-x86_64.cpp
 
 lib/md5-asm-x86_64.o: lib/md5-asm-x86_64.s
-	$(CC) -c -o $@ $(srcdir)/lib/md5-asm-x86_64.s
+	$(CC) -I. -c -o $@ $(srcdir)/lib/md5-asm-x86_64.s
 
 tls$(EXEEXT): $(TLS_OBJ)
 	$(CC) $(CFLAGS) $(LDFLAGS) -o $@ $(TLS_OBJ) $(LIBS)
diff --git a/lib/md5-asm-x86_64.s b/lib/md5-asm-x86_64.s
index a3126151..8299f203 100644
--- a/lib/md5-asm-x86_64.s
+++ b/lib/md5-asm-x86_64.s
@@ -24,6 +24,10 @@
  * show any significant difference in performance, though.
  */
 
+#include "config.h"
+
+#ifndef USE_OPENSSL
+
 .text
 .align 16
 
@@ -691,3 +695,5 @@ md5_process_asm:
 	ret
 .L_md5_process_asm_end:
 .size md5_process_asm,.L_md5_process_asm_end-md5_process_asm
+
+#endif

Comments?

ETA: I realize that this is not a perfect solution, since the use of md5_process_asm is further predicated on #if defined(HAVE_SIMD) && (CSUM_CHUNK == 64).

Please add reflink support to 3.2.0

Hi Wayne,

please add reflink support to 3.2.0 before it gets released. I'm waiting for this feature for a very long time already.
See https://bugzilla.samba.org/show_bug.cgi?id=10170

rsyncd problem after Debian testing upgrade to 3.2.1-1 from 3.1.3-8 (new systemd ProtectHome=on restriction)

First, thank you for a great product, I have been using rsync for
probably 30 years, and my current backup setup for many years without a
problem!

Since Debian pushed 3.2.1 to testing last Friday (6/26), I have been
seeing a problem with rsyncd.. I have 5 modules defined, and since the
update, two are seen as empty directories, while the other three
return the correct contents.

I saw the other crash bug reported which you fixed, but am not seeing a crash.

Here is my rsyncd.conf:

| uid = root
| gid = root
|
| max verbosity = 6
|
| max connections = 25
|
| read only = true
|
| address = localhost
| port = 17873
|
| use chroot = true
|
|
| [build]
| path = /build
|
| [etc]
| path = /etc
|
| [home]
| path = /home
|
| [music]
| path = /music
|
| [root]
| path = /root

The home and root modules are the ones which appear to be empty.

For example:

% rsync -vvvvvv --port=17873 rsync://localhost/home
opening tcp connection to localhost port 17873
Connected to localhost (::1)
msg checking charset: ANSI_X3.4-1968
sending daemon args: --server --sender -vvvvvvde.LsfxCIv . home/ (5 args)
(Client) Protocol versions: remote=31, negotiated=31
FILE_STRUCT_LEN=24, EXTRA_LEN=4
receiving file list ...
server_sender starting pid=36030
[sender] change_dir(/)
[sender] make_file(.,*,0)
[sender] pushing local filters for /
[sender] popping local filters
[sender] flist start=0, used=1, low=0, high=0
[sender] i=0 / ./ mode=040000 len=40 flags=1005
send_file_list done
[sender] flist_eof=1
send_files starting
recv_file_name(.)
received 1 names
done
[Receiver] flist_eof=1
[Receiver] flist start=0, used=1, low=0, high=0
[Receiver] i=0 0 ./ mode=040000 len=40 flags=1005
recv_file_list done
get_local_name count=1 home
generator starting pid=36029
delta-transmission enabled
recv_files(1) starting
recv_generator(.,0)
d--------- 40 2020/06/30 22:15:58 .
generate_files phase=1
send_files phase=1
recv_files phase=1
generate_files phase=2
send_files phase=2
send files finished
total: matches=0 hash_hits=0 false_alarms=0 data=0
recv_files phase=2
recv_files finished
generate_files phase=3
generate_files finished

sent 20 bytes received 508 bytes 1,056.00 bytes/sec
total size is 0 speedup is 0.00
[generator] _exit_cleanup(code=0, file=main.c, line=1779): entered
[generator] _exit_cleanup(code=0, file=main.c, line=1779): about to call exit(0)

however /home is 'drwxr-xr-x' owner/group=root and has three user home directories all
'drwxr-xr-x'.

As an example of one that works:

% rsync -vvvvvv --port=17873 rsync://localhost/build
opening tcp connection to localhost port 17873
Connected to localhost (::1)
msg checking charset: ANSI_X3.4-1968
sending daemon args: --server --sender -vvvvvvde.LsfxCIv . build/ (5 args)
(Client) Protocol versions: remote=31, negotiated=31
FILE_STRUCT_LEN=24, EXTRA_LEN=4
receiving file list ...
server_sender starting pid=36196
[sender] change_dir(/)
[sender] make_file(.,,0)
[sender] pushing local filters for /
[sender] make_file(OLDSTUFF,,2)
[sender] make_file(zsh,,2)
[sender] make_file(flashplayer,,2)
[sender] make_file(cddbd,,2)
[sender] make_file(xemacs,,2)
[sender] make_file(firefox,,2)
[sender] make_file(wmctrl,,2)
[sender] make_file(fvwm,,2)
[sender] make_file(SOURCE,,2)
[sender] make_file(xterm,,2)
[sender] make_file(xv,,2)
[sender] make_file(fetchmail,*,2)
[sender] popping local filters
[sender] flist start=0, used=13, low=0, high=12
[sender] i=0 / ./ mode=040755 len=4,096 flags=1005
[sender] i=1 / OLDSTUFF/ mode=040755 len=4,096 flags=1000
[sender] i=2 / SOURCE/ mode=040755 len=4,096 flags=1000
[sender] i=3 / cddbd/ mode=040755 len=4,096 flags=1000
[sender] i=4 / fetchmail/ mode=040755 len=4,096 flags=1000
[sender] i=5 / firefox/ mode=040755 len=4,096 flags=1000
[sender] i=6 / flashplayer/ mode=040755 len=4,096 flags=1000
[sender] i=7 / fvwm/ mode=040755 len=4,096 flags=1000
[sender] i=8 / wmctrl/ mode=040755 len=4,096 flags=1000
[sender] i=9 / xemacs/ mode=040755 len=4,096 flags=1000
[sender] i=10 / xterm/ mode=040755 len=4,096 flags=1000
[sender] i=11 / xv/ mode=040755 len=4,096 flags=1000
[sender] i=12 / zsh/ mode=040755 len=4,096 flags=1000
send_file_list done
[sender] flist_eof=1
send_files starting
recv_file_name(.)
recv_file_name(OLDSTUFF)
recv_file_name(zsh)
recv_file_name(flashplayer)
recv_file_name(cddbd)
recv_file_name(xemacs)
recv_file_name(firefox)
recv_file_name(wmctrl)
recv_file_name(fvwm)
recv_file_name(SOURCE)
recv_file_name(xterm)
recv_file_name(xv)
recv_file_name(fetchmail)
received 13 names
done
[Receiver] flist_eof=1
[Receiver] flist start=0, used=13, low=0, high=12
[Receiver] i=0 0 ./ mode=040755 len=4,096 flags=1005
[Receiver] i=1 1 OLDSTUFF/ mode=040755 len=4,096 flags=1000
[Receiver] i=2 1 SOURCE/ mode=040755 len=4,096 flags=1000
[Receiver] i=3 1 cddbd/ mode=040755 len=4,096 flags=1000
[Receiver] i=4 1 fetchmail/ mode=040755 len=4,096 flags=1000
[Receiver] i=5 1 firefox/ mode=040755 len=4,096 flags=1000
[Receiver] i=6 1 flashplayer/ mode=040755 len=4,096 flags=1000
[Receiver] i=7 1 fvwm/ mode=040755 len=4,096 flags=1000
[Receiver] i=8 1 wmctrl/ mode=040755 len=4,096 flags=1000
[Receiver] i=9 1 xemacs/ mode=040755 len=4,096 flags=1000
[Receiver] i=10 1 xterm/ mode=040755 len=4,096 flags=1000
[Receiver] i=11 1 xv/ mode=040755 len=4,096 flags=1000
[Receiver] i=12 1 zsh/ mode=040755 len=4,096 flags=1000
recv_file_list done
get_local_name count=13 build
generator starting pid=36195
delta-transmission enabled
recv_generator(.,0)
recv_files(13) starting
drwxr-xr-x 4,096 2019/12/05 23:34:12 .
recv_generator(OLDSTUFF,1)
drwxr-xr-x 4,096 2018/11/10 16:59:16 OLDSTUFF
recv_generator(SOURCE,2)
drwxr-xr-x 4,096 2019/12/05 23:30:05 SOURCE
recv_generator(cddbd,3)
drwxr-xr-x 4,096 2013/02/17 15:42:56 cddbd
recv_generator(fetchmail,4)
drwxr-xr-x 4,096 2018/11/25 20:00:06 fetchmail
recv_generator(firefox,5)
drwxr-xr-x 4,096 2019/04/11 23:06:41 firefox
recv_generator(flashplayer,6)
drwxr-xr-x 4,096 2019/05/17 14:07:31 flashplayer
recv_generator(fvwm,7)
drwxr-xr-x 4,096 2019/11/21 11:39:18 fvwm
recv_generator(wmctrl,8)
drwxr-xr-x 4,096 2019/12/05 22:59:48 wmctrl
recv_generator(xemacs,9)
drwxr-xr-x 4,096 2013/11/07 17:34:51 xemacs
recv_generator(xterm,10)
drwxr-xr-x 4,096 2019/12/05 23:34:12 xterm
recv_generator(xv,11)
drwxr-xr-x 4,096 2012/12/30 23:57:43 xv
recv_generator(zsh,12)
drwxr-xr-x 4,096 2013/04/10 11:07:24 zsh
generate_files phase=1
send_files phase=1
recv_files phase=1
generate_files phase=2
send_files phase=2
send files finished
total: matches=0 hash_hits=0 false_alarms=0 data=0
recv_files phase=2
recv_files finished
generate_files phase=3
generate_files finished

sent 20 bytes received 1,904 bytes 3,848.00 bytes/sec
total size is 0 speedup is 0.00
[generator] _exit_cleanup(code=0, file=main.c, line=1779): entered
[generator] _exit_cleanup(code=0, file=main.c, line=1779): about to call exit(0)

Note that /etc also works and has the same permissions, ownership, and
group as /home.

I did not see anything related in the release notes.

thanks for any advice on getting to the bottom of this, I would like
to get my remote backups working again asap.

Greg

setproctitle to include module name

Hi,

This is more of a wishlist. On a host with multiple rsync modules configured, the process listing looks something like this:

[module1]
path = ...
...
[module2]
path = ...

root 15749 0.0 0.0 12960 764 ? Ss Aug06 1:15 /usr/bin/rsync --daemon --no-detach
nobody 10137 0.1 0.0 127508 59168 ? S Aug14 3:34 \_ /usr/bin/rsync --daemon --no-detach
nobody 13059 0.1 0.0 130536 101228 ? S Aug14 5:10 \_ /usr/bin/rsync --daemon --no-detach
nobody 54596 0.1 0.0 130536 101248 ? S Aug14 4:38 \_ /usr/bin/rsync --daemon --no-detach
nobody 28047 0.1 0.0 130536 101248 ? S Aug15 4:50 \_ /usr/bin/rsync --daemon --no-detach

From a process listing, it's hard to tell which rsync child process is associated with which rsync module. I think the only way to achieve that is to have separate users and the module config stanzas so each one uses a separate user.

Would it be possible for rsync children processes to set the process title to include the rsync module?

rsync errors out reproducibly on specific files - caused by --compress

(initially reported as https://bugzilla.opensuse.org/show_bug.cgi?id=1175854 )

While rsync'ing /repositories from stage.opensuse.org to provo-mirror.opensuse.org, I got reproducible errors on specific files:

mirror@pontifex (provo-mirror):~> while true ; do /usr/bin/withlock -q /home/mirror/run/sync_opensuse.sh.lock rsync -v -H --compress -ar0 --stats --timeout=21600 --no-motd --fuzzy  --block-size=33000 --exclude-from=/home/mirror/rsync_exclude_update_provo-mirror.opensuse.org --exclude-from=/home/mirror/rsync_exclude_tumbleweed_provo-mirror.opensuse.org --exclude '/devel:/ARM:'--delete stage.opensuse.org::opensuse-full-really-everything-including-repositories/opensuse/repositories/ /srv/ftp/pub/opensuse/repositories/ ; echo ================================================================================ ; sleep 10 ; done
receiving incremental file list
GNOME:/Medias/images/iso/
GNOME:/Medias/images/iso/GNOME_Next.x86_64-3.36.5-Build13.378.iso
inflate returned -3 (224 bytes)
rsync error: error in rsync protocol data stream (code 12) at token.c(555) [receiver=3.1.3]
rsync: connection unexpectedly closed (9479495 bytes received so far) [generator]
rsync error: error in rsync protocol data stream (code 12) at io.c(235) [generator=3.1.3]
================================================================================
receiving incremental file list
GNOME:/Medias/images/iso/
GNOME:/Medias/images/iso/GNOME_Next.x86_64-3.36.5-Build13.378.iso
inflate returned -3 (224 bytes)
rsync error: error in rsync protocol data stream (code 12) at token.c(555) [receiver=3.1.3]
rsync: connection unexpectedly closed (9467555 bytes received so far) [generator]
rsync error: error in rsync protocol data stream (code 12) at io.c(235) [generator=3.1.3]
================================================================================
receiving incremental file list
GNOME:/Medias/images/iso/
GNOME:/Medias/images/iso/GNOME_Next.x86_64-3.36.5-Build13.378.iso
inflate returned -3 (224 bytes)
rsync error: error in rsync protocol data stream (code 12) at token.c(555) [receiver=3.1.3]
rsync: connection unexpectedly closed (9468879 bytes received so far) [generator]
rsync error: error in rsync protocol data stream (code 12) at io.c(235) [generator=3.1.3]
================================================================================

You might have noticed the --exclude '/devel:/ARM:' - that's the result of an earlier, similar error.

On the server side, rsyncd logs for one of these rsync runs:

2020/08/27 17:34:26 [14088] connect from provo-mirror.opensuse.org (91.193.113.70)
2020/08/27 17:34:26 [14088] rsync on opensuse-full-really-everything-including-repositories/opensuse/repositories/ from provo-mirror.opensuse.org (91.193.113.70)
2020/08/27 17:34:27 [14088] building file list
2020/08/27 17:35:32 [14088] deflate on token returned 0 (550 bytes left)
2020/08/27 17:35:32 [14088] rsync error: error in rsync protocol data stream (code 12) at token.c(427) [sender=3.1.3]

rsync works without problems if I drop the --compress option.

stage.opensuse.org runs openSUSE Leap 15.2, provo-mirror.opensuse.org runs openSUSE Leap 15.1, both with all updates installed.

rsync version is 3.1.3 on both servers (packages 3.1.3 rsync-3.1.3-lp151.4.3.1.x86_64 and rsync-3.1.3-lp152.5.7.x86_64)

Ignore_case patch creates an error with mismatch case in directories.

Sorry, I'm a (old) hardware man, so I'm not very familiar with all the 'protocols' around fixing software these days.
I don't intend to spend a week or so learning all the howto's of git / patch / etc etc etc. So sorry, but I've fixed it for myself (as far as I need it) this is to give someone else a chance to benefit from my fix.

I applied the patch 'ignore-case.diff' from 'rsync-patches-3.2.0.tar.gz' to the code in 'rsync-3.2.0.tar.gz' on LinuxMint 19.3,
Mostly it works, till today, trying to sync my music library with a copy on an external hard drive. I get this error.
Using the --ignore-case switch

ABORTING due to invalid path from sender: 1977 - Songs from the wood - Jethro Tull/01 - Songs From the Wood - Jethro Tull.flac [receiver=3.2.0]
What I see is that the two file paths differ in case in the Directory too, eg.
Target Dir has.
drwxr-sr-x 3 abcd users 4096 Jun 6 2014 1977 - Songs from the wood - Jethro Tull
drwxr-sr-x 2 abcd users 4096 Jun 6 2014 1977 - Songs from the Wood - Jethro Tull
drwxr-sr-x 2 abcd users 4096 Jun 6 2014 1977 - Songs From The Wood - Jethro Tull
Source Dir has.
drwxrwxrwx 1 abcd abcd 4096 Jun 6 2014 1977 - Songs from the wood - Jethro Tull
drwxrwxrwx 1 abcd abcd 4096 Jun 6 2014 1977 - Songs from the Wood - Jethro Tull
drwxrwxrwx 1 abcd abcd 4096 May 30 2012 1977 - Songs From The Wood - Jethro Tull

I managed to fix this with one line (in the patched file).
The message is created from a mismatch in the directory names at flist.c at line 2557
flist-fix-ignore-case-patch.txt

Hope this is of use.

rsync 3.2.1-1 crashes on Debian bullseye/sid (testing)

Hi, first of all, thanks for maintaining the rsync software!

Bug: since upgrade from 3.1.3-6 to 3.2.1-1 on the server (Debian bullseye/sid (testing)) I'm not able to rsync.

On the client side:

$rsync -av --rsync-path=/home/user/rsync-debug --password-file=pa.txt file.png  rsync://user@domain/path
rsync: didn't get server startup line
rsync error: error starting client-server protocol (code 5) at main.c(1675) [sender=3.1.3]

Server syslog:

Jun 29 09:06:49 localhost kernel: [2027341.018850] rsync[3763422]: segfault at 559cc8cefe4f ip 0000559cc8cda451 sp 00007fff4a01e830 error 7 in rsync[559cc8c96000+4e000]
Jun 29 09:06:49 localhost kernel: [2027341.018874] Code: 01 80 fb 25 75 d3 80 7d 01 25 0f 84 a9 00 00 00 be 25 00 00 00 4c 89 f7 4c 89 04 24 e8 18 c4 fb ff 4c 8b 04 24 48 85 c0 74 af <c6> 00 00 4c 89 f7 4c 89 44 24 08 48 89 04 24 e8 5b c
4 fb ff 48 8b

When downgrading to 3.1.3-6 everything works fine.

clang: note: diagnostic msg

macOS 10.15.6
Mac mini (2018)
3.2 GHz 6-Core Intel Core i7

Will compile with ./configure with --disable-simd

clang: note: diagnostic msg:

---

PLEASE ATTACH THE FOLLOWING FILES TO THE BUG REPORT:
Preprocessed source(s) and associated run script(s) are located at:
clang: note: diagnostic msg: /var/folders/nq/rvbx8hvx4xd0csp9mtf2cldc0000gn/T/simd-checksum-x86_64-863194.cpp
clang: note: diagnostic msg: /var/folders/nq/rvbx8hvx4xd0csp9mtf2cldc0000gn/T/simd-checksum-x86_64-863194.sh
clang: note: diagnostic msg: Crash backtrace is located in
clang: note: diagnostic msg: /Users/Tony/Library/Logs/DiagnosticReports/clang__.crash
clang: note: diagnostic msg: (choose the .crash file that corresponds to your crash)
clang: note: diagnostic msg:

---

If you can't fix the issue, re-run ./configure with --disable-simd.

Unknown do_compression value in token.c at line 1106

I just updated rsync to the latest version in git (on both servers) and when trying to sync some files remotely, I am getting the following error message

Unknown do_compression value in token.c at line 1106

Output

rsync -avz --stats --rsync-path='sudo rsync' --exclude 'distro/isolinux/livedvd32.img' --exclude 'distro/isolinux/livedvd64.img' --delete -P -e ssh [email protected]:/test /test

receiving incremental file list
distro/distro64/Clustering/
distro/distro64/Clustering/qemu-5.0.0-1.x86_64.rpm
              0   0%    0.00kB/s    0:00:00  Unknown do_compression value in token.c at line 1106

rsync error: requested action not supported (code 4) at token.c(1106) [receiver=3.2.3dev]
rsync: [generator] write error: Broken pipe (32)
rsync: connection unexpectedly closed (41589 bytes received so far) [generator]

rsync --version

rsync  version 3.2.3dev  protocol version 31
Copyright (C) 1996-2020 by Andrew Tridgell, Wayne Davison, and others.
Web site: https://rsync.samba.org/
Capabilities:
    64-bit files, 64-bit inums, 64-bit timestamps, 64-bit long ints,
    socketpairs, hardlinks, symlinks, IPv6, atimes, batchfiles, inplace,
    append, ACLs, xattrs, optional protect-args, iconv, symtimes, prealloc
Optimizations:
    SIMD, asm, openssl-crypto
Checksum list:
    md5 md4 none
Compress list:
    zstd zlibx zlib none

rsync comes with ABSOLUTELY NO WARRANTY.  This is free software, and you
are welcome to redistribute it under certain conditions.  See the GNU
General Public Licence for details

rsync-ssl什么时候支持可加密的server

1、当前rsync-ssl仅支持通过代理（Nginx、haproxy）方式加密server的client连接
2、有打算支持直接启动一个加密的server吗？

Website needs updates

There are no links to this GitHub on https://rsync.samba.org/ + https://rsync.samba.org/documentation.html + https://rsync.samba.org/issues.html + https://rsync.samba.org/FAQ.html

There is a bad link: https://rsync.samba.org/bugzilla.html, now it is here.

Only a little section "The latest version via GitHub" in https://rsync.samba.org/download.html

Building 3.2.1rc1 on macOS 10.14.6 fails

./configure --enable-ipv6 --disable-openssl --disable-xxhash --disable-zstd

And then make results in the following:

gcc -I. -I. -I./zlib -I./popt -g -O2 -DHAVE_CONFIG_H -Wall -W  -c acls.c -o acls.o
acls.c:740:84: warning: unused parameter 'mode' [-Wunused-parameter]
static int recv_rsync_acl(int f, item_list *racl_list, SMB_ACL_TYPE_T type, mode_t mode)
                                                                                   ^
1 warning generated.
gcc -I. -I. -I./zlib -I./popt -g -O2 -DHAVE_CONFIG_H -Wall -W  -c xattrs.c -o xattrs.o
gcc -I. -I. -I./zlib -I./popt -g -O2 -DHAVE_CONFIG_H -Wall -W  -c progress.c -o progress.o
gcc -I. -I. -I./zlib -I./popt -g -O2 -DHAVE_CONFIG_H -Wall -W  -c pipe.c -o pipe.o
gcc -I. -Wa,--noexecstack -c -o lib/md5-asm-x86_64.o ./lib/md5-asm-x86_64.S
g++ -I. -O2 -fno-exceptions -fno-rtti -c -o simd-checksum-x86_64.o ./simd-checksum-x86_64.cpp
clang: error: unable to execute command: Segmentation fault: 11
clang: error: clang frontend command failed due to signal (use -v to see invocation)

I can avoid all this if I also pass --disable-simd to configure.

Is it expected that building on macOS will require this flag? Or is it a bug?

Executable stack warning due to lib/md5-asm-x86_64.s

Since 3.2.0 I'm seeing:

Jun 20 11:31:04 ragnarok kernel: process '/usr/bin/rsync' started with executable stack

Filed in Gentoo here.

SIMD check in configure.ac is too strict

The line

if test x"$CXX" = x"g++"; then

is too strict for a check for g++. In Gentoo Linux we have x86_64-pc-linux-gnu-g++ (or similar string for other arches) in our CXX variable and thus the check fails even if the CPU supports SIMD.

Switch `--bwlimit` without effect

Rsync 3.1.3 on the server.

Our user start a transfer to our server with

rsync --bwlimit=1m subreads.bam [email protected]:/project/something

and on our server we see (obfuscated):

user   8219 24.5  0.0  15928  2132 ?        Ds   10:11  28:03 rsync --server -vlogDtpre.iLsfx --bwlimit=1024 --partial . /project/something/bla
user   8220  0.0  0.0  15668   192 ?        S    10:11   0:00 rsync --server -vlogDtpre.iLsfx --bwlimit=1024 --partial . /project/sweet/bla

So the limit should be one Megabyte per second. But we see constant transfer rates of 100 MB/s (no bursts).

Is that expected?

Force new features on ?

Hi,

What about forcing the new features on ?
I think about xxhash, SIMD, openssl...
So that someone compiling / packaging rsync would have to explicitly disable these features if they are not compatible with his system.
I think about this because rsync will flawlessly compile even if the required components for these new features are not found. At this moment the developer will feel happy, but the end-user will miss serious features.

Enforcing them would make these new features more easily / rapidly available in the various systems / distros IMO.

Thank you 👍

run windows binary in root directory

Hi,

I found a fatal error when run windows binary in root directory (C:\) , it works fine under any folder, such as "C:\New folder" :

C:\>rsync.exe

      0 [main] rsync (13504) C:\rsync.exe: *** fatal error - add_item ("\??", "/", ...) failed, errno 22
Stack trace:
Frame        Function    Args
000FFFFCCE0  00180062FFE (0018028DDFA, 00180268FC1, 00800000000, 000FFFF8B50)
000FFFFCCE0  0018004973A (00000000000, 000FFFFCCE0, 00180020010, 000FFFFABD6)
000FFFFCCE0  00180049772 (000FFFF9BD0, 00000000016, 00800000000, 000003F3F5C)
000FFFFCCE0  001800E8FCF (000FFFFCBD0, 000FFFFCDF0, 001800D7B88, 00000000000)
000FFFFCCE0  0018013BA65 (001800D407A, 00000000000, 00000000000, 00000000000)
000FFFFCCE0  00180049D95 (00000000000, 00000000000, 00000000000, 00000000000)
000FFFFFFF0  00180048846 (00000000000, 00000000000, 00000000000, 00000000000)
000FFFFFFF0  001800488F4 (00000000000, 00000000000, 00000000000, 00000000000)
End of stack trace

i have tried the following platforms & versions :
platform : windows 7 / windows 10 / server 2019
rsync version : 3.2.1, 3.2.0, 3.1.3, 3.1.2

reproduce steps :
condition 1 (version 3.2.1)

    1. install MSYS2 (MinGW-w64)
    2. compile source code
    3. copy files rsync.exe, /usr/bin/msys-2.0.dll to C:\
    4. run rsync.exe in DOS prompt then throw execption

condition 2 (version 3.1.3)

    1. install MSYS2 (MinGW-w64)
    2. install rsync from repository under MSYS2 shell :
         pacman -S rsync
    3. copy files /usr/bin/rsync.exe, /usr/bin/msys-2.0.dll, /usr/bin/msys-iconv-2.dll to C:\
    4. run rsync.exe in DOS prompt then throw execption

you can try compiled windows binary in https://jl-workshop.com/RSync-for-Windows-Free-Download/

Regards, Joe

On 32bit host, I get --max-alloc value is too big: 4G

We're running backups with rsync -H on multi-million files from multiple servers to a central storage. For this to work we have to increase the memory with the new --max-alloc option. While this works fine for most systems, there are still a number of 32bit systems around and now rsync fails on them.

From what I understand this is expected and the maximum possible value is --max-alloc=2G-1 for these hosts.

Now in our case it would be extremely helpful if rsync would not exit with an error in such a case but reduce the max-alloc value to the largest possible value and only print a warning about it.

That way our transfers could always use --max-alloc=4G-1 without knowing if one side is an old 32bit platform.

This is a small feature request and it would really be nice to have it. If there are better ways to solve our problem I appreciate to hear it.

Thanks,
Simon

license incompatibility issue (gpl3 with openssl)

Hello Wayne,

I see that you switched to github for issues and I guess you will also be accepting patches from here, I will be sending some patches that were hanging on Debian for some time soon.

The latest release of rsync (3.2.0) has a licensing issue caused by the addition of the build dependency libssl-dev. The problem can be easily solved by adding a OpenSSL exemption on your license definition, as suggested on:
https://lists.debian.org/debian-legal/2004/05/msg00595.html

 * In addition, as a special exception, the copyright holders give
 * permission to link the code of portions of this program with the
 * OpenSSL library under certain conditions as described in each
 * individual source file, and distribute linked combinations
 * including the two.
 * You must obey the GNU General Public License in all respects
 * for all of the code used other than OpenSSL.  If you modify
 * file(s) with this exception, you may extend this exception to your
 * version of the file(s), but you are not obligated to do so.  If you
 * do not wish to do so, delete this exception statement from your
 * version.  If you delete this exception statement from all source
 * files in the program, then also delete it here.

For more details about the issue, you can refer to:
https://people.gnome.org/~markmc/openssl-and-the-gpl.html
https://lists.debian.org/debian-legal/2002/10/msg00113.html

You might find that other distributions don't care about it, since debian is generally stricter with regards to licensing maters. I really appreciate if you could look into this one.

Thanks,

rsync transfer process hangs indefinitely

Hello,

First of all, thanks for rsync and apologies for not trying with the latest (3.2.3), but it's hard to upgrade since this is a production service that uses the Debian package. That said, I can do so if you believe it is needed.

Our use case: we have a single server that runs rsyncd and serves a large dataset, currently at 422GB (and is increasing over time). Then we have a bunch of other servers (currently 4 but also increasing over time) that all download at the same time, this dataset from the rsyncd host, using the following command:

rsync -aPH --chown=mysql:mysql --delete --inplace --bwlimit=40m rsync://${RSYNC_SERVER}/ezra/ ${MYSQL_WORK_DIR}

We notice that sometimes the rsync transfers hang. Eventually we applied a --bwlimit, which merely reduced the frequency at which this happens. This issue we have is back from 2017, when we were using rsync 3.1.2. So recently we noticed that rsync 3.2.1 included a relevant commit (01b9bbb) so we upgraded to it - it also reduced the frequency (I think) but the issue still happens occasionally.

Another observation is that the issue seems to be getting worse with the increase in the dataset and in the number of receiving hosts.

Below I'm attaching information which I believe may help to debug this (let me know if anything else is needed).

Thanks in advance!

rsync logs

Sending end (rsyncd):

2020/08/29 01:00:01 [23392] rsync on ezra/ from ezra4.foobar.com ([IPv6 REDACTED])[IPv6 REDACTED])
2020/08/29 01:00:01 [23392] building file list

Process tree

Receiving end:

root     26091  0.2  0.0  41796  5060 ?        S    Aug29   8:36 \_ rsync -aPH --chown=mysql:mysql --delete --inplace --bwlimit=40m rsync://strife.foobar.com/ezra/ /srv/ezra/data/mysql
root     26092  0.3  0.0  41828  3560 ?        S    Aug29  11:41     \_ rsync -aPH --chown=mysql:mysql --delete --inplace --bwlimit=40m rsync://strife.foobar.com/ezra/ /srv/ezra/data/mysql

Sending end (including daemon process):

root     26458  0.0  0.0  26680  1308 ?        Ss   Jul22   0:00 /usr/bin/rsync --daemon --no-detach
mysql    23392  2.8  0.0  54596 15412 ?        S    Aug29  90:23  \_ /usr/bin/rsync --daemon --no-detach

Send/Receive queues

Receiving end:

Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp6       0 529823 ezra4.foobar.com:32840  strife.foobar:rsync ESTABLISHED 26091/rsync

Sending end (rsyncd):

Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp6       0      0 strife.foobar:rsync ezra4.foobar.com:32840  ESTABLISHED 23392/rsync

System call trace

Receiving end (pid=26091):

select(5, [4], [3], [4], {tv_sec=59, tv_usec=166979}) = 0 (Timeout)
select(5, [4], [3], [4], {tv_sec=60, tv_usec=0}) = 0 (Timeout)
select(5, [4], [3], [4], {tv_sec=60, tv_usec=0}) = 0 (Timeout)
select(5, [4], [3], [4], {tv_sec=60, tv_usec=0}) = 0 (Timeout)
select(5, [4], [3], [4], {tv_sec=60, tv_usec=0}) = 0 (Timeout)
select(5, [4], [3], [4], {tv_sec=60, tv_usec=0}) = 0 (Timeout)

Receiving end (pid=26092):

select(4, [3], [], [3], {tv_sec=23, tv_usec=904884}) = 0 (Timeout)
select(4, [3], [], [3], {tv_sec=60, tv_usec=0}) = 0 (Timeout)
select(4, [3], [], [3], {tv_sec=60, tv_usec=0}) = 0 (Timeout)
select(4, [3], [], [3], {tv_sec=60, tv_usec=0}) = 0 (Timeout)
select(4, [3], [], [3], {tv_sec=60, tv_usec=0}) = 0 (Timeout)
select(4, [3], [], [3], {tv_sec=60, tv_usec=0}) = 0 (Timeout)
select(4, [3], [], [3], {tv_sec=60, tv_usec=0}) = 0 (Timeout)
select(4, [3], [], [3], {tv_sec=60, tv_usec=0}) = 0 (Timeout)

Sending end (pid=23392):

select(7, [6], [], [6], {tv_sec=47, tv_usec=71793}) = 0 (Timeout)
select(7, [6], [], [6], {tv_sec=60, tv_usec=0}) = 0 (Timeout)
select(7, [6], [], [6], {tv_sec=60, tv_usec=0}) = 0 (Timeout)
select(7, [6], [], [6], {tv_sec=60, tv_usec=0}) = 0 (Timeout)
select(7, [6], [], [6], {tv_sec=60, tv_usec=0}) = 0 (Timeout)

Network/TCP settings

The following are custom TCP settings tuned in an attempt to fix the issue.
Apparently, they didn't help.

Receiving end:

# settings for the socket receive buffer
net.core.rmem_default = 524288
net.core.rmem_max = 4194304

# settings for the socket send buffer
net.core.wmem_default = 524288

# tcp memory settings
net.ipv4.tcp_rmem = 100420 524288 4194304
net.ipv4.tcp_wmem = 100420 524288 4194304

System information

Receiving end:

• Debian GNU/Linux 9.6 (stretch)
• Linux ezra4 4.9.0-8-amd64 #1 SMP Debian 4.9.130-2 (2018-10-27) x86_64 GNU/Linux
• rsync version 3.2.1 protocol version 31

Sending end:

• Debian GNU/Linux 9.5 (stretch)
• Linux strife 4.9.0-8-amd64 #1 SMP Debian 4.9.110-3+deb9u6 (2018-10-08) x86_64 GNU/Linux
• rsync version 3.2.1 protocol version 31

rsyncd configuration

[ezra]
path = /srv/snapshots/ezra
lock file = /run/rsync/ezra.lock
uid = mysql
gid = mysql
comment = ezra MySQL data
max connections = 4
read only = true
hosts allow =  [IPv4 REDACTED] [IPv6 REDACTED]
exclude from = /srv/snapshots/ezra/.exclude
log file = /var/log/rsyncd.log

total files reported in stats differs for rsyncd

Hi Wayne, this is pretty minor but has bugged me for a long time.. I'm backing up to two locations, one local accessing directly, and a second remote using rsyncd. I've noticed that the 'Number of files' output by the --stats summary differs in these two cases; the 'Number of regular files transferred', 'Total file size', and 'Total transferred file size' do agree. I'm using an rsnapshot summarizer, for example:

SOURCE                TOTAL FILES   FILES TRANS      TOTAL MB     MB TRANS
--------------------------------------------------------------------------
/home/                     322600            65     264848.91       135.81
/music/                     30107             0     240162.43         0.00
/etc/                        1435            10          7.69         0.16
/root/                        322             0          1.79         0.00
/build/                    453006             0       9235.40         0.00
--------------------------------------------------------------------------
TOTAL                      807470            75     514256.20       135.98

vs

SOURCE                TOTAL FILES   FILES TRANS      TOTAL MB     MB TRANS
--------------------------------------------------------------------------
rsync://lwm/home           356528            65     264848.91       135.81
rsync://lwm/music           32758             0     240162.43         0.00
rsync://lwm/etc              2967            10          7.69         0.16
rsync://lwm/root              524             0          1.79         0.00
rsync://lwm/build          485837             0       9235.40         0.00
--------------------------------------------------------------------------
TOTAL                      878614            75     514256.20       135.98

FreeBSD help-from-md issue

Hi,

On FreeBSD, with 3.2.0pre2, make leads to the following issue :

./help-from-md "./" help-rsync.h
Discarding empty output for help-rsync.h file from ./
*** Error code 1

Stop.

Trying to figure-out why.

Thx 👍

Memory increase in 3.2.2

Hi,

Since rsync 3.2.2 the backups on some of my (low-mem) systems are failing due to rsync suddenly using a lot more memory. I've traced down the issue to 11eb67e and, after some trial and error, found what was causing the memory increase: Before that commit, realloc_array(NULL,..) would call malloc(), after that commit it would instead call calloc(), thus initializing memory that (I presume) would not otherwise get used.

Measurements using time -v:
Bad:

        User time (seconds): 0.44
        System time (seconds): 0.22
        Percent of CPU this job got: 23%
        Elapsed (wall clock) time (h:mm:ss or m:ss): 0:02.82
        Average shared text size (kbytes): 0
        Average unshared data size (kbytes): 0
        Average stack size (kbytes): 0
        Average total size (kbytes): 0
        Maximum resident set size (kbytes): 249068
        Average resident set size (kbytes): 0
        Major (requiring I/O) page faults: 0
        Minor (reclaiming a frame) page faults: 62776
        Voluntary context switches: 105
        Involuntary context switches: 675
        Swaps: 0
        File system inputs: 0
        File system outputs: 0
        Socket messages sent: 0
        Socket messages received: 0
        Signals delivered: 0
        Page size (bytes): 4096
        Exit status: 0

Good:

        User time (seconds): 0.10
        System time (seconds): 0.13
        Percent of CPU this job got: 10%
        Elapsed (wall clock) time (h:mm:ss or m:ss): 0:02.29
        Average shared text size (kbytes): 0
        Average unshared data size (kbytes): 0
        Average stack size (kbytes): 0
        Average total size (kbytes): 0
        Maximum resident set size (kbytes): 11816
        Average resident set size (kbytes): 0
        Major (requiring I/O) page faults: 0
        Minor (reclaiming a frame) page faults: 3394
        Voluntary context switches: 105
        Involuntary context switches: 259
        Swaps: 0
        File system inputs: 0
        File system outputs: 0
        Socket messages sent: 0
        Socket messages received: 0
        Signals delivered: 0
        Page size (bytes): 4096
        Exit status: 0

The difference in Maximum resident set size is pretty significant.

@ERROR: chdir failed (daemon failure due to new systemd ProtectHome=on restriction)

As a follow up for issue #43. After applying changes, I get a new error:

Client (rsync version 3.1.3 protocol version 31):

rsync -av --password-file=pa.txt file.png  rsync://user@host/dir/
@ERROR: chdir failed
rsync error: error starting client-server protocol (code 5) at main.c(1675) [sender=3.1.3]

Server (rsync version 3.2.1 protocol version 31):

2020/07/01 09:00:40 [2017875] rsync: [Receiver] chdir /home/localdir failed: No such file or directory (2)

All directories mentioned above exist. When downgrading to rsync 3.1.3 everything works fine.

https://github.com/WayneD/rsync/issues

sync of ordinary files over network fails verify with --sparse --inplace

Getting some puzzling errors doing the following backup procedure on a root filesystem that's on LVM, when using sparse flag:

lvcreate --snapshot --name baksnap --size 3G /dev/vg0/root
mount -o ro /dev/vg0/baksnap /var/tmp/snapmnt
rsync -izaHx \
    --delete --delete-excluded --delete-after \
    --partial --inplace --sparse \
    --numeric-ids --bwlimit=640k --rsh=ssh \
    /var/tmp/snapmnt \
    bakhost:/path/to/backups/root/

This is causing the following errors:

.d..t...... etc/lvm/archive/
<f+++++++++ etc/lvm/archive/vg0_01748-450988166.vg
<f+++++++++ etc/lvm/archive/vg0_01749-1953687750.vg
.d..t...... etc/lvm/backup/
<f..t...... etc/lvm/backup/vg0
<f..t...... root/.bash_history
<f..t...... etc/lvm/backup/vg0
ERROR: etc/lvm/backup/vg0 failed verification -- update retained.
<f..t...... root/.bash_history
ERROR: root/.bash_history failed verification -- update retained.
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1287)
[sender=3.2.2pre2]

It repeats the above as many times as I run it. Simply removing the --sparse flag makes the error go away.

This makes no sense to me in the first place, because I'm mounting the snapshot readonly, so how can it change during the copy (presumably this is why it fails verification)? And in the second place, I don't see how sparse flag would affect anything at all. I'm using --sparse --inplace because I have large VM images on this filesystem, and transfers are sometimes interrupted. However these are only very small text files showing the error (and again, they are in a readonly mount...)

I'm using Linux kernel 4.17.8 on this system, glibc 2.27, x86_64 arch.
I built 3.2.2pre2 with the following flags:

--disable-ipv6 --disable-openssl --disable-xxhash --disable-zstd --disable-lz4

Capabilities:
    64-bit files, 64-bit inums, 64-bit timestamps, 64-bit long ints,
    socketpairs, hardlinks, symlinks, no IPv6, atimes, batchfiles, inplace,
    append, ACLs, xattrs, optional protect-args, iconv, symtimes, prealloc
Checksum list:
    md5 md4 none
Compress list:
    zlibx zlib none

Not sure what is the issue here. I was excited to learn about --sparse --inplace support, but it doesn't seem usable in what seems like an ordinary use case. Any ideas what I could be doing wrong? Thanks.

Please bundle the man page in the tarball

I tried to build the new rsync on macOS. After figuring out which --disable-* flags now need to be passed, the configure script bailed out saying that I needed to install some external deps in order to build a man page.

The attempted build is from the distributed tarball and not a GitHub Release or a cloned repo.

I think the manpage should be prebuilt for the regular release source tarball as it used to be. Otherwise users have to install some heavy deps (python3 at the very least) just to be able to have a man page.

Add option to copy just metadata

Suggestion: Add an option (maybe --metadata-only) which will copy only metadata from a specified existing file, folder, or dirtree to a specified existing file, folder, or dirtree.

My use case involves copying metadata from only 1 existing source file or folder to 1 existing target file or folder (not to files or folders that may be within the target folder). I intend to use rsync for this at various points within a script.

A comment to https://askubuntu.com/a/150888 points out an existing option that is similar to what I'm asking for:

--archive --size-only is a great combo. Not only does it prevent recopying files that already exist in the destination, but it will also update their metadata. This was unexpected for me, because rsync's man page describes --size-only as "skipping" files whose sizes match. Turns out that it just skips the copy, but will still sync the metadata.

I'd prefer an option whose name & manpage description are more applicable (than --size-only) to my use case. For example, I don't need that rsync should "detect whether" to copy the metadata; I need rsync to copy the metadata to the target file or folder unconditionally. Also, the name & the manpage description of the relevant option (--metadata-only or whatever) should be about metadata specifically; avoiding, for example, the possibility of copying any contents to the target file or folder.

My interests about what metadata to copy are as follows:

• For certain, what I need copied includes owning user, owning group, the Unix classic permissions bits (rwxrwxrwx + sticky bits & setuid), and all extended attributes (xattrs) that are on the source file or folder.
• It is OK if I need to specify another option at the same time, to modify --metadata-only (or whatever it ends up being called) to get xattrs copied.
• I'm not asking for any of atime, btime, ctime, mtime to be copied. For instance, the target file or folder may have different contents than the source, so copying "the time the contents were modified" to be the same as the source mtime isn't necessarily meaningful.
• File or folder name should definitely NOT be copied.

I look forward to your comments.

BTW, I looked through recent issues & I noticed that WayneD and the other contributors have been jumping right onto reported bugs. That's first-rate. 👍

Better "connection unexpectedly closed" message

We often use rsync to get multiple boxes synchronized.

If one of these target systems has a problem, seeing, for example:

rsync: connection unexpectedly closed (0 bytes received so far) [sender]
rsync error: unexplained error (code 255) at io.c(226) [sender=3.1.2]

is not helpful in identifying the system that has the problem.

Something like:

rsync: connection with TARGET unexpectedly closed (0 bytes received so far) [sender]
rsync error: unexplained error (code 255) at io.c(226) [sender=3.1.2]

where TARGET was appropriately expanded would be much more useful.

Stats may be wrongly computed

Hi,

In some situations, especially when transfer is aborted, stats are wrongly computed, at both sides.

Here's an example of a CTRL+C interrupted transfer :

# du -b /tmp/source/
3221229568    /tmp/source/

# rsync -a /tmp/source/ ...
sending incremental file list
/tmp/
/tmp/source/
/tmp/source/bf1
          1.07G 100%  147.89MB/s    0:00:06 (xfr#1, to-chk=2/5)
/tmp/source/bf2
          1.07G 100%  132.94MB/s    0:00:07 (xfr#2, to-chk=1/5)
/tmp/source/bf3
        521.34M  48%  124.39MB/s    0:00:04  ^C

# "cat receiver.log"
rsync: connection unexpectedly closed (146 bytes received so far) [generator]
sent 81 bytes  received 146 bytes  total size 3221225472
rsync error: error in rsync protocol data stream (code 12) at io.c(226) [generator=3.2.0dev]

Results : on receiver side, sent / received numbers are wrong.

And with SIGUSR2 :

# du -b /tmp/source/
3221229568    /tmp/source/

# rsync -a /tmp/source/ ...
sending incremental file list
/tmp/
/tmp/source/
/tmp/source/bf1
          1.07G 100%  149.53MB/s    0:00:06 (xfr#1, to-chk=2/5)
/tmp/source/bf2
          1.07G 100%  133.87MB/s    0:00:07 (xfr#2, to-chk=1/5)
/tmp/source/bf3
        527.70M  49%  125.91MB/s    0:00:04  
Number of files: 5 (reg: 3, dir: 2)
Number of created files: 5 (reg: 3, dir: 2)
Number of deleted files: 0
Number of regular files transferred: 3
Total file size: 3.22G bytes
Total transferred file size: 3.22G bytes
Literal data: 2.15G bytes
Matched data: 0 bytes
File list size: 0
File list generation time: 0.001 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 0
Total bytes received: 0
sent 0 bytes  received 0 bytes  -0.00 bytes/sec
total size is 3.22G  speedup is -9,223,372,036,854,775,808nf

# "cat receiver.log"
rsync: connection unexpectedly closed (150 bytes received so far) [generator]
sent 81 bytes  received 150 bytes  total size 3221225472
rsync error: error in rsync protocol data stream (code 12) at io.c(226) [generator=3.2.0dev]

Results : on receiver side, sent / received numbers are wrong.
Results : on sender side, many of the --stats numbers are wrong.

Thank you 👍

maintained versions

What does the project consider a maintained version? Some projects such as PHP and Python declare support for each X.Y version, with defined end of life dates. Others like Redis maintain the current and previous major versions. Other projects don't use this concept at all, and just release continuously higher new versions. What is the rsync approach to versioning?

To rephrase with a specific example, if a security vulnerability was discovered that affected rsync 3.1.3, would the fix only be included in the next 3.2.x version, or would a new 3.1.4 version be released? I'd like to know because I maintain the rsync31 RPM package in the IUS repository, which provides version-named packages as alternatives to the older versions typically included in RHEL and CentOS. If there will never be another 3.1.x version, I plan to retire the rsync31 package. I could create a replacement rsync32 or rsync3 package as appropriate.

Please provide precompiled binaries

Compiling from source isn't trivial and build instructions are underdocumented and requires root access to install required dev packages. Updating such a basic utility doesn't justify the amount of work involved.

The latest of the major distro's ship with older versions of rsync that contain trivial bugs.

• ubuntu 20.04 - v3.1.3
• centos - v3.1.2
• Debian 10 - 3.1.3

It would help a lot if latest precompiled static binary and/or rpm/deb packages were made available.

Use after free in xattr code

I'm getting segfaults in find_matching_xattr when trying to use rsync for backup with the -X option - valgrind identifies use-after-free which I suspect is the underlying problem

This is in rsync 3.2.2

eg

==811998== Invalid read of size 4
==811998== at 0x440DA2: find_matching_xattr (xattrs.c:420)
==811998== by 0x442162: cache_tmp_xattr (xattrs.c:887)
==811998== by 0x428838: copy_valid_path (backup.c:133)
==811998== by 0x428838: get_backup_name (backup.c:173)
==811998== by 0x4288AB: make_backup (backup.c:239)
==811998== by 0x410D89: finish_transfer (rsync.c:712)
==811998== by 0x418FAB: recv_files (receiver.c:874)
==811998== by 0x422F6B: do_recv (main.c:1006)
==811998== by 0x423759: do_server_recv (main.c:1176)
==811998== by 0x423759: start_server (main.c:1210)
==811998== by 0x4238A4: child_main (main.c:1183)
==811998== by 0x443AE4: local_child (pipe.c:166)
==811998== by 0x4053DE: do_cmd (main.c:642)
==811998== by 0x4053DE: start_client (main.c:1519)
==811998== by 0x4053DE: main (main.c:1760)
==811998== Address 0x4f8a6d8 is 8 bytes inside a block of size 16 free'd
==811998== at 0x483AA0C: free (vg_replace_malloc.c:538)
==811998== by 0x442284: uncache_tmp_xattrs (xattrs.c:932)
==811998== by 0x4287D7: copy_valid_path (backup.c:150)
==811998== by 0x4287D7: get_backup_name (backup.c:173)
==811998== by 0x4288AB: make_backup (backup.c:239)
==811998== by 0x410D89: finish_transfer (rsync.c:712)
==811998== by 0x418FAB: recv_files (receiver.c:874)
==811998== by 0x422F6B: do_recv (main.c:1006)
==811998== by 0x423759: do_server_recv (main.c:1176)
==811998== by 0x423759: start_server (main.c:1210)
==811998== by 0x4238A4: child_main (main.c:1183)
==811998== by 0x443AE4: local_child (pipe.c:166)
==811998== by 0x4053DE: do_cmd (main.c:642)
==811998== by 0x4053DE: start_client (main.c:1519)
==811998== by 0x4053DE: main (main.c:1760)
==811998== Block was alloc'd at
==811998== at 0x483BB1A: calloc (vg_replace_malloc.c:760)
==811998== by 0x4212E4: my_alloc (util2.c:97)
==811998== by 0x4212E4: my_alloc (util2.c:87)
==811998== by 0x440F4A: rsync_xal_store.isra.0 (xattrs.c:477)
==811998== by 0x441DEE: receive_xattr (xattrs.c:871)
==811998== by 0x4078DA: recv_file_entry (flist.c:1130)
==811998== by 0x40F0D9: recv_file_list (flist.c:2532)
==811998== by 0x4234A7: do_server_recv (main.c:1132)
==811998== by 0x4234A7: start_server (main.c:1210)
==811998== by 0x4238A4: child_main (main.c:1183)
==811998== by 0x443AE4: local_child (pipe.c:166)
==811998== by 0x4053DE: do_cmd (main.c:642)
==811998== by 0x4053DE: start_client (main.c:1519)
==811998== by 0x4053DE: main (main.c:1760)

Cygwin SIMD

Hi,

I was not able to find any previous conversation regarding this, can't SIMD be supported with Cygwin ?
It does not for now.

$ uname -s -r
CYGWIN_NT-6.1 3.1.6(0.340/5/3)

$ ./rsync --version
rsync  version 3.2.3dev  protocol version 31
Copyright (C) 1996-2020 by Andrew Tridgell, Wayne Davison, and others.
Web site: https://rsync.samba.org/
Capabilities:
    64-bit files, 64-bit inums, 64-bit timestamps, 64-bit long ints,
    socketpairs, hardlinks, symlinks, IPv6, atimes, batchfiles, inplace,
    append, ACLs, no xattrs, optional protect-args, no iconv, symtimes,
    prealloc, stop-at
Optimizations:
    no SIMD, asm, openssl-crypto
Checksum list:
    xxh64 (xxhash) md5 md4 none
Compress list:
    zstd lz4 zlibx zlib none

Thx 👍

iconv filename conversion partly fails on mac

Situation

I'm using rsync (3.2.2) on MacOS (10.14) to sync parts of thumbdrives (FAT/NTFS formatted) to a remote Synology NAS (rsync 3.0.9), where they are accessible over SAMBA.
Some filenamens contain accents and umlauts (e.g. défilé, Überraschung). MacOS uses some crazy non-standard normalization to encode these in decomposed UTF8. Therefore i'm using the --iconv=UTF8-MAC,UTF-8 option. So far so good.

Bug/appearance

The problem arises with filenames that contain high surrogate pairs (e.g. U+1F4A9 💩 pile of poo, U+1D6F0 𝛰 mathematical italic capital omicron). MacOS' iconv will not convert these:
echo 𝛰 | iconv -f utf-8-mac -t utf-8
iconv: (stdin):1:0: cannot convert
Therefore rsync fails to (not-)convert them as well:
[sender] cannot convert filename: 🤬you too.msg (Illegal byte sequence)
even though they don't need to be converted, as they already are the same normalization and encoding that Linux uses.

Problem

I can't work arround this issue by rsyncing twice, as i need to use --delete. Also, there are already files containing both umlauts and emojis ("Süße Vögel 😍.jpg"). Not using iconv creates wrongly normalized umlaut-containing filenames on the target, which practically renders them inaccessible on the Synology NAS and sharing over SAMBA.

Cause

It seems that 4-byte UTF8 characters will just not convert from UTF8-MAC to [any encoding scheme].
Now you might argue that this is in fact an inssue with Apple's (lib)iconv. That may be technically true, but Apple is surely not going to fix/extend that code anytime soon, if at all, and especially not for their security-fixes-only OS.
I already read about SATOH Fumiyasu's project https://github.com/fumiyas/libiconv-utf8mac that probably would fix the issue if rsync is linked with it instead of Apple's libiconv - but i'm unable to do that. I am not a dev.

Fix ?

Again, I'm merely a junior sysadmin, not a dev. But I suppose rsync (when linked agains Apple's libiconv) should detect the use of --iconv=UTF(-)8-MAC,UTF(-)8. Then not convert whole filenames but only single unicode characters, and not try to convert, but just pass through UTF8 4-byte long Unicode characters (U+D800 to U+DFFF and >= U+10000).
Or maybe rsync could just incorporate said libiconv-utf8mac?

Overflow in read_varint()

I have a script that used rsync with local source and both local and remote destination paths, which for simplicity supplied an --rsh option in both cases. Probably as a result of changes in 4f6c8c6, if the option includes the character V this fails:

$ rsync --rsh 'V' -r /etc/motd /tmp/
Overflow in read_varint()
rsync error: error in rsync protocol data stream (code 12) at io.c(1747) [sender=3.2.3]
rsync error: received SIGUSR1 (code 19) at main.c(1592) [Receiver=3.2.3]

This seems to fix it, but I'm not sure if it's correct:

Index: compat.c
--- compat.c.orig
+++ compat.c
@@ -705,7 +705,7 @@ void setup_protocol(int f_out,int f_in)
 				do_negotiated_strings = 1;
 				compat_flags |= CF_VARINT_FLIST_FLAGS;
 			}
-			if (strchr(client_info, 'V') != NULL) { /* Support a pre-release 'V' that got superseded */
+			if (!local_server && strchr(client_info, 'V') != NULL) { /* Support a pre-release 'V' that got superseded */
 				if (!write_batch)
 					compat_flags |= CF_VARINT_FLIST_FLAGS;
 				write_byte(f_out, compat_flags);

3.2.0 Fails to build on Alpine Linux

g++ -g -O2 -fno-exceptions -fno-rtti -O2 -I/usr/local/include -I/usr/local/ssl/include -UHAVE_SIMD -c -o simd-checksum-x86_64.o ./simd-checksum-x86_64.cpp
./simd-checksum-x86_64.cpp: In function 'uint32_t get_checksum1_cpp(char*, int32_t)':
./simd-checksum-x86_64.cpp:88:52: error: multiversioning needs ifunc which is not supported on this target
   88 | __attribute__ ((target("default"))) MVSTATIC int32 get_checksum1_avx2_6(schar* buf, int32 len, int32 i, uint32* ps1, uint32* ps2) { return i; }
      |                                                    ^~~~~~~~~~~~~~~~~~~~
./simd-checksum-x86_64.cpp:474:1: error: use of multiversioned function without a default
  474 | }
      | ^
./simd-checksum-x86_64.cpp:89:52: error: multiversioning needs ifunc which is not supported on this target
   89 | __attribute__ ((target("default"))) MVSTATIC int32 get_checksum1_ssse3_32(schar* buf, int32 len, int32 i, uint32* ps1, uint32* ps2) { return i; }
      |                                                    ^~~~~~~~~~~~~~~~~~~~~~
./simd-checksum-x86_64.cpp:474:1: error: use of multiversioned function without a default
  474 | }
      | ^
gmake: *** [Makefile:135: simd-checksum-x86_64.o] Error 1

I'm guessing that's because ifunc is a glibc feature, which would not work on a Linux that uses non-GNU libc like Alpine with musl libc.

The same error occurs on SmartOS.

Failed test on macOS 10.14.6

I built the new rsync 3.2.0 from the source tarball using the following flags: --disable-openssl --disable-xxhash --disable-zstd --disable-simd.

(Without those building failed.)

Finally a build succeeded, so I ran make test. One test failed:

check how the files compare with diff:

-------------
Failed:  dir-diff
----- atimes log ends

build fails with SIMD checksum optimizations

Building 3.2.0pre2 in current Arch Linux environment I get:

g++ -march=x86-64 -mtune=generic -O2 -pipe -fno-plt -D_FORTIFY_SOURCE=2 -c -o simd-checksum-x86_64.o simd-checksum-x86_64.cpp
{standard input}: Assembler messages:
{standard input}:280: Error: symbol `_ZL21get_checksum1_avx2_64PaiiPjS0_' is already defined
{standard input}:301: Error: symbol `_ZL21get_checksum1_sse2_32PaiiPjS0_' is already defined
make: *** [Makefile:128: simd-checksum-x86_64.o] Error 1

rsync -v: say if no changes were made

Let's take a look at rsync -v output:

sent 433 bytes  received 1,162 bytes  354.44 bytes/sec
total size is 44,983  speedup is 28.20

A second run reveals

sent 20 bytes  received 63 bytes  23.71 bytes/sec
total size is 44,983  speedup is 541.96

A third run reveals

sent 20 bytes  received 63 bytes  18.44 bytes/sec
total size is 44,983  speedup is 541.96

OK, that is great, but all this info misses the main point:

On the second and third runs it should also say
"No changes made", "Files already (r)synced, nothing to do.", etc.

Sure, the savvy user will know
"sent 20 bytes received 63 bytes"
are the two magic numbers via which the pros can tell that no changes were made,
but why not cough up that fact that no changes were made in the -v messages?
It seems more significant than exact bytes per second decimals, etc.

rsync version 3.2.3 protocol version 31

Sender waits for IO timeout instead of exiting

Hi,

I found a bug / edge case where sender waits for IO timeout instead of exiting, whereas receiver has closed the transfer. Below are some test cases.

Source tree :

-rw------- 1 root root 1610612736 May 31 10:17 anotherfile.iso
-rw------- 1 root root 2147483648 May 31 10:16 bf2

Destination tree :

-rw------- 1 root root 2147483648 May 31 10:16 bf1

Destination disk space remaining : around 200MB, much less than anotherfile.iso size.
Options used for debug below : -M--debug=FUZZY2,FLIST4 --debug=FUZZY2,FLIST4.

rsync -e "ssh ..." --progress -a /tmp/source dest: :

FILE_STRUCT_LEN=24, EXTRA_LEN=4
FILE_STRUCT_LEN=24, EXTRA_LEN=4
sending incremental file list
[sender] make_file(source,*,0)
[sender] flist start=1, used=1, low=0, high=0
[sender] i=1 /tmp source/ mode=040700 len=4,096 uid=0 gid=0 flags=1005
send_file_list done
[sender] make_file(source/anotherfile.iso,*,2)
[sender] make_file(source/bf2,*,2)
[sender] flist start=3, used=2, low=0, high=1
[sender] i=3 /tmp source/anotherfile.iso mode=0100600 len=1,610,612,736 uid=0 gid=0 flags=1000
[sender] i=4 /tmp source/bf2 mode=0100600 len=2,147,483,648 uid=0 gid=0 flags=1000
[sender] flist_eof=1
file list sent
recv_file_name(source)
received 1 names
[Receiver] flist start=1, used=1, low=0, high=0
[Receiver] i=1 1 source/ mode=040700 len=4,096 gid=(0) flags=1405
recv_file_list done
[Receiver] receiving flist for dir 0
recv_file_name(source/anotherfile.iso)
recv_file_name(source/bf2)
received 2 names
[Receiver] flist start=3, used=2, low=0, high=1
[Receiver] i=3 2 source/anotherfile.iso mode=0100600 len=1,610,612,736 gid=(0) flags=1400
[Receiver] i=4 2 source/bf2 mode=0100600 len=2,147,483,648 gid=(0) flags=1400
recv_file_list done
delta-transmission enabled
[receiver] flist_eof=1
[generator] flist_eof=1
source/
source/anotherfile.iso
  1,610,612,736 100%   19.90MB/s    0:01:17 (xfr#1, to-chk=1/3)
source/bf2
  2,147,483,648 100%   19.96MB/s    0:01:42 (xfr#2, to-chk=0/3)
rsync: write failed on "anotherfile.iso": Disc quota exceeded (69)
rsync error: error in file IO (code 11) at receiver.c(378) [receiver=3.2.0dev]

Above we see that both files are fully sent, even if there's not enough space for the first one.

rsync -e "ssh ..." --progress -a -M--msgs2stderr /tmp/source dest: :

FILE_STRUCT_LEN=24, EXTRA_LEN=4
FILE_STRUCT_LEN=24, EXTRA_LEN=4
sending incremental file list
[sender] make_file(source,*,0)
[sender] flist start=1, used=1, low=0, high=0
[sender] i=1 /tmp source/ mode=040700 len=4,096 uid=0 gid=0 flags=1005
send_file_list done
[sender] make_file(source/anotherfile.iso,*,2)
[sender] make_file(source/bf2,*,2)
[sender] flist start=3, used=2, low=0, high=1
[sender] i=3 /tmp source/anotherfile.iso mode=0100600 len=1,610,612,736 uid=0 gid=0 flags=1000
[sender] i=4 /tmp source/bf2 mode=0100600 len=2,147,483,648 uid=0 gid=0 flags=1000
[sender] flist_eof=1
file list sent
recv_file_name(source)
received 1 names
[Receiver] flist start=1, used=1, low=0, high=0
[Receiver] i=1 1 source/ mode=040700 len=4,096 gid=(0) flags=1405
recv_file_list done
[Receiver] receiving flist for dir 0
recv_file_name(source/anotherfile.iso)
recv_file_name(source/bf2)
received 2 names
[Receiver] flist start=3, used=2, low=0, high=1
[Receiver] i=3 2 source/anotherfile.iso mode=0100600 len=1,610,612,736 gid=(0) flags=1400
[Receiver] i=4 2 source/bf2 mode=0100600 len=2,147,483,648 gid=(0) flags=1400
recv_file_list done
delta-transmission enabled
[receiver] flist_eof=1
[generator] flist_eof=1
source/
source/anotherfile.iso
    253,526,016  15%   20.05MB/s    0:01:06
rsync: write failed on "anotherfile.iso": Disc quota exceeded (69)
rsync error: error in file IO (code 11) at receiver.c(378) [receiver=3.2.0dev]
rsync: [sender] write error: Broken pipe (32)

-M--msgs2stderr really helps here, perfect (thank you #3 !).

rsync -e "ssh ..." --progress -a -M--msgs2stderr -y --timeout=600 /tmp/source dest: :

FILE_STRUCT_LEN=24, EXTRA_LEN=4
FILE_STRUCT_LEN=24, EXTRA_LEN=4
sending incremental file list
[sender] make_file(source,*,0)
[sender] flist start=1, used=1, low=0, high=0
[sender] i=1 /tmp source/ mode=040700 len=4,096 uid=0 gid=0 flags=1005
send_file_list done
[sender] make_file(source/anotherfile.iso,*,2)
[sender] make_file(source/bf2,*,2)
[sender] flist start=3, used=2, low=0, high=1
[sender] i=3 /tmp source/anotherfile.iso mode=0100600 len=1,610,612,736 uid=0 gid=0 flags=1000
[sender] i=4 /tmp source/bf2 mode=0100600 len=2,147,483,648 uid=0 gid=0 flags=1000
[sender] flist_eof=1
file list sent
recv_file_name(source)
received 1 names
[Receiver] flist start=1, used=1, low=0, high=0
[Receiver] i=1 1 source/ mode=040700 len=4,096 gid=(0) flags=1405
recv_file_list done
[Receiver] receiving flist for dir 0
recv_file_name(source/anotherfile.iso)
recv_file_name(source/bf2)
received 2 names
[Receiver] flist start=3, used=2, low=0, high=1
[Receiver] i=3 2 source/anotherfile.iso mode=0100600 len=1,610,612,736 gid=(0) flags=1400
[Receiver] i=4 2 source/bf2 mode=0100600 len=2,147,483,648 gid=(0) flags=1400
recv_file_list done
delta-transmission enabled
[receiver] flist_eof=1
[generator] make_file(source/bf1,*,1)
[generator] flist start=0, used=1, low=0, high=0
[generator] i=0 0 source/bf1 mode=0100600 len=2,147,483,648 gid=1025 flags=1001
fuzzy distance for source/bf1 = 54.04428
fuzzy size/modtime match for source/bf1
fuzzy basis selected for source/bf2: source/bf1
source/
source/anotherfile.iso
     84,475,904   5%   20.11MB/s    0:01:14
[generator] flist_eof=1
    253,132,800  15%   20.03MB/s    0:01:06
rsync: write failed on "anotherfile.iso": Disc quota exceeded (69)
rsync error: error in file IO (code 11) at receiver.c(378) [receiver=3.2.0dev]
[sender] io timeout after 600 seconds -- exiting
rsync error: timeout in data send/receive (code 30) at io.c(195) [sender=3.2.0dev]

Here we are. What is strange in the above test case is that when a fuzzy basis is selected, even if it's for the second file, the transfer will not abort while sending the first file (as in the second test case), but will hang until IO timeout is reached (or indefinitely if no timeout is set).
As then if socket was not closed by the receiver 😕

Thank you very much for your support 👍

Ben

Rsync doesnt work with zstd compression and fails

I have been testing since Rsync 3.2.0 and even the latest rsync from git today. and it seems when running rsync with compression (zstd), it never seems to finished and errors with error code 10. I have zstd 1.4.5 installed on my system.
Using the parameters below. (not full command)

sudo -E /usr/bin/rsync -az --zc zstd -e 'ssh -CA -p 22 -oStrictHostKeyChecking=no -oPasswordAuthentication=no -oBatchMode=yes -oUserKnownHostsFile=/dev/null' --rsync-path='sudo rsync' --progress -vv --force --ignore-errors --delete-excluded --delete --numeric-ids --no-specials --no-devices --timeout=3600 --prune-empty-dirs

It always fails with

rsync: [generator] write error: Broken pipe (32)
rsync error: error in socket IO (code 10) at io.c(817) [generator=3.2.2pre3]

However, if I try to force the compression to zlib (--zc zlib) or disable compression, it seems to work. Not sure if by default it's using zstd between both ends.

Patch hfs-compression.diff errors

Patch hfs-compression.diff errors
macOS 10.15.6

attrs.c:274:1: error: unknown type name '$$$ERROR$$$'
$$$ERROR$$$ the old patch needs reworking since rsync_xal_get() has tota...
^
xattrs.c:274:16: error: expected ';' after top level declarator
$$$ERROR$$$ the old patch needs reworking since rsync_xal_get() has tota...
^
;
xattrs.c:1365:2: error: #endif without #if

Allow per-directory relative paths for --compare-dest, --copy-dest and --link-dest

After a bit of confusion, I've realised that options such as --compare-dest given a relative path are relative to the root of the rsync destination, rather than as I had mistakenly assumed relative to the current target directory of a scan, i.e- rsync --link-dest=baz src/ dest/ for foo/bar will search dest/baz/foo/bar rather than dest/foo/bar/baz as I had thought, which makes this relative form a bit less useful.

What I would like to see is essentially the same behaviour as per-directory merge rules, whereby a path is relative to each directory scanned. This would allow some useful new searching options.

For example, if you have a hierarchy of TV shows, and the current structure has every episode in a single directory, and you later restructure this with per-season sub-directories (i.e- missing files are one place deeper in the hierarchy), then you could potentially avoid a large amount of re-copying by doing something like:

rsync -a --delete-delay --link-dest=': ..' src/ dest/

i.e- for each missing file, rsync will also try looking within the immediately enclosing folder, even better if combined with --fuzzy (at a potential cost to scan speed). If your files might have gone two or three levels deeper then you could also add --link-dest=': ../..' --link-dest=': ../../..' and so-on.

I'm undecided on what the best way to implement this as an option would be. There are several alternatives I can think of off the top of my head:

• Prefix: As used in my examples, I went with a prefix to the path of : since this would be consistent with per-directory merge rules. Downside to this is that colons are valid characters on some file-systems, so while the chance of collision with a genuine root-relative path is low it's not impossible. The only other prefix I can think of would be / but it wouldn't have the consistency factor with merge rules, and is even less intuitive.
• New Options: The overkill solution is a new option for each of the current --*-dest options, e.g- --compare-dest-per-dir, --copy-dest-per-dir and --link-dest-per-dir. Would work, but would be three new options.
• --search-enclosing=n: One new option specifying the number of enclosing directories relative to the current directory to search. Default of 0 is current behaviour (current directory only), 1 means search parent as well (in both destination and --*-dest alternatives), 3 means search up to three levels higher and so-on. This only covers the use-case of scanning enclosing though, and might restrict usefulness.
• Flags: Rather than full new options (as per New Options above) it might be possible to instead use one or two flag options that change behaviour. For example --dir-relative-dest would tell rsync to treat all --*-dest options with relative paths as being per-directory, rather than relative to destination root. A second option --both-relative-dest would have rsync try both root-relative and per-directory relative paths.

FreeBSD patches

Hi,

In current FreeBSD rsync port version (3.1.3), there are various patches.

Here they are, so that we can decide whether or not they should be included here.

extrapatch-main.c, included when patches/fileflags.diff is enabled :
extrapatch-main.c.txt

patch-siginfo, whichis , as per my findings, never included, could be an old file in the port archive...
patch-siginfo.txt

extrapatch-acl, included when patches/acls.diff is enabled :
extrapatch-acl.txt

Will be useful to @rosorio, the package maintainer.

Thank you 👍

Windows Binaries Pretty Please?

Hello @WayneD ,

I wish to appeal to you for official rsync Windows binaries, to be included with the stable releases available on this GitHub. Rsync is a great tool, but difficult to access for us Windows users without pre-compiled binaries or package managers.

Thanks in advance for your consideration.

11eb67eec9b4d990ae4df680cf7db77dad1b8630 broke --bwlimit=0

according the to man page (https://github.com/WayneD/rsync/blob/master/rsync.1.md) a value of 0 passed to --bwlimit means unlimited. 11eb67e broke that behavior ass seen here:

 $ rsync  --bwlimit=0  /source/ /dest/
 rsync: --bwlimit value is too small: 0
 rsync error: syntax or usage error (code 1) at main.c(1694) [client=3.2.2]