kristapsdz / openrsync Goto Github PK
View Code? Open in Web Editor NEWBSD-licensed implementation of rsync
License: ISC License
BSD-licensed implementation of rsync
License: ISC License
Hi,
I've successfully built openrsync for for uLinux. Here is how:
# pwd
/root/openrsync
# ls -lah
total 24K
drwxr-xr-x 2 root root 4.0K Apr 19 07:08 .
drwxr-xr-x 11 root root 4.0K Apr 19 06:51 ..
-rw-r--r-- 1 root root 193 Apr 19 07:03 .checksum
-rw-r--r-- 1 root root 156 Apr 19 07:08 .manifest
-rw-r--r-- 1 root root 635 Apr 19 07:06 Pkgfile
-rw-r--r-- 1 root root 674 Apr 19 07:08 configure.patch
# cat *
#!/bin/sh
# shellcheck disable=SC2034
# Description: Utility for incremental file transfers over networks
# URL: https://rsync.samba.org
# Maintainer: James Mills, prologic at shortcircuit dot net dot au
name=openrsync
version=20200330
gitver=8b612161bd6c251485725af08d352ae1a7d82eca
release=1
source="https://github.com/kristapsdz/openrsync/archive/$gitver.zip
configure.patch"
build () {
cd $name-$gitver || exit 1
patch -i "$SRC"/configure.patch
./configure \
PREFIX=/usr \
DESTDIR="$PKG"
make -j "$(nproc)"
make DESTDIR="$PKG" install
ln -s $name "$PKG"/usr/bin/rsync
rm -rf "$PKG"/usr/man
}
--- configure.orig
+++ configure
@@ -89,21 +89,7 @@
# It does have gcc, so try that instead.
# Prefer clang, though.
-which ${CC} 2>/dev/null 1>&2 || {
- echo "${CC} not found: trying clang" 1>&2
- echo "${CC} not found: trying clang" 1>&3
- CC=clang
- which ${CC} 2>/dev/null 1>&2 || {
- echo "${CC} not found: trying gcc" 1>&2
- echo "${CC} not found: trying gcc" 1>&3
- CC=gcc
- which ${CC} 2>/dev/null 1>&2 || {
- echo "gcc not found: giving up" 1>&2
- echo "gcc not found: giving up" 1>&3
- exit 1
- }
- }
-}
+CC=tcc
#----------------------------------------------------------------------
# Allow certain variables to be overriden on the command line.
#
Unfortunately I run into some problems "client-side" when doing an rsync from my macOS host to the target server running uLinux with openrsync:
prologic@Jamess-iMac
Sun Apr 19 17:06:35
~/Projects/ulinux.org
12
$ rsync -avz --delete public/ [email protected]:/var/www/
rsync: -z not supported yet
building file list ... done
404.html
about/index.html
assets/fonts/1506293e.woff
downloader.c:465: error: realloc: Out of memory
receiver.c:424: error: rsync_downloader
server.c:151: error: rsync_receiver
rsync: writefd_unbuffered failed to write 16385 bytes [sender]: Broken pipe (32)
rsync: connection unexpectedly closed (68 bytes received so far) [sender]
rsync error: error in rsync protocol data stream (code 12) at /AppleInternal/BuildRoot/Library/Caches/com.apple.xbs/Sources/rsync/rsync-54/rsync/io.c(453) [sender=2.6.9]
Any ideas what's going on here? is this a problem with rsync
on my macOS client or openrsync
on the server?
So far I've had to comment out the entire block of code in the ./configure
script that tries to detect a CC because it just doesn't work. The CC on uLinux is TinyCC (tcc
).
Secondly I run into the following compilation issues:
# make CC=tcc
tcc -g -W -Wall -Wextra -Wmissing-prototypes -Wstrict-prototypes -Wwrite-strings -Wno-unused-parameter -c -o blocks.o blocks.c
In file included from blocks.c:17:
config.h:119: warning: #warning No suitable endian.h could be found.
In file included from blocks.c:17:
config.h:120: warning: #warning Please e-mail the maintainers with your OS.
tcc -g -W -Wall -Wextra -Wmissing-prototypes -Wstrict-prototypes -Wwrite-strings -Wno-unused-parameter -c -o client.o client.c
In file included from client.c:17:
config.h:119: warning: #warning No suitable endian.h could be found.
In file included from client.c:17:
config.h:120: warning: #warning Please e-mail the maintainers with your OS.
tcc -g -W -Wall -Wextra -Wmissing-prototypes -Wstrict-prototypes -Wwrite-strings -Wno-unused-parameter -c -o compats.o compats.c
In file included from compats.c:1:
config.h:119: warning: #warning No suitable endian.h could be found.
In file included from compats.c:1:
config.h:120: warning: #warning Please e-mail the maintainers with your OS.
compats.c:1568: error: #error No getprogname available.
make: *** [<builtin>: compats.o] Error 1
#
And so I'm contact you as it asks :)
Here is the Pkgfile
(so far):
The attached patch addresses most of the issues which prevent openrsync from compiling on FreeBSD. This adds in the necessary include files and works around OpenBSD-specific features such as pledge and unveil.
There are still some linker issues with hashes, but this patch gets us more than halfway to a build on FreeBSD 11.x.
On Debian 10 (Buster), I get the following error
root@rpki0:~/openrsync# ./configure
config.log: writing...
configure.local: no (fully automatic configuration)
arc4random: no
b64_ntop: yes (with -lresolv)
capsicum: no
err: yes
explicit_bzero: yes
getprogname: no
INFTIM: no
md5: no
memmem: yes
memrchr: yes
memset_s: no
PATH_MAX: yes
pledge: no
program_invocation_short_name: yes
reallocarray: no
recallocarray: no
sandbox_init: no
seccomp-filter: yes
SOCK_NONBLOCK: yes
strlcat: no
strlcpy: no
strndup: yes
strnlen: yes
strtonum: no
sys_queue: no
systrace: no
unveil: no
zlib: no
__progname: yes
config.h: written
Makefile.configure: written
root@rpki0:~/openrsync# make
cc -g -W -Wall -Wextra -Wmissing-prototypes -Wstrict-prototypes -Wwrite-strings -Wno-unused-parameter -c -o blocks.o blocks.c
cc -g -W -Wall -Wextra -Wmissing-prototypes -Wstrict-prototypes -Wwrite-strings -Wno-unused-parameter -c -o client.o client.c
cc -g -W -Wall -Wextra -Wmissing-prototypes -Wstrict-prototypes -Wwrite-strings -Wno-unused-parameter -c -o compats.o compats.c
cc -g -W -Wall -Wextra -Wmissing-prototypes -Wstrict-prototypes -Wwrite-strings -Wno-unused-parameter -c -o downloader.o downloader.c
cc -g -W -Wall -Wextra -Wmissing-prototypes -Wstrict-prototypes -Wwrite-strings -Wno-unused-parameter -c -o fargs.o fargs.c
cc -g -W -Wall -Wextra -Wmissing-prototypes -Wstrict-prototypes -Wwrite-strings -Wno-unused-parameter -c -o flist.o flist.c
In file included from �[01m�[Kflist.c:34�[m�[K:
�[01m�[Kflist.c:�[m�[K In function ‘�[01m�[Kflist_recv�[m�[K’:
�[01m�[Kflist.c:741:4:�[m�[K �[01;35m�[Kwarning: �[m�[Kimplicit declaration of function ‘�[01m�[Kmajor�[m�[K’ [�[01;35m�[K-Wimplicit-function-declaration�[m�[K]
�[01;35m�[Kmajor�[m�[K(ff->st.rdev), minor(ff->st.rdev));
�[01;35m�[K^~~~~�[m�[K
�[01m�[Kextern.h:240:45:�[m�[K �[01;36m�[Knote: �[m�[Kin definition of macro ‘�[01m�[KLOG3�[m�[K’
rsync_log(__FILE__, __LINE__, 2, (_fmt), ##�[01;36m�[K__VA_ARGS__�[m�[K)
�[01;36m�[K^~~~~~~~~~~�[m�[K
�[01m�[Kflist.c:741:24:�[m�[K �[01;35m�[Kwarning: �[m�[Kimplicit declaration of function ‘�[01m�[Kminor�[m�[K’; did you mean ‘�[01m�[Kmknod�[m�[K’? [�[01;35m�[K-Wimplicit-function-declaration�[m�[K]
major(ff->st.rdev), �[01;35m�[Kminor�[m�[K(ff->st.rdev));
�[01;35m�[K^~~~~�[m�[K
�[01m�[Kextern.h:240:45:�[m�[K �[01;36m�[Knote: �[m�[Kin definition of macro ‘�[01m�[KLOG3�[m�[K’
rsync_log(__FILE__, __LINE__, 2, (_fmt), ##�[01;36m�[K__VA_ARGS__�[m�[K)
�[01;36m�[K^~~~~~~~~~~�[m�[K
cc -g -W -Wall -Wextra -Wmissing-prototypes -Wstrict-prototypes -Wwrite-strings -Wno-unused-parameter -c -o hash.o hash.c
cc -g -W -Wall -Wextra -Wmissing-prototypes -Wstrict-prototypes -Wwrite-strings -Wno-unused-parameter -c -o ids.o ids.c
cc -g -W -Wall -Wextra -Wmissing-prototypes -Wstrict-prototypes -Wwrite-strings -Wno-unused-parameter -c -o io.o io.c
cc -g -W -Wall -Wextra -Wmissing-prototypes -Wstrict-prototypes -Wwrite-strings -Wno-unused-parameter -c -o log.o log.c
cc -g -W -Wall -Wextra -Wmissing-prototypes -Wstrict-prototypes -Wwrite-strings -Wno-unused-parameter -c -o md4.o md4.c
cc -g -W -Wall -Wextra -Wmissing-prototypes -Wstrict-prototypes -Wwrite-strings -Wno-unused-parameter -c -o misc.o misc.c
cc -g -W -Wall -Wextra -Wmissing-prototypes -Wstrict-prototypes -Wwrite-strings -Wno-unused-parameter -c -o mkpath.o mkpath.c
cc -g -W -Wall -Wextra -Wmissing-prototypes -Wstrict-prototypes -Wwrite-strings -Wno-unused-parameter -c -o mktemp.o mktemp.c
cc -g -W -Wall -Wextra -Wmissing-prototypes -Wstrict-prototypes -Wwrite-strings -Wno-unused-parameter -c -o receiver.o receiver.c
cc -g -W -Wall -Wextra -Wmissing-prototypes -Wstrict-prototypes -Wwrite-strings -Wno-unused-parameter -c -o sender.o sender.c
cc -g -W -Wall -Wextra -Wmissing-prototypes -Wstrict-prototypes -Wwrite-strings -Wno-unused-parameter -c -o server.o server.c
cc -g -W -Wall -Wextra -Wmissing-prototypes -Wstrict-prototypes -Wwrite-strings -Wno-unused-parameter -c -o session.o session.c
cc -g -W -Wall -Wextra -Wmissing-prototypes -Wstrict-prototypes -Wwrite-strings -Wno-unused-parameter -c -o socket.o socket.c
cc -g -W -Wall -Wextra -Wmissing-prototypes -Wstrict-prototypes -Wwrite-strings -Wno-unused-parameter -c -o symlinks.o symlinks.c
cc -g -W -Wall -Wextra -Wmissing-prototypes -Wstrict-prototypes -Wwrite-strings -Wno-unused-parameter -c -o uploader.o uploader.c
cc -g -W -Wall -Wextra -Wmissing-prototypes -Wstrict-prototypes -Wwrite-strings -Wno-unused-parameter -c -o main.o main.c
cc -o openrsync blocks.o client.o compats.o downloader.o fargs.o flist.o hash.o ids.o io.o log.o md4.o misc.o mkpath.o mktemp.o receiver.o sender.o server.o session.o socket.o symlinks.o uploader.o main.o -lm
/usr/bin/ld: flist.o: in function `flist_recv':
/root/openrsync/flist.c:737: undefined reference to `minor'
/usr/bin/ld: /root/openrsync/flist.c:737: undefined reference to `major'
collect2: error: ld returned 1 exit status
make: *** [Makefile:34: openrsync] Error 1
root@rpki0:~/openrsync#
Hi, i'm trying to porting this project to FreeBSD. I have an only a problem with "recallocarray". Seems not present in FreeBSD. Any ideas?
I tried using openrsync to sync files the same way I used ssh, and it worked:
openrsync -av [email protected]:/home/liu/binlog .
However, I want to start an openrsync process like rsync, listen to a port; then use ip:pot and username&password on the client side to synchronize data, but it fails. Can you help me to see how this method is used? Thank you very much!
Hi, I'm opening this issue just in case you don't know this requirement for snapshot packages mirrors, so you can keep it in mind for future development of openrsync. Due to the way OpenBSD pkg_add works, mirrors are required to use --delete-delay --delay-updates
to sync from its source. openrsync will need something equivalent for the use in mirrors.
I often use --exclude
when using rsync(1) to back up a home directories. Example:
rsync \
--exclude "vm/*" \
--exclude "*.core" \
--exclude ".gimp-2.8/gimpswap.*" \
-va $HOME myuser@remote:~/backup
In some cases I can emulate this with openrsync(1) by building a list of directories and running rsync on them individually.
Is it possible to add the "--remove-source-files" switch from rsync? It is very usefull especially during migrations and backups + probably quite easy to implement...
If I try and use the command as follows it doesn't work. Syntax error apparently.
rsync -av --rsync-path=openrsync --delete --exclude "test.txt" tmp openbsd.server:tmp/
building file list ... done
openrsync: syntax error in received rules
rsync: connection unexpectedly closed (8 bytes received so far) [sender]
rsync error: error in rsync protocol data stream (code 12) at /AppleInternal/Library/BuildRoots/11aa8fb2-5f4b-11ee-bc7f-926038f30c31/Library/Caches/com.apple.xbs/Sources/rsync/rsync/io.c(453) [sender=2.6.9]
if I run it either without --delete, but --exclude arg present, it works. It also works if I include --delete, but not use --exclude. Just together it doesn't want to work.
rsync -av --rsync-path=openrsync --exclude "test.txt" tmp openbsd-server:tmp/
building file list ... done
sent 114 bytes received 16 bytes 86.67 bytes/sec
total size is 12 speedup is 0.09
rsync -av --rsync-path=openrsync --delete tmp openbsd-server:tmp/
building file list ... done
sent 118 bytes received 16 bytes 89.33 bytes/sec
total size is 12 speedup is 0.09
sorry to bother you, can you tell what i'm doing wrong?
openrsync -ar --del --timeout=10 /var /mnt/root/
where /var is the directory on the ramdisk(mfs), and /mnt/root is the nfs mount point.
in this case, mbufs grow catastrophically- by more than a thousand, and do not return back.
openrsync --rsync-path rsync -ar --del --timeout=10 /var /mnt/root/
in this case, i don't see any problems(and of course no problems with rsync itself without your wrapper). except for one thing- there is no rsync initially in openbsd(yes, there is some problem with this for me).
by the way, if you run the second variant after the first variant, then mbufs return to the state that was before the first variant %\
$ uname -a
OpenBSD xyz.inet6.se 6.4 GENERIC.MP#699 amd64
$ ./openrsync -av rsync://ftp.eu.openbsd.org/OpenBSD/6.4/octeon .
assertion "j < idsz" failed: file "ids.c", line 87, function "idents_assign_uid"
Abort trap (core dumped)
...
(gdb) backtrace
#0 thrkill () at -:3
#1 0x00000ecf667b528e in _libc_abort () at /usr/src/lib/libc/stdlib/abort.c:51
#2 0x00000ecf6676bb92 in _libc___assert2 (file=Variable "file" is not available.
)
at /usr/src/lib/libc/gen/assert.c:52
#3 0x00000ecc859b0dce in idents_assign_uid (sess=0x7f7ffffce678,
fl=0xece9bb3a000, flsz=18, ids=0x0, idsz=0) at ids.c:87
#4 0x00000ecc859ad762 in flist_recv (sess=0x7f7ffffce678, fd=3,
flp=0x7f7ffffce5b8, sz=0x7f7ffffce5a8) at flist.c:771
#5 0x00000ecc859b70c6 in rsync_receiver (sess=0x7f7ffffce678, fdin=3,
fdout=3, root=0xecf5a682970 ".") at receiver.c:214
#6 0x00000ecc859bb8af in rsync_socket (opts=0x7f7ffffceb80, f=0xecefd5fd0c0)
at socket.c:417
#7 0x00000ecc859c0e7f in main (argc=2, argv=0x7f7ffffcf048) at main.c:456
Same for the in-obsd not-yet-linked-to-build rsync(1), but I cloned your version to verify if it was the same error.
Hello,
Back in 2008 I requested a command switch for rsync to copy raw device data to a file, vs simply re-creating device nodes. The goal being to stage and backup partitions loaded with hard link snapshots, to off site, remote media.
At that time, with dynamic files properly managed, rsync was used to create hard link snapshot backups, at a high frequency, to a local dedicated partition. This solution perfectly fit the requirements, including nominal resource consumption, and short backup interval. However, duplicating the backup partition to off site media was problematic, due to the very large number of links.
We managed by snapshotting hard links of our monthly backups, to a local disk which was then physically transferred to the remote site. Then on a daily interval, a remote hard link snapshot was created, from newest local one. That gave us one daily remote snapshot backup, but not all the local intervals, even though that would have been nominal additional data.
With a switch to copy device node data, offsite snapshot maintenance could be initialized from a r/w mounted device file (original partition) to an unmounted remote device file. After the initial (dirty) transfer, the remote backup partition would be maintained by re-mounting the local snapshot ro, transferring the (clean) device node data (with a large checksum block-size), then re-mounting the local partition r/w, to resume regular local snapshots.
This process could be repeated at least once per day (possibly more often) to make all of the local snapshots available remotely.
When I made the original feature request, a rsync developer provided a patch, https://lists.samba.org/archive/rsync/2008-March/020573.html but when I followed up later, I learned it wouldn't make it into the regular release cycle because it didn't have a sponsor...
This would be a great feature and I believe simple to implement? Maybe openrsync could support it?
-George
Hi Kristaps,
I hope you will read this on one of your favorite diving places :-). I would like to bring to your attention a "missing" switch which will be worth adding. Let me explain a bit. The easiest way to backup an OpenBSD machine onto an advanced files system like HAMMER, HAMMER2, or ZFS is to just rsync to it. HAMMER history or ZFS snapshot will take care of the rest. However, unless the switch --inplace is used HAMMER will recreate the file and the history will be lost. Please refer for the contest
These are the changes needed to make the program build and run on linux
Is it possible (within reasonable bounds) to include support of bi-directional synchronisation, akin to Unison?
Is it possible to support the sender mode for a client? This is a basic function. Thanks.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.