Coder Social home page Coder Social logo

openrsync's People

Contributors

kristapsdz avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

openrsync's Issues

openrsync on uLinux

Hi,

I've successfully built openrsync for for uLinux. Here is how:

# pwd
/root/openrsync
# ls -lah
total 24K
drwxr-xr-x    2 root     root        4.0K Apr 19 07:08 .
drwxr-xr-x   11 root     root        4.0K Apr 19 06:51 ..
-rw-r--r--    1 root     root         193 Apr 19 07:03 .checksum
-rw-r--r--    1 root     root         156 Apr 19 07:08 .manifest
-rw-r--r--    1 root     root         635 Apr 19 07:06 Pkgfile
-rw-r--r--    1 root     root         674 Apr 19 07:08 configure.patch
# cat *
#!/bin/sh
# shellcheck disable=SC2034

# Description: Utility for incremental file transfers over networks
# URL:         https://rsync.samba.org
# Maintainer:  James Mills, prologic at shortcircuit dot net dot au

name=openrsync
version=20200330
gitver=8b612161bd6c251485725af08d352ae1a7d82eca
release=1
source="https://github.com/kristapsdz/openrsync/archive/$gitver.zip
configure.patch"

build () {
  cd $name-$gitver || exit 1

  patch -i "$SRC"/configure.patch

  ./configure \
    PREFIX=/usr \
    DESTDIR="$PKG"

  make -j "$(nproc)"
  make DESTDIR="$PKG" install

  ln -s $name "$PKG"/usr/bin/rsync

  rm -rf "$PKG"/usr/man
}
--- configure.orig
+++ configure
@@ -89,21 +89,7 @@
 # It does have gcc, so try that instead.
 # Prefer clang, though.

-which ${CC} 2>/dev/null 1>&2 || {
-	echo "${CC} not found: trying clang" 1>&2
-	echo "${CC} not found: trying clang" 1>&3
-	CC=clang
-	which ${CC} 2>/dev/null 1>&2 || {
-		echo "${CC} not found: trying gcc" 1>&2
-		echo "${CC} not found: trying gcc" 1>&3
-		CC=gcc
-		which ${CC} 2>/dev/null 1>&2 || {
-			echo "gcc not found: giving up" 1>&2
-			echo "gcc not found: giving up" 1>&3
-			exit 1
-		}
-	}
-}
+CC=tcc

 #----------------------------------------------------------------------
 # Allow certain variables to be overriden on the command line.
#

Unfortunately I run into some problems "client-side" when doing an rsync from my macOS host to the target server running uLinux with openrsync:

prologic@Jamess-iMac
Sun Apr 19 17:06:35
~/Projects/ulinux.org
 12
$ rsync -avz --delete public/ [email protected]:/var/www/
rsync: -z not supported yet
building file list ... done
404.html
about/index.html
assets/fonts/1506293e.woff
downloader.c:465: error: realloc: Out of memory
receiver.c:424: error: rsync_downloader
server.c:151: error: rsync_receiver
rsync: writefd_unbuffered failed to write 16385 bytes [sender]: Broken pipe (32)
rsync: connection unexpectedly closed (68 bytes received so far) [sender]
rsync error: error in rsync protocol data stream (code 12) at /AppleInternal/BuildRoot/Library/Caches/com.apple.xbs/Sources/rsync/rsync-54/rsync/io.c(453) [sender=2.6.9]

Any ideas what's going on here? is this a problem with rsync on my macOS client or openrsync on the server?
So far I've had to comment out the entire block of code in the ./configure script that tries to detect a CC because it just doesn't work. The CC on uLinux is TinyCC (tcc).

Secondly I run into the following compilation issues:

# make CC=tcc
tcc -g -W -Wall -Wextra -Wmissing-prototypes -Wstrict-prototypes -Wwrite-strings -Wno-unused-parameter   -c -o blocks.o blocks.c
In file included from blocks.c:17:
config.h:119: warning: #warning No suitable endian.h could be found.
In file included from blocks.c:17:
config.h:120: warning: #warning Please e-mail the maintainers with your OS.
tcc -g -W -Wall -Wextra -Wmissing-prototypes -Wstrict-prototypes -Wwrite-strings -Wno-unused-parameter   -c -o client.o client.c
In file included from client.c:17:
config.h:119: warning: #warning No suitable endian.h could be found.
In file included from client.c:17:
config.h:120: warning: #warning Please e-mail the maintainers with your OS.
tcc -g -W -Wall -Wextra -Wmissing-prototypes -Wstrict-prototypes -Wwrite-strings -Wno-unused-parameter   -c -o compats.o compats.c
In file included from compats.c:1:
config.h:119: warning: #warning No suitable endian.h could be found.
In file included from compats.c:1:
config.h:120: warning: #warning Please e-mail the maintainers with your OS.
compats.c:1568: error: #error No getprogname available.
make: *** [<builtin>: compats.o] Error 1
#

And so I'm contact you as it asks :)

Here is the Pkgfile (so far):

Adding portability to FreeBSD

The attached patch addresses most of the issues which prevent openrsync from compiling on FreeBSD. This adds in the necessary include files and works around OpenBSD-specific features such as pledge and unveil.

There are still some linker issues with hashes, but this patch gets us more than halfway to a build on FreeBSD 11.x.

rsync-freebsd-patch.txt

"undefined reference to `major'" on debian 10 buster

On Debian 10 (Buster), I get the following error

root@rpki0:~/openrsync# ./configure
config.log: writing...
configure.local: no (fully automatic configuration)
arc4random: no
b64_ntop: yes (with -lresolv)
capsicum: no
err: yes 
explicit_bzero: yes 
getprogname: no
INFTIM: no
md5: no
memmem: yes 
memrchr: yes 
memset_s: no
PATH_MAX: yes 
pledge: no
program_invocation_short_name: yes 
reallocarray: no
recallocarray: no
sandbox_init: no
seccomp-filter: yes 
SOCK_NONBLOCK: yes 
strlcat: no
strlcpy: no
strndup: yes 
strnlen: yes 
strtonum: no
sys_queue: no
systrace: no
unveil: no
zlib: no
__progname: yes 
config.h: written
Makefile.configure: written
root@rpki0:~/openrsync# make
cc -g -W -Wall -Wextra -Wmissing-prototypes -Wstrict-prototypes -Wwrite-strings -Wno-unused-parameter   -c -o blocks.o blocks.c
cc -g -W -Wall -Wextra -Wmissing-prototypes -Wstrict-prototypes -Wwrite-strings -Wno-unused-parameter   -c -o client.o client.c
cc -g -W -Wall -Wextra -Wmissing-prototypes -Wstrict-prototypes -Wwrite-strings -Wno-unused-parameter   -c -o compats.o compats.c
cc -g -W -Wall -Wextra -Wmissing-prototypes -Wstrict-prototypes -Wwrite-strings -Wno-unused-parameter   -c -o downloader.o downloader.c
cc -g -W -Wall -Wextra -Wmissing-prototypes -Wstrict-prototypes -Wwrite-strings -Wno-unused-parameter   -c -o fargs.o fargs.c
cc -g -W -Wall -Wextra -Wmissing-prototypes -Wstrict-prototypes -Wwrite-strings -Wno-unused-parameter   -c -o flist.o flist.c
In file included from �[01m�[Kflist.c:34�[m�[K:
�[01m�[Kflist.c:�[m�[K In function ‘�[01m�[Kflist_recv�[m�[K’:
�[01m�[Kflist.c:741:4:�[m�[K �[01;35m�[Kwarning: �[m�[Kimplicit declaration of function ‘�[01m�[Kmajor�[m�[K’ [�[01;35m�[K-Wimplicit-function-declaration�[m�[K]
    �[01;35m�[Kmajor�[m�[K(ff->st.rdev), minor(ff->st.rdev));
    �[01;35m�[K^~~~~�[m�[K
�[01m�[Kextern.h:240:45:�[m�[K �[01;36m�[Knote: �[m�[Kin definition of macro ‘�[01m�[KLOG3�[m�[K’
  rsync_log(__FILE__, __LINE__, 2, (_fmt), ##�[01;36m�[K__VA_ARGS__�[m�[K)
                                             �[01;36m�[K^~~~~~~~~~~�[m�[K
�[01m�[Kflist.c:741:24:�[m�[K �[01;35m�[Kwarning: �[m�[Kimplicit declaration of function ‘�[01m�[Kminor�[m�[K’; did you mean ‘�[01m�[Kmknod�[m�[K’? [�[01;35m�[K-Wimplicit-function-declaration�[m�[K]
    major(ff->st.rdev), �[01;35m�[Kminor�[m�[K(ff->st.rdev));
                        �[01;35m�[K^~~~~�[m�[K
�[01m�[Kextern.h:240:45:�[m�[K �[01;36m�[Knote: �[m�[Kin definition of macro ‘�[01m�[KLOG3�[m�[K’
  rsync_log(__FILE__, __LINE__, 2, (_fmt), ##�[01;36m�[K__VA_ARGS__�[m�[K)
                                             �[01;36m�[K^~~~~~~~~~~�[m�[K
cc -g -W -Wall -Wextra -Wmissing-prototypes -Wstrict-prototypes -Wwrite-strings -Wno-unused-parameter   -c -o hash.o hash.c
cc -g -W -Wall -Wextra -Wmissing-prototypes -Wstrict-prototypes -Wwrite-strings -Wno-unused-parameter   -c -o ids.o ids.c
cc -g -W -Wall -Wextra -Wmissing-prototypes -Wstrict-prototypes -Wwrite-strings -Wno-unused-parameter   -c -o io.o io.c
cc -g -W -Wall -Wextra -Wmissing-prototypes -Wstrict-prototypes -Wwrite-strings -Wno-unused-parameter   -c -o log.o log.c
cc -g -W -Wall -Wextra -Wmissing-prototypes -Wstrict-prototypes -Wwrite-strings -Wno-unused-parameter   -c -o md4.o md4.c
cc -g -W -Wall -Wextra -Wmissing-prototypes -Wstrict-prototypes -Wwrite-strings -Wno-unused-parameter   -c -o misc.o misc.c
cc -g -W -Wall -Wextra -Wmissing-prototypes -Wstrict-prototypes -Wwrite-strings -Wno-unused-parameter   -c -o mkpath.o mkpath.c
cc -g -W -Wall -Wextra -Wmissing-prototypes -Wstrict-prototypes -Wwrite-strings -Wno-unused-parameter   -c -o mktemp.o mktemp.c
cc -g -W -Wall -Wextra -Wmissing-prototypes -Wstrict-prototypes -Wwrite-strings -Wno-unused-parameter   -c -o receiver.o receiver.c
cc -g -W -Wall -Wextra -Wmissing-prototypes -Wstrict-prototypes -Wwrite-strings -Wno-unused-parameter   -c -o sender.o sender.c
cc -g -W -Wall -Wextra -Wmissing-prototypes -Wstrict-prototypes -Wwrite-strings -Wno-unused-parameter   -c -o server.o server.c
cc -g -W -Wall -Wextra -Wmissing-prototypes -Wstrict-prototypes -Wwrite-strings -Wno-unused-parameter   -c -o session.o session.c
cc -g -W -Wall -Wextra -Wmissing-prototypes -Wstrict-prototypes -Wwrite-strings -Wno-unused-parameter   -c -o socket.o socket.c
cc -g -W -Wall -Wextra -Wmissing-prototypes -Wstrict-prototypes -Wwrite-strings -Wno-unused-parameter   -c -o symlinks.o symlinks.c
cc -g -W -Wall -Wextra -Wmissing-prototypes -Wstrict-prototypes -Wwrite-strings -Wno-unused-parameter   -c -o uploader.o uploader.c
cc -g -W -Wall -Wextra -Wmissing-prototypes -Wstrict-prototypes -Wwrite-strings -Wno-unused-parameter   -c -o main.o main.c
cc -o openrsync blocks.o client.o compats.o downloader.o fargs.o flist.o hash.o ids.o io.o log.o md4.o misc.o mkpath.o mktemp.o receiver.o sender.o server.o session.o socket.o symlinks.o uploader.o main.o -lm
/usr/bin/ld: flist.o: in function `flist_recv':
/root/openrsync/flist.c:737: undefined reference to `minor'
/usr/bin/ld: /root/openrsync/flist.c:737: undefined reference to `major'
collect2: error: ld returned 1 exit status
make: *** [Makefile:34: openrsync] Error 1
root@rpki0:~/openrsync#

alternative to recallocarray?

Hi, i'm trying to porting this project to FreeBSD. I have an only a problem with "recallocarray". Seems not present in FreeBSD. Any ideas?

【help!!】how to use server-client way in openrsync?

I tried using openrsync to sync files the same way I used ssh, and it worked:

openrsync -av [email protected]:/home/liu/binlog .

However, I want to start an openrsync process like rsync, listen to a port; then use ip:pot and username&password on the client side to synchronize data, but it fails. Can you help me to see how this method is used? Thank you very much!

--delete-delay --delay-updates

Hi, I'm opening this issue just in case you don't know this requirement for snapshot packages mirrors, so you can keep it in mind for future development of openrsync. Due to the way OpenBSD pkg_add works, mirrors are required to use --delete-delay --delay-updates to sync from its source. openrsync will need something equivalent for the use in mirrors.

Feature request: --exclude

I often use --exclude when using rsync(1) to back up a home directories. Example:

rsync \ 
    --exclude "vm/*" \
    --exclude "*.core" \
    --exclude ".gimp-2.8/gimpswap.*" \
    -va $HOME myuser@remote:~/backup

In some cases I can emulate this with openrsync(1) by building a list of directories and running rsync on them individually.

--remove-source-files

Is it possible to add the "--remove-source-files" switch from rsync? It is very usefull especially during migrations and backups + probably quite easy to implement...

Trying to sync from macos rsync to server with openrsync, but delete in combination with exclude arg not working

If I try and use the command as follows it doesn't work. Syntax error apparently.

rsync -av --rsync-path=openrsync --delete --exclude "test.txt" tmp openbsd.server:tmp/
building file list ... done
openrsync: syntax error in received rules
rsync: connection unexpectedly closed (8 bytes received so far) [sender]
rsync error: error in rsync protocol data stream (code 12) at /AppleInternal/Library/BuildRoots/11aa8fb2-5f4b-11ee-bc7f-926038f30c31/Library/Caches/com.apple.xbs/Sources/rsync/rsync/io.c(453) [sender=2.6.9]

if I run it either without --delete, but --exclude arg present, it works. It also works if I include --delete, but not use --exclude. Just together it doesn't want to work.

rsync -av --rsync-path=openrsync --exclude "test.txt" tmp openbsd-server:tmp/
building file list ... done

sent 114 bytes received 16 bytes 86.67 bytes/sec
total size is 12 speedup is 0.09

rsync -av --rsync-path=openrsync --delete tmp openbsd-server:tmp/
building file list ... done

sent 118 bytes received 16 bytes 89.33 bytes/sec
total size is 12 speedup is 0.09

mbufs

sorry to bother you, can you tell what i'm doing wrong?
openrsync -ar --del --timeout=10 /var /mnt/root/
where /var is the directory on the ramdisk(mfs), and /mnt/root is the nfs mount point.
in this case, mbufs grow catastrophically- by more than a thousand, and do not return back.
openrsync --rsync-path rsync -ar --del --timeout=10 /var /mnt/root/
in this case, i don't see any problems(and of course no problems with rsync itself without your wrapper). except for one thing- there is no rsync initially in openbsd(yes, there is some problem with this for me).
by the way, if you run the second variant after the first variant, then mbufs return to the state that was before the first variant %\

bombing out on assert in ids.c

$ uname -a
OpenBSD xyz.inet6.se 6.4 GENERIC.MP#699 amd64
$ ./openrsync -av rsync://ftp.eu.openbsd.org/OpenBSD/6.4/octeon .
assertion "j < idsz" failed: file "ids.c", line 87, function "idents_assign_uid"
Abort trap (core dumped)
...
(gdb) backtrace
#0 thrkill () at -:3
#1 0x00000ecf667b528e in _libc_abort () at /usr/src/lib/libc/stdlib/abort.c:51
#2 0x00000ecf6676bb92 in _libc___assert2 (file=Variable "file" is not available.
)
at /usr/src/lib/libc/gen/assert.c:52
#3 0x00000ecc859b0dce in idents_assign_uid (sess=0x7f7ffffce678,
fl=0xece9bb3a000, flsz=18, ids=0x0, idsz=0) at ids.c:87
#4 0x00000ecc859ad762 in flist_recv (sess=0x7f7ffffce678, fd=3,
flp=0x7f7ffffce5b8, sz=0x7f7ffffce5a8) at flist.c:771
#5 0x00000ecc859b70c6 in rsync_receiver (sess=0x7f7ffffce678, fdin=3,
fdout=3, root=0xecf5a682970 ".") at receiver.c:214
#6 0x00000ecc859bb8af in rsync_socket (opts=0x7f7ffffceb80, f=0xecefd5fd0c0)
at socket.c:417
#7 0x00000ecc859c0e7f in main (argc=2, argv=0x7f7ffffcf048) at main.c:456

Same for the in-obsd not-yet-linked-to-build rsync(1), but I cloned your version to verify if it was the same error.

openrsync raw device

Hello,

Back in 2008 I requested a command switch for rsync to copy raw device data to a file, vs simply re-creating device nodes. The goal being to stage and backup partitions loaded with hard link snapshots, to off site, remote media.

At that time, with dynamic files properly managed, rsync was used to create hard link snapshot backups, at a high frequency, to a local dedicated partition. This solution perfectly fit the requirements, including nominal resource consumption, and short backup interval. However, duplicating the backup partition to off site media was problematic, due to the very large number of links.

We managed by snapshotting hard links of our monthly backups, to a local disk which was then physically transferred to the remote site. Then on a daily interval, a remote hard link snapshot was created, from newest local one. That gave us one daily remote snapshot backup, but not all the local intervals, even though that would have been nominal additional data.

With a switch to copy device node data, offsite snapshot maintenance could be initialized from a r/w mounted device file (original partition) to an unmounted remote device file. After the initial (dirty) transfer, the remote backup partition would be maintained by re-mounting the local snapshot ro, transferring the (clean) device node data (with a large checksum block-size), then re-mounting the local partition r/w, to resume regular local snapshots.

This process could be repeated at least once per day (possibly more often) to make all of the local snapshots available remotely.

When I made the original feature request, a rsync developer provided a patch, https://lists.samba.org/archive/rsync/2008-March/020573.html but when I followed up later, I learned it wouldn't make it into the regular release cycle because it didn't have a sponsor...

This would be a great feature and I believe simple to implement? Maybe openrsync could support it?

-George

--inplace option

Hi Kristaps,

I hope you will read this on one of your favorite diving places :-). I would like to bring to your attention a "missing" switch which will be worth adding. Let me explain a bit. The easiest way to backup an OpenBSD machine onto an advanced files system like HAMMER, HAMMER2, or ZFS is to just rsync to it. HAMMER history or ZFS snapshot will take care of the rest. However, unless the switch --inplace is used HAMMER will recreate the file and the history will be lost. Please refer for the contest

HAMMER history friendly backup tool

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.