dooblem / bsync Goto Github PK
View Code? Open in Web Editor NEWBidirectional Synchronization using Rsync
License: GNU General Public License v3.0
Bidirectional Synchronization using Rsync
License: GNU General Public License v3.0
First let me thank you for putting together this script as there are very few tools for devices with embed Linux OS like NAS or Openwrt routers which don't have sufficient resources as normal Linux distros .
I gave a try to sync my NAS with my openwrt router which has external disk connected to it.
Immediately I notice that I bumped in the issue of hard coded binary references in the script.
As on my NAS where the script is running the paths are the following ones:
Linux 2.6.12.6-VENUS mips GNU/Linux
/tmp/usbmounts/sda1/root # which python3
/opt/bin/python3
/tmp/usbmounts/sda1/root # which find
/opt/bin/find
/tmp/usbmounts/sda1/root # which rsync
/opt/bin/rsync
My router with Opewrt Barrier Braker on it:
Linux 3.10.49 mips GNU/Linux
root@DIR:~# which rsync
/usr/bin/rsync
root@DIR:~# which find
/usr/bin/find
Resulting with EM: remote find not found
I think it would be a nice approach to modify the script so that it runs:
which find
which rsync
which gfind
and use the results to assign the full paths for this binaries to the corresponding variables
or if they don't exist to throw the proper EM to ask user to install namely the missing package(s).
I also did notice that the find provided by the openwrt busybox doesn't recognize the parameter -printf
find: unrecognized: -printf
BusyBox v1.22.1 (2015-03-28 06:57:02 CET) multi-call binary.
Usage: find [-HL] [PATH]... [OPTIONS] [ACTIONS]
Search for files and perform actions on them.
First failed action stops processing of current file.
Defaults: PATH is current directory, action is '-print'
-L,-follow Follow symlinks
-H ...on command line only
-xdev Don't descend directories on other filesystems
-maxdepth N Descend at most N levels. -maxdepth 0 applies
actions to command line arguments only
-mindepth N Don't act on first N levels
-depth Act on directory *after* traversing it
Actions:
( ACTIONS ) Group actions for -o / -a
! ACT Invert ACT's success/failure
ACT1 [-a] ACT2 If ACT1 fails, stop, else do ACT2
ACT1 -o ACT2 If ACT1 succeeds, stop, else do ACT2
Note: -a has higher priority than -o
-name PATTERN Match file name (w/o directory name) to PATTERN
-iname PATTERN Case insensitive -name
-path PATTERN Match path to PATTERN
-ipath PATTERN Case insensitive -path
-regex PATTERN Match path to regex PATTERN
-type X File type is X (one of: f,d,l,b,c,...)
-perm MASK At least one mask bit (+MASK), all bits (-MASK),
or exactly MASK bits are set in file's mode
-mtime DAYS mtime is greater than (+N), less than (-N),
or exactly N days in the past
-user NAME/ID File is owned by given user
-group NAME/ID File is owned by given group
-size N[bck] File size is N (c:bytes,k:kbytes,b:512 bytes(def.))
+/-N: file size is bigger/smaller than N
-prune If current file is directory, don't descend into it
If none of the following actions is specified, -print is assumed
-print Print file name
-print0 Print file name, NUL terminated
-exec CMD ARG ; Run CMD with all instances of {} replaced by
file name. Fails if CMD exits with nonzero
But openwrt has separate command
root@DIR:~# printf
ash: usage: printf FORMAT [ARGUMENT...]
Any way to do the same task via it?
Thanks you.
Hi,
I'd like to use bsync to sync files on my hoster, but my hoster only allows me to transfer files using FTPS, not SFTP.
Unfortunately, on the attempt to connect I get this error message, so I assume FTPS is not supported:
$ ./bsync www ftps://username:[email protected]
ssh: Could not resolve hostname ftps: nodename nor servname provided, or not known
Error: could not open SSH connection.
I'd love to see this being implemented, or at least, get a suitable alternative that is a binary that requires no installation whatsoever.
I've tried
*.lock and *.sr.lock and .lock and .sr.lock
but it still tries to sync them, any help?
As I remember from Unison, mount status of the directories (recursively) must be checked before (or even during?) synchronizing. If there is something different, I must be warned seriously (for example force user to type: "yes i know") if there is a different or missing mount entry either in localhost or remote host.
Because this problem leads unwanted erase of subfolders.
Example:
We have DIR1 and DIR2 to sync.
DIR1/aaa/bbb is mounted via sshfs from a remote machine, or with "-o bind" option from localhost.
Sync the two directories.
Umount DIR1/aaa/bbb (it is unmounted because a network/cable connection error)
Sync again: Problem.
We should be warned about that.
E.g. I have a ~/.ssh/config set up so:
ControlMaster auto
ControlPath /tmp/ssh_mux_%h_%p_%r
ControlPersist 1h
ForwardAgent yes
Host rs
Hostname remoteserverip
User myusername
It does use the host shortcut I set up, but it doesn't respect/utilize the control persist I am using. This should use the existing connection instead of creating a new one and requesting my password every time.
Hello,
Im trying to sync a folder over a ssh connection and a local folder but I get the following error: unix listener too long for Unix domain socket
I tried to apply the fix suggested there : https://gist.github.com/andyvanee/bcf95b1044b80e72b4a42933549a079b without success
Any suggestion ?
What should I understand from that error message:
Loading filelists...
Error: difference in snaps for path: bin/firefox.d
Dear Sirs,
is it possible at .bsync-ignore file to have wildcard character in the filename e.x.
dovecot.*
in order to get ignored all dovecot.index , dovecot.index.log etc files?
Thanks
Was testing the most recent version and got the following error:
Traceback (most recent call last):
File "/home/reagle/bin/bsync", line 866, in <module>
show_conflict(f1, f2, path)
File "/home/reagle/bin/bsync", line 479, in show_conflict
p1 = getfilemode(f1.type, f1.perms)+" "+f1.size+"B ("+getdatestr(f1)+")"
File "/home/reagle/bin/bsync", line 469, in getfilemode
return type + stat.filemode(int(perms, 8))[1:]
ValueError: invalid literal for int() with base 8: ''
it will be nice to have bsync as a daemon and then using rest api to fix conflits and use FS notify to auto detect changes and sync files almost after file is just closed.
Hi.
Considered that is difficult to add support for ACL synchronization to bsync, I propose to add support for running an external script between apply_small_actions and apply_rsync_actions.
The external script could take care of ACL synchronization, and if I'm not wrong the best moment to do it is between the fast moves and deletion (apply_small_actions) and the possible slow rsync operations, to reduce the time window where ACL are not synchronized.
If using --acls rsync argument, is not needed to resync ACL after rsync operations.
The external script could be specified by a command line option -s, for example.
If you agree, I can branch and propose a pull request.
(I did now another pull request for batch mode and autoanswer, as discussed in another thread)
Stefano
Rsync doesn't delete removed file by default in the second destination, so why bsync can't do it?
When you remove unused files on your laptop that could be fine if there is always a copy on the server.
desired behavior:
If a flag is set (ex: -k for "keep"), removed files are wrote in .bsync-ignore & withdrawn from .bsync-snap-*.
You can retrieve later local removed files by deleting them from .bsync-ignore.
From Cerem Cem ASLAN idea :
Any solutions? I tried just syncing from local to remote. I don't have issues with rsync, but, I need the bidirectional sync.
$ ./bsync ./from-folder [email protected]:/home/server/
unix_listener: path "/var/folders/nk/pmcqhz_d6219kzk1df42591m0000gn/T/tmprkyksx7e/[email protected]:22.abuSqvTY1vWqQvTP" too long for Unix domain socket
Error: could not open SSH connection.
According to my expectations, running bsync with sudo must keep permissions and ownership just like "cp -a".
when i use the script for my home folder on linux , it copies all .file (hidden files) ,if you add this option to ignore hidden files , i would be very happy.
Problem can not be clearly understood:
ceremcem@cca-erik:/media/ceremcem/zfs_zencefil$ bsync ./-.-\ cca-home+/ /media/ceremcem/7c32ca7d-fb9e-4c0b-8e89-6de320908b8b/cca-sync/-.-\ cca-home+/
Loading filelists...
Old filelist not found. Starting with empty history.
Traceback (most recent call last):
File "/home/ceremcem/bin/bsync", line 779, in <module>
dir2 = load_dir(ssh2, dir2name, ignores)
File "/home/ceremcem/bin/bsync", line 403, in load_dir
record = read_file_record(gen)
File "/home/ceremcem/bin/bsync", line 307, in read_file_record
i,p,t,d,s,perms = next(gen),next(gen),next(gen),next(gen),next(gen),next(gen)
File "/home/ceremcem/bin/bsync", line 294, in fileLineIter
charsJustRead = inputFile.read(readSize)
File "/usr/lib/python3.3/codecs.py", line 313, in decode
(result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x8d in position 1277: invalid start byte
ho to edit the .bsync-ignore File to exclude all hits of a directory in the complete subtree.
For example to exclude all directories that match "dir1"
*/dir1
does not work...
thanks
If I type find I get a find. It looks like I have one installed via Brew.
/u/l/bin ❯❯❯ ls -la find
lrwxr-xr-x 1 tom access_bpf 34 2 Sep 2016 find -> ../Cellar/findutils/4.6.0/bin/find
I ask before when I try to run bsync I get error:
zsh: /usr/local/bin/bsync: bad interpreter: /usr/bin/python3: no such file or directory
/u/l/bin ❯❯❯ brew install python3 ✘ 127
Error: python 2.7.14_3 is already installed
To upgrade to 3.6.5, run `brew upgrade python`
/u/l/bin ❯❯❯ bsync
zsh: /usr/local/bin/bsync: bad interpreter: /usr/bin/python3: no such file or directory
[[email protected]:/usr/local/bin]
/u/l/bin ❯❯❯ python ✘ 127
Python 2.7.10 (default, Feb 7 2017, 00:08:15)
[GCC 4.2.1 Compatible Apple LLVM 8.0.0 (clang-800.0.34)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>>
KeyboardInterrupt
>>> exit
Use exit() or Ctrl-D (i.e. EOF) to exit
>>> ^D
[[email protected]:/usr/local/bin]
/u/l/bin ❯❯❯ python3
Python 3.6.5 (default, Mar 30 2018, 06:42:10)
[GCC 4.2.1 Compatible Apple LLVM 9.0.0 (clang-900.0.39.2)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>>
I'm trying to remove folders called node_modules and .npm from my backup. However bsync always wants to copy them across?!
Especially keen to know please, can I use:
# for comments?
$ and ^ for start and end of line like in regex?
? for a single char random
* for any char random in this path?
/ naturally this delimits the directories. or does it?
Here is my .bsync-ignore file, which seems to half work:
nginx/fastcgi.conf
nginx/fastcgi_params
nginx/koi-utf
nginx/koi-win
nginx/mime.types
# nginx/nginx.conf <----- this is the file we want!
# nginx/proxy_params <----- this is the file we want!
nginx/scgi_params
nginx/uwsgi_params
nginx/win-utf
proc
.meteor/*
snort
.npm/*
node_modules/*
/home/cc/.*
/home/cc/crowdcourier/.*
/home/cc/crowdcourier/app/.*
crowdcourier/.*
# ROOT OF FILE SYSTEM ################
# root
bin
boot
dev
# etc
home
initrd.img
initrd.img
lib
lib64
lost+found
media
mnt
opt
proc
root
run
sbin
snap
srv
sys
tmp
usr
var
vmlinuz
vmlinuz.ol
webmin-set
# CHEESE FILES
anacrontab
appstream.conf
bluetooth
anacrontab
appstream.conf
bluetooth
brlapi.key
brltty
brltty.conf
catdocrc
chatscripts
cracklib
cryptsetup-initramfs
cups
cupshelpers
$dconf
doc-base
drbl
fstab-backup
fstab-hot
fstab-main-system
fwupd
fwupd.conf
gconf
gdb
gdm3
geoclue
ghostscript
grub-customizer
hddtemp.db
hp
ifplugd
ImageMagick-6
java-8-openjdk
kde4rc
kerneloops.conf
ksysguarddrc
kubuntu-default-settings
libao.conf
libpaper.d
libreoffice
lintianrc
menu-methods
mpd.conf
mtools.conf
netplan
netscsid.conf
nilfs_cleanerd.conf
PackageKit
papersize
pcmcia
pnm2ppa.conf
python3.6
sddm
signon-ui
speech-dispatcher
sudoers
sudoers.dist
sysstat
thunderbird
timidity
usb_modeswitch.conf
usb_modeswitch.d
whoopsie
wildmidi
wodim.conf
wpa_supplicant
xfe
# DOTFILES
.java/
.swp
# files
popularity-contest.conf
mailcap.order
group
legal
ld.so.conf
nsswitch.conf
wgetrc
shadow-
mke2fs.conf
os-release
lsb-release
gshadow-
apg.conf
updatedb.conf
magic
ca-certificates.conf.dpkg-old
insserv.conf
mime.types
securetty
vtrgb
bmon.conf
locale.gen
bindresvport.blacklist
hdparm.conf
subgid-
mailcap
subuid
shadow
networks
iptables.up.rules
ucf.conf
subgid
rpc
qemu-ifup
modules
cczerc
ld.so.cache
login.defs
crypttab
passwd-
drirc
mtab
.bsync-snap-*
subuid-
passwd
gshadow
rmt
inputrc
pam.conf
.pwd.lock
protocols
localtime
resolv.conf
ethertypes
overlayroot.conf
at.deny
locale.alias
magic.mime
machine-id
vdpau_wrapper.cfg
services
ltrace.conf
group-
manpath.config
libaudit.conf
ip6tables.up.rules
qemu-ifdown
rc.local
list.txt
request-key.conf
timezone
gai.conf
popularity-contest.conf
adduser.conf
mailcap.order
kernel-img.conf
group
screenrc
crontab
profile
legal
ld.so.conf
debian_version
nsswitch.conf
wgetrc
issue.net
sos.conf
shadow-
mke2fs.conf
os-release
hostname
lsb-release
gshadow-
apg.conf
updatedb.conf
debconf.conf
magic
ca-certificates.conf.dpkg-old
insserv.conf
deluser.conf
mime.types
environment
securetty
fstab
vtrgb
issue
bmon.conf
locale.gen
bindresvport.blacklist
hdparm.conf
subgid-
mailcap
bash_completion
subuid
shadow
networks
iptables.up.rules
ucf.conf
subgid
rpc
qemu-ifup
modules
sysctl.conf
cczerc
ld.so.cache
login.defs
logrotate.conf
crypttab
passwd-
drirc
mtab
subuid-
passwd
gshadow
rmt
inputrc
bash.bashrc
pam.conf
rsyslog.conf
.pwd.lock
protocols
sensors3.conf
ca-certificates.conf
localtime
zsh_command_not_found
.bsync-snap-20180410221645.364918
resolv.conf
ethertypes
overlayroot.conf
at.deny
locale.alias
magic.mime
machine-id
vdpau_wrapper.cfg
files.txt
services
ltrace.conf
group-
manpath.config
libaudit.conf
nanorc
ip6tables.up.rules
fuse.conf
host.conf
qemu-ifdown
rc.local
request-key.conf
timezone
gai.conf
# DIRECTORIES
at-spi2
network
terminfo
init.d
vim
iscsi
rc5.d
ppp
dnsmasq.d
firefox
request-key.d
xml
polkit-1
rc1.d
mysql
sane.d
init
systemd
apport
xdg
fonts
gtk-3.0
sysctl.d
lvm
pm
insserv.conf.d
kernel
ufw
dictionaries-common
modules-load.d
samba
cron.monthly
rcS.d
esound
skel
UPower
sensors.d
gss
ldap
console-setup
calendar
gtk-2.0
byobu
logrotate.d
python3.5
binfmt.d
libnl-3
apparmor.d
dpkg
ca-certificates
vmware-tools
sasl2
apache2
cron.weekly
cron.hourly
depmod.d
udisks2
pulse
texmf
groff
zsh
tmpfiles.d
python
python2.7
rc3.d
rc2.d
insserv
rsyslog.d
ssl
update-motd.d
selinux
rc0.d
acpi
cron.d
lighttpd
resolvconf
libvirt
apparmor
pam.d
initramfs-tools
pki
X11
apt
python3
default
docker
sgml
NetworkManager
dhcp
grub.d
emacs
openal
ssh
webmin
apm
bash_completion.d
modprobe.d
mdadm
dkms
sudoers.d
update-manager
kbd
logcheck
cron.daily
lynx
security
rc4.d
opt
mplayer
udev
avahi
rc6.d
profile.d
newt
dbus-1
iproute2
ld.so.conf.d
gnome
update-notifier
alternatives
dnsmasq.d-available
perl
I am getting the following error when try to run it:
[root@linux-utils test]# ./bsync
File "./bsync", line 97
print(s, file=sys.stderr)
^
SyntaxError: invalid syntax
[root@linux-utils test]# python -V
Python 2.6.6
[root@linux-utils test]# uname -a
Linux linux-utils 2.6.32-504.16.2.el6.x86_64 #1 SMP Tue Mar 10 17:01:00 EDT 2015 x86_64 x86_64 x86_64 GNU/Linux
If we interrupt synchronization by using Ctrl+C, temporary files that are used by rsync are left in the filesystem. Once we restart sync process, this tmp file is propagated between folders.
Maybe bsync should use a temporary folder to copy the file, and then use mv. That will be atomic.
Hi it will be nice to support multiple nodes sync
left center right when solving conflicts.
Feedback on the use of bsync between a mac with HFS+ filesystem and a NAS with ext4 partition.
When using special caracters as ñ, ó or í, but not á (forinstance), the synchrnization becomes very unstable through ssh. Move change, rmdir and so on, give errors.
On the contrary, between my laptop with Debian and ext4 partition there is no problem.
Thus, I guess there's a problem with caracter encoding between both type of partition (HFS and ext4) and bsync does not manage it.
Hi,
I am a new user of bsync and I find it extremely useful.
I don't have any issue when synchronizing two machines running under ubuntu 16.04.
Now I am trying to synchronize with my external USB hard drive, ntfs formatted.
I have the following issue :
Error: difference in snaps for path:
followed by the name of a file that is a symbolic link. And no way to get rid of it.
I've made some tests : this message does not appear when synchronizing to a USB key formatted ext4, but appear with the ones formatted ntfs.
I really would like to keep my external hard drive ntfs because I need to make regularly transfer with machines running Windows OS.
How do you understand the error message I get ? Is there a way to pass by ?
Thanks. (below, the complete session)
cleobuline 21/11/2016 17:08:44 /home/guy : bsync -iv 01-Math/ /media/guy/Hypathie/01-Math/
Loading filelists...
Loading original filelist from snap files...
Old filelist not found. Starting with empty history.
Loading dir1 filelist...
Loading dir2 filelist...
Analysing original paths...
Analysing remaining new paths in dir1...
Analysing remaining new paths in dir2...(LEFT DIR CONTENT) (RIGHT DIR CONTENT) (ACTION)
programme_agregation.txt~ --> (copy)
liens_et_docs_20161017.txt~ --> (copy)Todo in 01-Math/:
Todo in /media/guy/Hypathie/01-Math/: cp:2
Apply actions? [y/N] yApplying actions...
Applying actions in dir2...
rsync: liens_et_docs_20161017.txt~
rsync: programme_agregation.txt~
Applying actions in dir1...
Updating filelists...
Updating snap files: .bsync-snap-20161121170918.712810...
Done!
First is OK... Now I synchronize a second time :
cleobuline 21/11/2016 17:09:20 /home/guy :
cleobuline 21/11/2016 17:09:20 /home/guy : bsync -iv 01-Math/ /media/guy/Hypathie/01-Math/
Loading filelists...
Loading original filelist from snap files...
Loading .bsync-snap-20161121170918.712810...
Error: difference in snaps for path: Geometrie/Combes_Algebre_et_geometrie_ed1_v1.pdf
cleobuline 21/11/2016 17:09:23 /home/guy :
cleobuline 21/11/2016 17:09:24 /home/guy :
This file Geometrie/Combes_Algebre_et_geometrie_ed1_v1.pdf is a symbolic link
When transferring large files it might be nice to set rsync's transfer rate --bwlimit=RATE
.
Hi,
I saw on row 765 of bsync that there is a -c parameter who means "check".
It is not display on help menu and i don't know what it means, can you explain us?
Thanks
I have a directory with a lot of symlinks (both files and dirs), and, when I sync it, the expected behavior would be for the actual file or directory to be synced, not the symlink itself. rsync
has the -L
flag which does exactly this, so I hope this wouldn't be too hard to implement:
In the same way that bsync
has the -o
flag to pass arguments to ssh
, I propose for there to be a similar flag that allows us to pass custom arguments to rsync
. However, this would require some further changes: for instance, to the code that generates the .bsync-snap
to account for when we want to follow symlinks (pass the -L
flag to find
).
However, I also had to edit the flags that are currently passed to rsync
by default, namely -a
which conflicts with -L
. According to man rsync
, -a
is the same as -rlptgoD
, so I just changed -a
to -rLptgoD
to test, and the rsync part worked.
Hi dooblem,
I'm Sorry, this is not a issue but I don´t find other method to contact to you.
I have one question, is it possible synchronised directories without interactive prompt?.
I want to use the cron daemon to automate the synchronization of a directory, but this will not be possible if I have an interactive prompt.
For example:
[root@aest-segu42 apeiti]# /root/bsync -v /home/apeiti/ [email protected]:/home/apeiti
Password:
Loading filelists...
Loading original filelist from snap files...
Loading .bsync-snap-20140616160929.950125...
Loading dir1 filelist...
Loading dir2 filelist...
Analysing original paths...
Analysing remaining new paths in dir1...
Analysing remaining new paths in dir2...
(LEFT DIR CONTENT) (RIGHT DIR CONTENT) (ACTION)
<-- A000552.dmp (copy)
passwd_acceso1 <-- passwd_acceso1 (sync)
Todo in /home/apeiti/: cp:1 sync:1
Todo in [email protected]:/home/apeiti:
Apply actions? [y/N] y
Applying actions...
Applying actions in dir2...
Applying actions in dir1...
rsync: A000552.dmp
rsync: passwd_acceso1
Updating filelists...
Updating snap files: .bsync-snap-20140616161840.452684...
Done!
Cleaning SSH master...
[root@aest-segu42 apeiti]#
Thanks a lot
Hi
I propose following changes, and attach a diff file bsync.txt with a suggested implementation:
I know, it can be very dangerous to allow automatic resolution of conflicts, but that's what I need and I think is very helpful to give possibility to a completely background run.
A feature to backup conflicted files and directory would be wonderful. In rsync there is --backup option, but I think is not useful in the cases when rsync is called inside bsync.
I never coded in python, so be indulgent with provided code.
PS
bsync is one of few programs I could find that allow remote and bidirectional synchronization based on rsync with move and rename detection of files and directory.
Thank you for sharing it
Using last Bsync on two similar Ubuntu 16.04.5 LTS, underlying filesystem is Btrfs.
$ bsync /dir -i -v -b user@hostname:/dir
Loading filelists...
Loading original filelist from snap files...
Loading .bsync-snap-20210708181831.750984...
Loading dir1 filelist...
Loading dir2 filelist...
Analysing original paths...
Analysing remaining new paths in dir1...
Analysing remaining new paths in dir2...
(LEFT DIR CONTENT) (RIGHT DIR CONTENT) (ACTION)
mailbox/username/Maildir/cur/1625768344.H43283P9597.xxxxxx:2,Tbcj --> (copy)
mailbox/username/Maildir/cur/1625768373.M569836P4621.xxxxxx:2,aci --> (copy)
mailbox/username/Maildir/cur/1625768373.M569837P4621.xxxxxx:2,Tbcj --> (copy)
mailbox/username/Maildir/cur/1625768373.M569838P4621.xxxxxx:2,Tbcj --> (copy)
mailbox/username/Maildir/cur/1625768373.M569839P4621.xxxxxx:2,Tbcj --> (copy)
mailbox/username/Maildir/cur/1625768373.M569840P4621.xxxxxx:2,Tbcj --> (copy)
mailbox/username/Maildir/cur/1625768373.M569841P4621.xxxxxx:2,Tbcj --> (copy)
mailbox/username/Maildir/cur/1625768375.H897729P9502.xxxxxx:2,Tbcj --> (copy)
mailbox/username/Maildir/cur/1625765442.H606313P9101.xxxxxx:2, <-- (rm)
<-- mailbox/username/Maildir/new/1625765172.H923861P9051.xxxxxx (copy)
<-- mailbox/username/Maildir/new/1625765428.H151405P9093.xxxxxx (copy)
<-- mailbox/username/Maildir/new/1625765442.H606313P9101.xxxxxx (copy)
<-- mailbox/username/Maildir/new/1625765772.H980977P9169.xxxxxx (copy)
<-- mailbox/username/Maildir/new/1625765965.H849086P9202.xxxxxx (copy)
<-- mailbox/username/Maildir/new/1625766466.H954726P9290.xxxxxx (copy)
<-- mailbox/username/Maildir/new/1625767067.H976495P9394.xxxxxx (copy)
<-- mailbox/username/Maildir/new/1625767849.H415399P9524.xxxxxx (copy)
<-- mailbox/username/Maildir/new/1625768344.H43283P9597.xxxxxx (copy)
Todo in /dir: rm:1 cp:9
Todo in user@hostname:/dir: cp:8
Applying actions...
Applying actions in dir2...
rsync: mailbox/username/Maildir/cur/1625768344.H43283P9597.xxxxxx:2,Tbcj
rsync: mailbox/username/Maildir/cur/1625768373.M569836P4621.xxxxxx:2,aci
rsync: mailbox/username/Maildir/cur/1625768373.M569837P4621.xxxxxx:2,Tbcj
rsync: mailbox/username/Maildir/cur/1625768373.M569838P4621.xxxxxx:2,Tbcj
rsync: mailbox/username/Maildir/cur/1625768373.M569839P4621.xxxxxx:2,Tbcj
rsync: mailbox/username/Maildir/cur/1625768373.M569840P4621.xxxxxx:2,Tbcj
rsync: mailbox/username/Maildir/cur/1625768373.M569841P4621.xxxxxx:2,Tbcj
rsync: mailbox/username/Maildir/cur/1625768375.H897729P9502.xxxxxx:2,Tbcj
Applying actions in dir1...
rsync: mailbox/username/Maildir/new/1625765172.H923861P9051.xxxxxx
rsync: mailbox/username/Maildir/new/1625765428.H151405P9093.xxxxxx
rsync: mailbox/username/Maildir/new/1625765442.H606313P9101.xxxxxx
rsync: mailbox/username/Maildir/new/1625765772.H980977P9169.xxxxxx
rsync: mailbox/username/Maildir/new/1625765965.H849086P9202.xxxxxx
rsync: mailbox/username/Maildir/new/1625766466.H954726P9290.xxxxxx
rsync: mailbox/username/Maildir/new/1625767067.H976495P9394.xxxxxx
rsync: mailbox/username/Maildir/new/1625767849.H415399P9524.xxxxxx
rsync: mailbox/username/Maildir/new/1625768344.H43283P9597.xxxxxx
Updating filelists...
Updating snap files: .bsync-snap-20210708182009.368791...
Done!
Cleaning SSH master...
Few seconds after:
$ bsync /dir -i -v -b user@hostname:/dir
Loading filelists...
Loading original filelist from snap files...
Loading .bsync-snap-20210708182009.368791...
Loading dir1 filelist...
Loading dir2 filelist...
Analysing original paths...
Analysing remaining new paths in dir1...
Analysing remaining new paths in dir2...
(LEFT DIR CONTENT) (RIGHT DIR CONTENT) (ACTION)
mailbox/username/Maildir/cur/1625768409.M755002P4621.xxxxxx:2,aci --> (copy)
mailbox/username/Maildir/cur/1625768409.M755003P4621.xxxxxx:2,Tbcj --> (copy)
mailbox/username/Maildir/cur/1625768409.M755004P4621.xxxxxx:2,Tbcj --> (copy)
mailbox/username/Maildir/cur/1625768409.M755005P4621.xxxxxx:2,Tbcj --> (copy)
mailbox/username/Maildir/cur/1625768409.M755006P4621.xxxxxx:2,Tbcj --> (copy)
mailbox/username/Maildir/cur/1625768409.M755007P4621.xxxxxx:2,Tbcj --> (copy)
mailbox/username/Maildir/cur/1625768409.M755008P4621.xxxxxx:2,Tbcj --> (copy)
mailbox/username/Maildir/cur/1625765442.H606313P9101.xxxxxx:2, <-- (rm)
<-- mailbox/username/Maildir/new/1625765172.H923861P9051.xxxxxx (copy)
<-- mailbox/username/Maildir/new/1625765428.H151405P9093.xxxxxx (copy)
<-- mailbox/username/Maildir/new/1625765442.H606313P9101.xxxxxx (copy)
<-- mailbox/username/Maildir/new/1625765772.H980977P9169.xxxxxx (copy)
<-- mailbox/username/Maildir/new/1625765965.H849086P9202.xxxxxx (copy)
<-- mailbox/username/Maildir/new/1625766466.H954726P9290.xxxxxx (copy)
<-- mailbox/username/Maildir/new/1625767067.H976495P9394.xxxxxx (copy)
<-- mailbox/username/Maildir/new/1625767849.H415399P9524.xxxxxx (copy)
<-- mailbox/username/Maildir/new/1625768344.H43283P9597.xxxxxx (copy)
Todo in /dir: rm:1 cp:9
Todo in user@hostname:/dir: cp:7
Applying actions...
Applying actions in dir2...
rsync: mailbox/username/Maildir/cur/1625768409.M755002P4621.xxxxxx:2,aci
rsync: mailbox/username/Maildir/cur/1625768409.M755003P4621.xxxxxx:2,Tbcj
rsync: mailbox/username/Maildir/cur/1625768409.M755004P4621.xxxxxx:2,Tbcj
rsync: mailbox/username/Maildir/cur/1625768409.M755005P4621.xxxxxx:2,Tbcj
rsync: mailbox/username/Maildir/cur/1625768409.M755006P4621.xxxxxx:2,Tbcj
rsync: mailbox/username/Maildir/cur/1625768409.M755007P4621.xxxxxx:2,Tbcj
rsync: mailbox/username/Maildir/cur/1625768409.M755008P4621.xxxxxx:2,Tbcj
Applying actions in dir1...
rsync: mailbox/username/Maildir/new/1625765172.H923861P9051.xxxxxx
rsync: mailbox/username/Maildir/new/1625765428.H151405P9093.xxxxxx
rsync: mailbox/username/Maildir/new/1625765442.H606313P9101.xxxxxx
rsync: mailbox/username/Maildir/new/1625765772.H980977P9169.xxxxxx
rsync: mailbox/username/Maildir/new/1625765965.H849086P9202.xxxxxx
rsync: mailbox/username/Maildir/new/1625766466.H954726P9290.xxxxxx
rsync: mailbox/username/Maildir/new/1625767067.H976495P9394.xxxxxx
rsync: mailbox/username/Maildir/new/1625767849.H415399P9524.xxxxxx
rsync: mailbox/username/Maildir/new/1625768344.H43283P9597.xxxxxx
Updating filelists...
Updating snap files: .bsync-snap-20210708182100.993313...
Done!
Cleaning SSH master...
1625765442.H606313P9101.xxxxxx
was remove or move from dir1 between each sync.
Any idea of what's going on?
Hi
I've moved some videos to a subfolder, and bsync wants to duplicate these videos.
Is it normal that bsync proposes to copy files with same name and size in both directions (in different folders)?
In these case, an optional additionnal test would be to checksum these files and if they are identical, do a file move.
Hello,
I guess that line 634 there is a missing test: moves!=[]
If not move operation are impossible via ssh.
E.
Hi Maurice,
maybe you could add this small adjustment for users in README:
when connecting to remote host instead:
./bsync ALICE_DIR bob@sshserver:BOB_DIR
add port number
./bsync "-p 4000" ALICE_DIR bob@sshserver:BOB_DIR
I read the code but it was not clear for me where to put the port number.
Excellent script and very usefull.
Bob
I get the following error when I do
./bsync /sdcard/work/zot/ home:/home/abhishek/work/zot
unix_listener: "/data/data/com.termux/files/usr/tmp/tmpu4bc4w1x/bsync_abhishek@mybighostname:myport.0texbz3aTGia5AqP" too long for Unix domain socket
Error: could not open SSH connection.
The code does not work "out-of-the-box" on TCSH (both local and server):
python bsync user@host:/path/2/stuff
Gives the following error/response:
cd: Too many arguments.
Error: could not open directory: user@host:/path/2/stuff (is it created?)
I've used bsync and it is awesome. How can we use it to keep a watch on the local changes and synchronize the host and remote on real-time?
i mean the title is self explaning.
(i m not able to realize this myself, but i could support the work by investing a certain amount.)
Using latest master as of 2021/01/01 on Ubuntu 20.04 (both sides).
rsync version 3.1.3 protocol version 31
If remote directory has a space, transfer fails as shown below. If I rename the remote dir to remove the space, transfer succeeds. I've tried with quotes and with an escaped (backslash) space on the command-line.
$ bsync -v Calibre/ 'remotehost:Calibre Library/'
user@remotehost's password:
Loading filelists...
Loading original filelist from snap files...
Old filelist not found. Starting with empty history.
Loading dir1 filelist...
Loading dir2 filelist...
.... skipped some conflict-resolution questions and list of pending transfers ...
Todo in Calibre/: sync:5
Todo in remotehost:Calibre Library/: cp:260 sync:2
Apply actions? [y/N] y
Applying actions...
Applying actions in dir2...
Applying actions in dir1...
rsync version 3.1.3 protocol version 31
Copyright (C) 1996-2018 by Andrew Tridgell, Wayne Davison, and others.
Web site: http://rsync.samba.org/
Capabilities:
64-bit files, 64-bit inums, 64-bit timestamps, 64-bit long ints,
socketpairs, hardlinks, symlinks, IPv6, batchfiles, inplace,
append, ACLs, xattrs, iconv, symtimes, prealloc
rsync comes with ABSOLUTELY NO WARRANTY. This is free software, and you
are welcome to redistribute it under certain conditions. See the GNU
General Public Licence for details.
rsync is a file transfer program capable of efficient remote update
via a fast differencing algorithm.
Usage: rsync [OPTION]... SRC [SRC]... DEST
or rsync [OPTION]... SRC [SRC]... [USER@]HOST:DEST
or rsync [OPTION]... SRC [SRC]... [USER@]HOST::DEST
or rsync [OPTION]... SRC [SRC]... rsync://[USER@]HOST[:PORT]/DEST
or rsync [OPTION]... [USER@]HOST:SRC [DEST]
or rsync [OPTION]... [USER@]HOST::SRC [DEST]
or rsync [OPTION]... rsync://[USER@]HOST[:PORT]/SRC [DEST]
The ':' usages connect via remote shell, while '::' & 'rsync://' usages connect
to an rsync daemon, and require SRC or DEST to start with a module name.
Options
-v, --verbose increase verbosity
--info=FLAGS fine-grained informational verbosity
--debug=FLAGS fine-grained debug verbosity
--msgs2stderr special output handling for debugging
-q, --quiet suppress non-error messages
--no-motd suppress daemon-mode MOTD (see manpage caveat)
-c, --checksum skip based on checksum, not mod-time & size
-a, --archive archive mode; equals -rlptgoD (no -H,-A,-X)
--no-OPTION turn off an implied OPTION (e.g. --no-D)
-r, --recursive recurse into directories
-R, --relative use relative path names
--no-implied-dirs don't send implied dirs with --relative
-b, --backup make backups (see --suffix & --backup-dir)
--backup-dir=DIR make backups into hierarchy based in DIR
--suffix=SUFFIX set backup suffix (default ~ w/o --backup-dir)
-u, --update skip files that are newer on the receiver
--inplace update destination files in-place (SEE MAN PAGE)
--append append data onto shorter files
--append-verify like --append, but with old data in file checksum
-d, --dirs transfer directories without recursing
-l, --links copy symlinks as symlinks
-L, --copy-links transform symlink into referent file/dir
--copy-unsafe-links only "unsafe" symlinks are transformed
--safe-links ignore symlinks that point outside the source tree
--munge-links munge symlinks to make them safer (but unusable)
-k, --copy-dirlinks transform symlink to a dir into referent dir
-K, --keep-dirlinks treat symlinked dir on receiver as dir
-H, --hard-links preserve hard links
-p, --perms preserve permissions
-E, --executability preserve the file's executability
--chmod=CHMOD affect file and/or directory permissions
-A, --acls preserve ACLs (implies --perms)
-X, --xattrs preserve extended attributes
-o, --owner preserve owner (super-user only)
-g, --group preserve group
--devices preserve device files (super-user only)
--copy-devices copy device contents as regular file
--specials preserve special files
-D same as --devices --specials
-t, --times preserve modification times
-O, --omit-dir-times omit directories from --times
-J, --omit-link-times omit symlinks from --times
--super receiver attempts super-user activities
--fake-super store/recover privileged attrs using xattrs
-S, --sparse turn sequences of nulls into sparse blocks
--preallocate allocate dest files before writing them
-n, --dry-run perform a trial run with no changes made
-W, --whole-file copy files whole (without delta-xfer algorithm)
--checksum-choice=STR choose the checksum algorithms
-x, --one-file-system don't cross filesystem boundaries
-B, --block-size=SIZE force a fixed checksum block-size
-e, --rsh=COMMAND specify the remote shell to use
--rsync-path=PROGRAM specify the rsync to run on the remote machine
--existing skip creating new files on receiver
--ignore-existing skip updating files that already exist on receiver
--remove-source-files sender removes synchronized files (non-dirs)
--del an alias for --delete-during
--delete delete extraneous files from destination dirs
--delete-before receiver deletes before transfer, not during
--delete-during receiver deletes during the transfer
--delete-delay find deletions during, delete after
--delete-after receiver deletes after transfer, not during
--delete-excluded also delete excluded files from destination dirs
--ignore-missing-args ignore missing source args without error
--delete-missing-args delete missing source args from destination
--ignore-errors delete even if there are I/O errors
--force force deletion of directories even if not empty
--max-delete=NUM don't delete more than NUM files
--max-size=SIZE don't transfer any file larger than SIZE
--min-size=SIZE don't transfer any file smaller than SIZE
--partial keep partially transferred files
--partial-dir=DIR put a partially transferred file into DIR
--delay-updates put all updated files into place at transfer's end
-m, --prune-empty-dirs prune empty directory chains from the file-list
--numeric-ids don't map uid/gid values by user/group name
--usermap=STRING custom username mapping
--groupmap=STRING custom groupname mapping
--chown=USER:GROUP simple username/groupname mapping
--timeout=SECONDS set I/O timeout in seconds
--contimeout=SECONDS set daemon connection timeout in seconds
-I, --ignore-times don't skip files that match in size and mod-time
-M, --remote-option=OPTION send OPTION to the remote side only
--size-only skip files that match in size
-@, --modify-window=NUM set the accuracy for mod-time comparisons
-T, --temp-dir=DIR create temporary files in directory DIR
-y, --fuzzy find similar file for basis if no dest file
--compare-dest=DIR also compare destination files relative to DIR
--copy-dest=DIR ... and include copies of unchanged files
--link-dest=DIR hardlink to files in DIR when unchanged
-z, --compress compress file data during the transfer
--compress-level=NUM explicitly set compression level
--skip-compress=LIST skip compressing files with a suffix in LIST
-C, --cvs-exclude auto-ignore files the same way CVS does
-f, --filter=RULE add a file-filtering RULE
-F same as --filter='dir-merge /.rsync-filter'
repeated: --filter='- .rsync-filter'
--exclude=PATTERN exclude files matching PATTERN
--exclude-from=FILE read exclude patterns from FILE
--include=PATTERN don't exclude files matching PATTERN
--include-from=FILE read include patterns from FILE
--files-from=FILE read list of source-file names from FILE
-0, --from0 all *-from/filter files are delimited by 0s
-s, --protect-args no space-splitting; only wildcard special-chars
--address=ADDRESS bind address for outgoing socket to daemon
--port=PORT specify double-colon alternate port number
--sockopts=OPTIONS specify custom TCP options
--blocking-io use blocking I/O for the remote shell
--stats give some file-transfer stats
-8, --8-bit-output leave high-bit chars unescaped in output
-h, --human-readable output numbers in a human-readable format
--progress show progress during transfer
-P same as --partial --progress
-i, --itemize-changes output a change-summary for all updates
--out-format=FORMAT output updates using the specified FORMAT
--log-file=FILE log what we're doing to the specified FILE
--log-file-format=FMT log updates using the specified FMT
--password-file=FILE read daemon-access password from FILE
--list-only list the files instead of copying them
--bwlimit=RATE limit socket I/O bandwidth
--stop-at=y-m-dTh:m Stop rsync at year-month-dayThour:minute
--time-limit=MINS Stop rsync after MINS minutes have elapsed
--outbuf=N|L|B set output buffering to None, Line, or Block
--write-batch=FILE write a batched update to FILE
--only-write-batch=FILE like --write-batch but w/o updating destination
--read-batch=FILE read a batched update from FILE
--protocol=NUM force an older protocol version to be used
--iconv=CONVERT_SPEC request charset conversion of filenames
--checksum-seed=NUM set block/file checksum seed (advanced)
--noatime do not alter atime when opening source files
-4, --ipv4 prefer IPv4
-6, --ipv6 prefer IPv6
--version print version number
(-h) --help show this help (-h is --help only if used alone)
Use "rsync --daemon --help" to see the daemon-mode command-line options.
Please see the rsync(1) and rsyncd.conf(5) man pages for full documentation.
See http://rsync.samba.org/ for updates, bug reports, and answers
rsync error: syntax or usage error (code 1) at options.c(2369) [server=3.1.3]
rsync: connection unexpectedly closed (0 bytes received so far) [Receiver]
rsync error: error in rsync protocol data stream (code 12) at io.c(235) [Receiver=3.1.3]
Error in rsync process.
Cleaning SSH master...
Hi
When changing only permission and ownership in one side, bsync doesn't detect the modification and doesn't propagate it.
I'm doing it with a separate bash script: it synchronizes only files differents by permissions or ownership, and uses rsync and some trick to force synchronization in the right direction (I can't let rsync decide the direction of synchronization based on modification time, because obviously we need to check the change time).
Here is the bash script I'm using: probably not possible to use any of it to integrate in bsync, but I show just as an example. Also, this is a simple, not updated version, that only synchronize in one direction from remote to local.
#!/bin/bash
scriptname="$(basename "$(test -L "$0" && readlink "$0" || echo "$0")")"
local_path="$1"
remote_ip="$2"
remote_path="$3"
RSYNC_ARGS="$4"
[ -z "$RSYNC_ARGS" ] && RSYNC_ARGS="-a -A -X -z -E"
local_path="${local_path%%/}"
remote_path="${remote_path%%/}"
tmp_file="/tmp/${scriptname}_$RANDOM"
/usr/bin/rsync --rsync-path="rsync" $RSYNC_ARGS -ui -e "/usr/bin/ssh -C -i /root/.ssh/id_rsa -p 22" \
"$local_path/" root@$remote_ip:"$remote_path/" --dry-run | \
grep -Ev "^[^ ]*(c|s|t)[^ ]* " | \
grep -E "^[^ ]*(p|o|g|a)[^ ]* " | \
sed -e 's/^[^ ]* //' >"$tmp_file"
cd "$local_path"
cat "$tmp_file" | xargs -I {} stat -c '%n|%Z|%Y' "{}" | sort >"${tmp_file}.1"
cat "$tmp_file" | /usr/bin/ssh -C -i /root/.ssh/id_rsa -p 22 root@$remote_ip "cd \"$remote_path\" ; xargs -I {} stat -c '%n|%Z|%Y' \"{}\"" | sort >"${tmp_file}.2"
join -j 1 -t \| -o 1.1,1.2,2.2 "${tmp_file}.1" "${tmp_file}.2" | awk -F'|' '{if ($2 < $3) print $1}' | \
/usr/bin/rsync --rsync-path="rsync" $RSYNC_ARGS -ui -e "/usr/bin/ssh -C -i /root/.ssh/id_rsa -p 22" \
root@$remote_ip:"$remote_path/" "$local_path/" --files-from=-
rm -f "$tmp_file"*
Hi,
I needed to login my server with an ID file. So I modified the code.
Here is my patch: http://pastebin.com/UNZVpmZi
I'll dig when I have time:
Loading filelists...
Old filelist not found. Starting with empty history.
find: `ceremcem/httpd/ceremcem.net/ağkökü/phpmathpublisher/doc': Permission denied
find: `ceremcem/httpd/ceremcem.net/ağkökü/phpmathpublisher/doc_fr': Permission denied
Find Error in ceremcem/
Permission may be denied, but sync must keep going on as much as possible.
The following sequence crashes bsync:
#! /bin/sh
rm -rf testdir1 testdir2
mkdir testdir1 testdir2
echo xxxxx > testdir1/data
bsync testdir1 testdir2
sleep 2
rm -rf testdir1
touch testdir2/data
bsync testdir1 testdir2
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.