oxplot / rsyncbtrfs Goto Github PK
View Code? Open in Web Editor NEWBackup tool combining rsync and btrfs
Backup tool combining rsync and btrfs
Hi,
from your description:
You can use --bind-mount argument when backing up to instruct rsyncbtrfs to bind mount the source directory under a temp path. This is useful when you don't want to backup all the mount points under the source:
Why not just use the rsync option:
-x, --one-file-system don't cross filesystem boundaries
Basically I think this does the same with less overhead.
So I have yet to get rsyncbtrfs to make a backup that doesn't fail. Usually, It appears that temp/cache files that change on disk or disappear before rsync finishes is the culprit. I've tried 4 backup now and each time it fails with something like:
nmartin@deb:~/Downloads$ rsyncbtrfs backup /home/nmartin /media/backup3tb/nmartin
Create subvolume '/media/backup3tb/nmartin/.inprog-PmnUqyW/vol'
file has vanished: "/home/nmartin/.local/share/gvfs-metadata/label-ARCH_201504-291a92f3.log"
rsync warning: some files vanished before they could be transferred (code 24) at main.c(1070) [sender=3.0.9]
rmdir: failed to remove `/media/backup3tb/nmartin/.inprog-PmnUqyW': Directory not empty
this specific failure was due to a usb pen drive being unplugged during the backup. I guess I could build an excludes file to pass to the backup script, but is there a better method to detect 'OK' errors from rsync vs critical errors?
Hi,
Instead of doing this
trap "
umount "'"'"$TMP_MOUNT"'"'" &>/dev/null;
rmdir "'"'"$TMP_MOUNT"'"'";
btrfs subvolume delete "'"'"$TMP_INPROG/vol"'"'" &>/dev/null;
rm -f "'"'"$TMP_INPROG/cur"'"'";
rmdir "'"'"$TMP_INPROG"'"'";
" INT QUIT TERM EXIT
it can be simplified to this:
trap '
umount "$TMP_MOUNT" &>/dev/null
rmdir "$TMP_MOUNT"
btrfs subvolume delete "$TMP_INPROG/vol" &>/dev/null
rm -f "$TMP_INPROG/cur"
rmdir "$TMP_INPROG"
' INT QUIT TERM EXIT
First, the semicolons are not needed.
Second, the explosion of quotes can be reduced to, for example, "$TMP_MOUNT". Notice that the environment variables are known when the trap is executed.
See https://btrfs.wiki.kernel.org/index.php/Incremental_Backup
This is one of the requests from #2
Hello Mansour:
I hope things are going well. I have been using your Python script for my backups for almost 2 years now and everything is going well.
I have learned more about btrfs, and many new features have been added, which got me thinking:
https://btrfs.wiki.kernel.org/index.php/UseCases#How_can_I_use_btrfs_for_backups.2Ftime-machine.3F
I think you should contact them and get your script added to the list.
One minor issue I tend to run into is that the source filesystem (one being backed up) is typically mounted RW and not RO. I solve this problem by creating a RO snapshot (the source is also btrfs) working from that. It might be nice to add this to your script; the user can specify that the source filesystem is btrfs.
I know this might be inconvenience because of the name of the script, but btrfs now supports send/recv functionality that could perform incremental backup without rsync:
https://btrfs.wiki.kernel.org/index.php/Incremental_Backup
it would be nice to make this an option.
I know all of this is alot of work, but I think this is a great script and item #3 would greatly increase the user-base. I am willing to contribute some money to have items 2-3 implemented since I am sure you have other tasks competing for your time, and time is money!
Nasser
hi,
have you never thinked about the idea of add a retention policy for backup?
for example, if you run the script without any argument, it works normally...if you add a parameter ( eg, X ) script will work and if it counts X subvolumes, it will create new subvolume and remove le oldest one.
Some users might want to backup a tree with every mount point under it.
We should probably modify the default behavior to no-bind-mount. Then the user can pass a argument like --bind-mount
.
rsync allows specifying multiple source paths:
rsync [OPTION...] SRC... [DEST]
This would be useful also for rsyncbtrfs, if we wanted to include multiple different paths into one backup subvolume. Unfortunately, with the way rsyncbtrfs wraps the rsync call, this is not possible. Well, almost... we could use the totally unintuitive call
rsyncbtrfs backup src1 dst src2 [OPTIONS...]
which would call:
rsync [OPTIONS...] src2 src1/ tmp_dst/vol
Note the slash behind src1
, which means that the hierarchy is scrambled.
Are there some rsync options I'm not aware of to allow specifying multiple source paths, or would it be necessary to change rsyncbtrfs?
One should be able to somehow manage the number of backups, either by giving a simple number or a maximum age or the minimum of free space to be left.
rsyncbtrfs should then delete the corresponding subvolumes.
Let the user choose to mount the source location as RO before backing up in order to ensure no files are modified while backing up.
This is one of the requests from #2
Hi,
The &> syntax is fine for bash, but not on systems that have /bin/sh -> /bin/dash (e.g. default Ubuntu).
I have two commits for you if you want to. See [email protected]:keestux/rsyncbtrfs.git
BTW. I hate the github pull requests because they don't like "fast forward" merging.
Hello:
This is more of a feature request than an issue, but in order to use this script on a long-term basis it would make sense to be able to specify the intervals over which a user wants to save snapshots. If I am running this script weekly, at some point I am going to want to remove old snapshots, else my disk will fill up.
I would be interested in having this functionality implemented. I have been using a relatively complicated shell script that uses the "cp -a" option for incremental backups. BTRFS is more appealing because of compression and the snapshot formalism. Unfortunately I am not very good with bash, so I cannot implement this myself.
Nasser
If rsync fails, then /cur is inconsistent. Consequently the latest complete update is not in /cur but in the second to last snapshot. This is confusing. The script would be better if the rsyncbtrfs did:
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.