The current version of MySQL (I work with 5.6) does not support the --defaults-extra-file using a password which is transmitted, resulting in a backup instead get "mysqldump: Got error: 1045:" Access denied for user ..... . "

Why not just use the standard option -p?
such as: -p$BM_MYSQL_ADMINPASS

Hi,

I can't find why my incremental files are not uploaded...

I find some fix but they don't work.
I try to edit /usr/bin/backup-manager-upload :
adding while (<$g_root_dir/*incremental*>) { push @{$ra_files}, $_; }

Upload method is FTP and everything works fine except incremental.bin upload...

Thx for help

System is Debian Jessie

Hi,

not sure if you know, but S3 has a 5G file size limit which is a problem if you want to copy 9G backup. Here's what I faced today:

backup-manager --verbose

/ebs/archives/localhost.combinlogs.20091022.master.tar.gz.gpg: ok (134M, 33de5a1a62c515110538824377f65562)
/ebs/archives/localhost.commysql.20091022.master.tar.gz.gpg: ok (9135M, 15b58d23f9aab50b3001bb281edfcf0d)
Trying to upload files to s3 service
Connected to s3.amazon.com
opened /ebs/archives/localhost.com-20091022.md5 of length 174 and will name the key localhost.com-20091022.md5
opened /ebs/archives/localhost.combinlogs.20091022.master.tar.gz.gpg of length 136275275 and will name the key localhost.combinlogs.20091022.master.tar.gz.gpg
Error reported by backup-manager-upload for method "s3", check "/ebs/tmp/bmu-log.uD1361".

cat /ebs/tmp/bmu-log.uD1361

Out of memory!

It is usually addressed by spliting file before uploading. Since your scripts are mainly bash based, it may make sense to just use "split" to do the splittin, maybe based on BM_TARBALL_SLICESIZE if not dar is used. In the meantime we'll probably fall back to dar if that will work around the issue.

BM_MYSQL_SAFEDUMPS add "--opt" to the command line: https://github.com/sukria/Backup-Manager/blob/0c6c3cf314c1df0d807adb617f6cd9e38f3b5378/lib/backup-methods.sh

But according to the documentation, this option is enabled by default, and can be disabled by using "--skip-opt": http://dev.mysql.com/doc/refman/5.1/en/mysqldump.html#option_mysqldump_opt

So it is like BM_MYSQL_SAFEDUMPS option does not have any impact.

Morever, "--opt" is a shortcut for "--add-drop-table --add-locks --create-options --disable-keys --extended-insert --lock-tables --quick --set-charset". So the tables are locked during the backup, which is really annoying for large databases as it can take several minutes.

If you are using innoDB tables, a better option is to disable locks "--lock-tables=false" and append "--single-transactions": http://dev.mysql.com/doc/refman/5.1/en/mysqldump.html#option_mysqldump_single-transaction as suggested here: http://stackoverflow.com/questions/104612/run-mysqldump-without-locking-tables

With the master version of Backup-Manager, this can be done by using BM_MYSQL_EXTRA_OPTIONS="--lock-tables=false --single-transactions" I guess.

So, 2 suggestions:

• if I am right, the BM_MYSQL_SAFEDUMPS is unuseful (at least from MySQL 5.0). We can keep it or change the behavior (to append --lock-tables if BM_MYSQL_SAFEDUMPS=false).
• we could add an option: MYSQL_INNO_DB = true, and in this case, replace locks by single transaction in combination with BM_MYSQL_SAFEDUMPS=true

"make install" creates paths starting with double-slash, which should not be used unless it is a path on the network.

In the Makefile, there are paths combined from $(DESTDIR)/$(PREFIX) and other similar constructs. Defaults are set as:
DESTDIR?=
PREFIX?="/usr/local"
which results in paths beginning with "//usr/local/...", which is apparently not intended and wrong.

One fix I can suggest is to require DESTDIR to end with a slash (if not empty), and change all "$(DESTDIR)/$(" to "$(DESTDIR)$(".

It looks like http://www.backup-manager.org/ has been taken over.
Please consider fixing links to it, e.g. the on in the GitHub header.
Thanks and best, Sebastian

Hi.

I just accidentally deleted some data, and had to restore from backup, made with backup-manager. Most of the data was there, but some files were missing from the backup tarball (the master), for no obvious reason. It's not filenames: some files named 'Makefile' were skipped and some not. It's not paths: some directories had all subdirectories backed up, but only some of the files. I can clearly tell when stuff is missing, since lots of the backed up directories had a git repo, and the repo knows when files are deleted. Any idea how to track this down?

Thanks

how to config backup-manager if i save mysql backups in separate files

It would be great, if Backup-Manager would support SFTP (not SCP) as an upload protocol, where a SSH-terminal access is forbidden.

In case you are unaware of it, the backup-manager.org domain now appears to be "parked" on sedo and all related URLs simply display a few backup related links not associated with your project.

Screenshot 12:19:12 5:16 PM

http://wiki.backup-manager.org/index.php/Main_Page#Documentation

Hi,
Not sure if my problem is related to the configuration. I think I kept it very simple.
When running the backup, it will upload the same file two times.

Getting lock for backup-manager 13814 with /etc/backup-manager.conf
Cleaning /var/archives
Using method "tarball".
/var/archives/bnmmnb-etc.20140506.master.tar.gz: ok (3M, 88830cea5027470d15210a1340951f45)
Using the upload method "ftp".
Trying to upload files with ftp
Logged on 10.xx.xx.xx, in public_html (FTP binary mode)
Cleaning remote directory through FTP
File /var/archives/bnmmnb-20140506.md5 transfered
File /var/archives/bnmmnb-etc.20140506.master.tar.gz transfered
All transfers done, loging out from 10.xx.xx.xx
Logged on 10.xx.xx.xx, in public_html (FTP binary mode)
Cleaning remote directory through FTP
File /var/archives/bnmmnb-20140506.md5 transfered
File /var/archives/bnmmnb-etc.20140506.master.tar.gz transfered
All transfers done, loging out from 10.xx.xx.xx
No burning method used.

configuration file

export BM_REPOSITORY_ROOT="/var/archives"
export BM_TEMP_DIR="/tmp"
export BM_REPOSITORY_SECURE="true"
export BM_REPOSITORY_USER="root"
export BM_REPOSITORY_GROUP="root"
export BM_REPOSITORY_CHMOD="770"
export BM_ARCHIVE_CHMOD="660"
export BM_ARCHIVE_TTL="1"
export BM_REPOSITORY_RECURSIVEPURGE="false"
export BM_ARCHIVE_PURGEDUPS="true"
export BM_ARCHIVE_PREFIX="bnmmnb"
export BM_ARCHIVE_STRICTPURGE="true"
export BM_ARCHIVE_NICE_LEVEL="10"
export BM_ARCHIVE_METHOD="tarball"
export BM_ENCRYPTION_METHOD="false"
export BM_ENCRYPTION_RECIPIENT=""
export BM_TARBALL_NAMEFORMAT="long"
export BM_TARBALL_FILETYPE="tar.gz"
export BM_TARBALL_OVER_SSH="false"
export BM_TARBALL_DUMPSYMLINKS="false"
export BM_TARBALL_DIRECTORIES="/etc"
export BM_TARBALL_BLACKLIST="/var/archives"
export BM_TARBALL_SLICESIZE="1000M"
export BM_TARBALL_EXTRA_OPTIONS=""
export BM_TARBALLINC_MASTERDATETYPE="weekly"
export BM_TARBALLINC_MASTERDATEVALUE="1"
export BM_MYSQL_DATABASES="ALL"
export BM_MYSQL_SAFEDUMPS="true"
export BM_MYSQL_ADMINLOGIN="root"
export BM_MYSQL_ADMINPASS=""
export BM_MYSQL_HOST="localhost"
export BM_MYSQL_PORT="3306"
export BM_MYSQL_FILETYPE="bzip2"
export BM_SVN_REPOSITORIES=""
export BM_SVN_COMPRESSWITH="bzip2"
declare -a BM_PIPE_COMMAND
declare -a BM_PIPE_NAME
declare -a BM_PIPE_FILETYPE
declare -a BM_PIPE_COMPRESS
export BM_PIPE_COMMAND
export BM_PIPE_NAME
export BM_PIPE_FILETYPE
export BM_PIPE_COMPRESS
export BM_UPLOAD_METHOD="ftp"
export BM_UPLOAD_HOSTS="10.xx.xx.xx"
export BM_UPLOAD_DESTINATION="public_html"
export BM_UPLOAD_SSH_USER=""
export BM_UPLOAD_SSH_KEY=""
export BM_UPLOAD_SSH_HOSTS=""
export BM_UPLOAD_SSH_PORT=""
export BM_UPLOAD_SSH_DESTINATION=""
export BM_UPLOAD_SSH_PURGE="true"
export BM_UPLOAD_SSH_TTL=""
export BM_UPLOAD_SSHGPG_RECIPIENT=""
export BM_UPLOAD_FTP_SECURE="false"
export BM_UPLOAD_FTP_PASSIVE="true"
export BM_UPLOAD_FTP_USER="bnmmnb"
export BM_UPLOAD_FTP_PASSWORD="somerandompassword"
export BM_UPLOAD_FTP_HOSTS="10.xx.xx.xx"
export BM_UPLOAD_FTP_PURGE="true"
export BM_UPLOAD_FTP_TTL="14"
export BM_UPLOAD_FTP_DESTINATION=""
export BM_UPLOAD_S3_DESTINATION=""
export BM_UPLOAD_S3_ACCESS_KEY=""
export BM_UPLOAD_S3_SECRET_KEY=""
export BM_UPLOAD_S3_PURGE="false"
export BM_UPLOAD_RSYNC_DIRECTORIES=""
export BM_UPLOAD_RSYNC_DESTINATION=""
export BM_UPLOAD_RSYNC_HOSTS=""
export BM_UPLOAD_RSYNC_DUMPSYMLINKS="false"
export BM_BURNING_METHOD=""
export BM_BURNING_CHKMD5="false"
export BM_BURNING_DEVICE=""
export BM_BURNING_DEVFORCED=""
export BM_BURNING_ISO_FLAGS="-R -J"
export BM_BURNING_MAXSIZE=""
export BM_LOGGER="true"
export BM_LOGGER_LEVEL="info"
export BM_LOGGER_FACILITY="user"
export BM_PRE_BACKUP_COMMAND=""
export BM_POST_BACKUP_COMMAND=""

I have a server configured with mysql backup with compression, and this starves the CPU even when I am running it at nice level 15. I think this is happening because nice needs to be invoked for every command in the pipeline, and right now it seems to be invoked only for mysqldump, not for the following compression command.

we have currently

nice -n nice_value mysqldump .... | bzip2 ...

which could be modified to

nice -n nice_value mysqldump .... | nice -n nice_value bzip2 ...

Hi. Here's a git tree to make the test suite work:

https://github.com/dkogan/Backup-Manager/tree/fix_test_suite

Github is confused, so you don't get a pull request

S3 uploads are currently sent to Amazon S3 on port 80 (unencrypted). It is necessary to set

secure => 1

at about line 874 in backup-manager-upload to enable SSL connections to S3. This is highly recommended for better data security.

According to the gzip manpage, using the --rsyncable flag improves data usage:

--rsyncable
While compressing, synchronize the output occasionally based on the input.
This increases size by less than 1 percent most cases,  but  means  that  the 
rsync(1)  program can take advantage of similarities in the uncompressed input when 
syncronizing two files compressed with this flag. gunzip cannot tell the difference
between a compressed file created with this option, and one created without it.

Currently BM allow MySQL dump. Why not to add also PG dump ?

To reduce bandwidth usage for rsync uploads, add the -z flag to enable compression.

Hi,

is it possible to do a new release with all evolution since 2010 ?

Thx

Hi, i'm working on lzma and i want just the incremental compression/decompression that you have used in your application
Can you help me in this please

http://www.backup-manager.org/ doesn't seem to talk about backup-manager a lot.

I am checking the backup-manager-upload script from devel branch, finding out the send_files_with_s3 just ignores the $repository argument which causing the BM_UPLOAD_DESTINATION doesn't work for s3 upload method, is this expected?

I also noticed that

The bucket to upload to. This bucket must be dedicated to backup-manager
export BM_UPLOAD_S3_DESTINATION=""

does "dedicated" imply the s3 upload method will take no destination / prefix?

In my opinion, I think it's very useful to specify a s3 key prefix in conf file so it allows backup-manager to share the bucket with other guys (by using different prefix).

I like the tool very much, and want to use it on all mu servers, but I'm woried that there is no development since Sep 30, 2014. Is this repository still active?
I've already got a patch almost ready and will create a pull request in a few hours

Some paths in backup-manager exec are hardcoded

# All the paths we provide
libdir="/usr/share/backup-manager"
vardir="/var/lib/backup-manager"
bmu="/usr/bin/backup-manager-upload"
bmp="/usr/bin/backup-manager-purge"

Ii's problematic when BM is installed with PREFIX=/usr/local

Config file is:
export BM_REPOSITORY_ROOT="/backups/org/projects"
export BM_TEMP_DIR="/tmp"
export BM_REPOSITORY_SECURE="true"
export BM_REPOSITORY_USER="root"
export BM_REPOSITORY_GROUP="root"
export BM_REPOSITORY_CHMOD="770"
export BM_ARCHIVE_CHMOD="660"
export BM_ARCHIVE_TTL="10"
export BM_ARCHIVE_STRICTPURGE="false"
export BM_ARCHIVE_NICE_LEVEL="10"
export BM_ARCHIVE_METHOD="tarball-incremental"

export BM_TARBALL_NAMEFORMAT="short"
export BM_TARBALL_FILETYPE="tar.gz"
export BM_TARBALL_OVER_SSH="false"
export BM_TARBALL_DUMPSYMLINKS="false"

export BM_TARBALL_DIRECTORIES="/store/org/projects"

export BM_TARBALL_BLACKLIST="*/nobackup"
export BM_TARBALL_EXTRA_OPTIONS=""

export BM_TARBALLINC_MASTERDATETYPE="weekly"
export BM_TARBALLINC_MASTERDATEVALUE="5"

export BM_LOGGER="true"
export BM_LOGGER_FACILITY="user"
export BM_PRE_BACKUP_COMMAND=""
export BM_POST_BACKUP_COMMAND=""

It creates master and increment archives successfully, but don't remove files older that 10 days.
Version is 0.7.9-3 on Debian Squeeze.

What is wrong in config file?
How can I debug all the process? Adding "--verbose" didn't help, output is only:
When validating the configuration file /etc/backup-manager.conf, 4 warnings were found.

I have a directory that is a subdirectory of a directory I am backing up using Backup Manager. This subdirectory has some very large files in it. I don't want anything from that subdirectory included as part of the backup. Here is the main directory I am backing up:

BM_TARBALL_TARGETS[0]="/home/trails"

And here are a couple different ways I tried to exclude the directory using the blacklist feature without success:

export BM_TARBALL_BLACKLIST="/dev /sys /proc /tmp /home/trails/data"
export BM_TARBALL_BLACKLIST="/dev /sys /proc /tmp /home/trails/data/*"

I'd appreciate any advice on how to set this up correct to exclude files in the 'data' directory. Thanks in advance for your help!

BM_TARBALL_BLACKLIST could use wildcards. A good example is skipping .gvfs directories which throw access denied errors for root user:

BM_TARBALL_BLACKLIST="/home/*/.gvfs"

Currently it needs to be exact:

BM_TARBALL_BLACKLIST="/home/bob/.gvfs /home/george/.gvfs /home/paul/.gvfs /home/john/.gvfs /home/ringo/.gvfs ..."

After doing "make install" there is no easy way to uninstall backup-manager.

Good practice in makefiles with "install" is to provide an "uninstall".

Hello,

Just a question, has anyone tried to get this great script to work on FreeBSD 9? I saw the remark about the group ID on FreeBSD so I am wondering if that was just a long time ago on a much older version.

Cheers

Due to backup-manager making excludes relative to the actual target-path, tar will leave out identically named directories anywhere in the target space.
Example: target is /var, exclude is /var/archives, then /var/lib/mailman/archives will be omitted also. To get the intended behaviour comment out the lines

            pattern="${pattern#$target}"
            length=$(expr length $pattern)
            pattern=$(expr substr $pattern 2 $length)

in file lib/backup_methods.sh near line 259 (function __get_flags_relative_blacklist)

With "tarball-incremental" method, it generates *.tar.gz and *.incremental.bin files. The tar.gz files are chmod'ed with BM_ARCHIVE_CHMOD if BM_REPOSITORY_SECURE is true; but not the incremental.bin ones.

By default it has -rw------- permissions, which will fail remote backup when pulling the archives to another location with a dedicated user.

Temporary fix:
Create a /home/backup/fix-perms.sh script:

#!/bin/bash
chown root:backup /home/backup/*.incremental.bin && chmod g+r /home/backup/*.incremental.bin

Then, in /etc/backup-manager.conf:

export BM_POST_BACKUP_COMMAND="/home/backup/fix-perms.sh"

If a .my.cnf file already exists, backup-manager will create an empty sql file (and a 14 byte bz2 file if compression is on). This happens because it uses the --defaults-extra-file option from mysql, and this overrides everything except the ~/.my.cnf. The failure is not reported anywhere (not even in the verbose mode).

It should not fail silently - if mysqldump failed, $PIPESTATUS could be used to determine failure.

Hi,
I'm always using BM on few servers and, some years ago, I've wrote a patch to use pbzip2 if installed, for a parallel compression process.

Unfortunately, I've lost my patch… but I can write it again and pull request it, but is this repo always managed by someone ? Or need I fork to another repo ?

I wanted to also write a new upload method using HubiC.

I have use backup-manager on Debian Squeeze and it shows me sometimeswarning at output:
When validating the configuration file /etc/backup-manager.conf, 1 warnings were found.
When validating the configuration file /etc/backup-manager.conf, 3 warnings were found.

But I can't find what is the problem string and what is wrong on it.

Adding "--verbose" param to command didn't help.
How I can debug this problems?
Can I see which key, which test is failed or string number where the warning was found?

I can't open the backup-manager.org site very long time.
Is there hosting/domain problems or project moved to other domain?

Development of new version and support old versions will continue? or the project have any problems and developing is stopped?

backup-manager is the best tool for backup data, please don't stop the development!

Hi, from Debian Repository I have 0.7.10.1-debian1 version installed.

How I can update it to latest?

I don't know how to disable "UPLOAD" option: when I write "export BM_UPLOAD_METHOD="none" script wait for "Insert CD-ROM Drive to continue". How I can resolve that problem? I don't have CD-ROM at my VDS. :)

Hi, refer to the postgresql documentation (http://www.postgresql.org/docs/8.4/static/libpq-pgpass.html), the format for .pgpass file is :

hostname:port:database:username:password

Database option is missing in backup-methods.sh. You can use * wildcard.

Hello,

Thanks for your great tool ! My servers feels better with it. 👍

Is it possible to make an hourly backup ?
I have find out some things about a BM_ARCHIVE_FREQUENCY but it's not in the doc or anywhere else...

Thanks.

When BM_ARCHIVE_PURGEDUPS is set to true, a symlink is created from the old file to the new one. However, when a configuration file didn't change for 8 days, the symlink limit is reached, and old file can't be accessed by name.

I got no obvious solution for that.

If symlinks are created from the new file to the old file, then we'll get in trouble when the real purge comes.
It is maybe possible to go backward in previous symlinks and fix them when creating a new one, but then the script could get really complex.
Finally, the use of hardlinks could change this. However, I'm nearly sure there's also a limitations on the number of hardlinks an inode can have.

Hi. Here's a git tree to clean up the links to the dead backup-manager.org website. Github is confused, so you don't get a pull request.

https://github.com/dkogan/Backup-Manager/tree/fix_dead_links

The regexp test in backups-methods.sh for pgsql failed each time for me

if ! grep -qE "(${BM_PGSQL_HOST}|[^:]*):(${BM_PGSQL_PORT}|[^:]*):[^:]*:${BM_PGSQL_ADMINLOGIN}:${BM_PGSQL_ADMINPASS}" $pgsql_conffile; then

Hello

OpenStack seems to be a very popular protocole.
It would be great to have Backup Manager support this, what do you think ?

regards

When using tarball-incremental archive method with a tar.lz filetype, backup-manager actually uses tarball leading to the creation of a master archive every day.

How to schedule auto backup in MySQL backup manager version 1.3.1.0 ? Help me out ! ! !

Hello,

Here's my configuration:

export BM_TARBALL_BLACKLIST="/var/archives/* /dev/* /home/user/no_backup/* /media/* /mnt/* /proc/* /run/* /sys/* /tmp/* /lost+found /boot/lost+found /home/lost+found /var/lost+found"

But after that I've still files that are in the backup in /home/user/no_backup/* or /proc/* but it seems to work for /run/* /sys/*. It's completely random, why?

For information, if I use the asterisk to avoid to manually recreate folders after restoring my system.

Hi there,

Sorry, it's not really a issue, but i dont realy know where i can post my question...

Is there a way to backup files to google drive ?

Thk :)

Ma2x

A bug report was filed on the Debian BTS: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=784446

This describes backup-manager purging incremental archives while keeping the master, which makes the archive set incomplete. A fix is available in https://github.com/dkogan/Backup-Manager/tree/fix_overzealous_ttl_purge

This is the fix_overzealous_ttl_purge branch

eg: The "pgsql" method is chosen, but $pgdump is not found.

When pgdump is actually missing it displays:
The "pgsql" method is chosen, but is not found.

which is not very helpful

(actually only tested with french translation)

If database not change dumps diff only by date
maybi enable this option by default?