camptocamp / bivac

🏕 📦 Backup Interface for Volumes Attached to Containers

Bivac : Backup Interface for Volumes Attached to Containers

Website: https://camptocamp.github.io/bivac

Bivac lets you backup all your containers volumes deployed on Docker Engine, Cattle or Kubernetes using Restic.

Documentation

• Overview
• Installation
  • Docker
  • Rancher (Cattle)
  • Kubernetes
• Usage
  • Backup a volume
  • Restore a volume
  • Manage a remote Restic repository
  • Troubleshooting
• API

bivac's Issues

Push backup start time metric to prometheus gateway

It would be interesting to have the backup start time as soon as it starts available in prometheus.
To not delete old events we could POST the backup start time ASAP, then PUT the rest of the metrics at the end of the backup, adding the backup start time again (because PUT deletes all the metrics within the grouping key) so that we can have metrics on the backup time.

Add --ssh-options="-oStrictHostKeyChecking=no" to duplicity parameters

Launch duplicity cleanup after backup

Prometheus metrics should be grouped by volume

Errors with Exoscale

Using Exoscale, we are getting errors when the backups are partial:

11/07/2016 10:51:42{"level":"debug","msg":"Local and Remote metadata are synchronized, no sync needed.\r\nLast full backup date: Sat Jul  9 02:00:01 2016\r\nAttempt 1 failed. S3ResponseError: S3ResponseError: 403 Forbidden\r\n\u003c?xml version=\"1.0\" encoding=\"UTF-8\"?\u003e\u003cError\u003e\u003cCode\u003eAccessDenied\u003c/Code\u003e\u003cMessage\u003eAccess Denied\u003c/Message\u003e\u003cRequestId\u003e0b6d469e-82a3-499d-af22-c5fa1b5bf6cd\u003c/RequestId\u003e\u003cHostId\u003e0b6d469e-82a3-499d-af22-c5fa1b5bf6cd\u003c/HostId\u003e\u003c/Error\u003e\r\nManifests not equal because different volume numbers\r\nFatal Error: Remote manifest does not match local one.  Either the remote backup set or the local archive directory has been corrupted.\r\n","time":"2016-07-11T08:51:42Z"}

Two problems here:

• the error goes undetected by Conplicity
• this should not happen, and is probably linked to the fact that we don't set ACLs

Remove lockfile from duplicity_cache

Sometimes we have this error message:

Another instance is already running with this archive directory
If you are sure that this is the  only instance running you may delete
the following lockfile and run the command again :
     /root/.cache/duplicity/postgresql-puppetdb/lockfile.lock

Image pulling is broken

Since migrating to docker/engine-api, image pulling does not work anymore. It seems the pullImage method exits before the thread is done pulling the image.

The image does get pulled, and is found on a second run of conplicity.

Failed to obtain reader, Failed to marshal fields to JSON, json: unsupported type: <-chan struct {}

Launch duplicity remove-older-than after backup

Otherwise backups are never purged.
We should use DUPLICITY_REMOVE_OLDER_THAN to configure the threshold

Don't backup duplicity_cache

Refactoring BackupVolume

Improve coverage

• Make coverage pass for handler/handler_test.go (flags)
• Add tests (TestMain) for conplicity.go
• Add tests for providers/
• Test bad stdout in volume/volume_test.go

Fails on second backup

When conplicity performs two backups and the second backup requires a prepare command, it fails with:

7/5/2016 2:26:17 PM�[31mFATA�[0m[0589] Failed to start exec: Error response from daemon: http: Hijack is incompatible with use of CloseNotifier

Error: "Failed to parse Duplicity output for last full backup date of foo"

Try to detect volume content and launch pre_command accordingly

Send lastbackup and lastfull metrics to prometheus pushgateway

Parsing duplicity collection-status output

Add LICENSE file

Add unit tests

Change project name

TODO:

• Choose new project name
• Rename project in GitHub
• Rename project in Code
• Change logo
• Change Hugo site

Send "duplicity verify" exit code to prometheus pushgateway

Use docker verify and create a prometheus file containing the state of the backup

Allow to configure GPG encryption

Mounts is not populated without inspect before Docker 1.11

Allow to set DUPLICITY_FULL_IF_OLDER_THAN

Metrics not collected when something fails

It looks like a volume metrics are not collected when an error is thrown.

8/2/2016 6:38:21 AM{"level":"error","msg":"Failed to backup volume openvas_cache: failed to backup volume: failed to retrieve last backup info: failed to parse Duplicity output for last full backup date of openvas_cache","time":"2016-08-02T04:38:21Z"}

=> no metrics for openvas_cache volume (even backupExitCode)

Allow to blacklist some volumes with environment variable

Add rclone engine

Overriding using volume labels does not seems to work

I create a volume with a label io.conplicity.engine=rclone and conplicity still backups it with duplicity.
Maybe related to moby/moby#22838

Add DUPLICITY_OPTS environment variable

This will allow us to pass any option to duplicity without having to expose a new environment variable.

Add remove-all-but-n-full <count> support

Parallellize backups

It should be possible to parallellize backups.

Cannot use scp or sftp with ssh keys

First of all, thank you for this really promising tool.

I am struggling a bit here as I do not use AWS and I would like to see Conplicity (which I run in a container) send the backup over ssh.

Problem is: there is currently no way to pass authentication keys all the way down to the Duplicity container, hence a permission denied is all I can get so far.

I am really happy to help implementing this, but TBH, I do not really now how to do it or where to start.

Handle exceptions properly

Currently, if someone goes wrong with one backup, all backups are stopped.

We should handle that better, outputing an error for the current backup and continuing with the next one.

Pass backup policy parameters as volume labels

Each container should be able to expose backup policy parameters are labels, in the form io.conplicity.<volume>.param.

Examples:

• io.conplicity.pgdata.driver: postgresql
• io.conplicity.pgdata.retention: 7d

Use docker run to get backup provider instead of os.Stat

Add support for no compression

Persist duplicity cache

Send metrics to pushgateway only if slice is not empty

Document PUSHGATEWAY_URL

Support CLI flags

In addition to environment variables, support CLI flags for options.

Blacklist lost+found directory

Get real hostname instead of container's

Currently, conplicity uses the container's hostname in the upload path.

For example, on Rancher, it will typically be conplicity_1, conplicity_2, etc. However, these containers are not strictly linked to their Docker hosts.

It would be better to use the name of the Docker host instead. The info cannot be obtained from the Docker API, but it can be obtained through orchestration API, such as Rancher's.

For example, we could have a --rancher-hostname option which would get the hostname using Rancher's metadata API instead of using the container's hostname.

Use --name parameter now that archive-dir persists

Create a logo

Too much volumes created

It looks like 2 unnamed volumes are created per volume backed up. I can't figure out why yet...

Add remove-all-inc-of-but-n-full <count> support

Fix stdout redirection

See fgrehm/go-dockerpty#4

Crash when no volume is present

Make does not detect changes in all source files

Workaround for janitor

It looks like janitor removes the duplicity_cache volume because there is not duplicity container running when it does the cleanup.
It even looks like it does it the wrong way by deleting /var/lib/docker/volumes/duplicity_cache without calling docker volume rm duplicity_cache, so Docker thinks that the named volume still exists and will not recreate it at next run of docker run -v duplicity_cache:/root/.cache/duplicity camptocamp/duplicity ... and thus fails with:

stat /var/lib/docker/volumes/duplicity_cache/_data: no such file or directory

Use github.com/docker/engine-api instead of fsouza/go-dockerclient?

I don't know if this is a good idea or not...