p-bizouard / cloudbackup

Orchestrateur de sauvegardes diverses : Instances Openstack, Bases de données, fichiers distants ou locaux. Sauvegarde via Restic et notifications en cas d'erreurs.

CloudBackup

Description

CloudBackup is a backup software powered by Restic. It can backup:

• Openstack instances
• MySQL/PostgreSQL databases (direct connection or with ssh gateway)
• Custom remote command by ssh (download a single file. Should use tar to backup a directory)
• Remote directory mounted with sshfs
• SFTP
• Everything with Rclone (source and destination must both use rclone)
• Check and alert on en external restic repository

The dashboard show all backup statuses, and provide the restic commands to download your data.

Global workflow

Openstack instance

Todo

• Optional restic backup arguments
• Start manual backup

Requirements

• Git
• Docker / docker-compose

Local installation - execute once

• Install dependencies, configure environments, run containers:

cp -f .env .env.local

• Launch php + database + assets

./start-dev.sh

If errors pop up because you already have containers using required ports, you can stop all running containers with docker stop $(docker ps -aq)

• Create database and load fixtures:

docker-compose exec php bin/console doctrine:database:create
docker-compose exec php bin/console doctrine:migrations:migrate
docker-compose exec php bin/console hautelook:fixtures:load --env=dev

Development environment

Launch the development environment

./start-dev.sh

Access to your containers

• Mailhog : http://localhost:8025/
• phpPgAdmin : http://localhost:9080/
• Webapp : http://localhost/

Start backups

docker-compose exec php bin/console app:backup:start

Useful commands - Symfony with docker cheatsheet

See the Symfony with docker cheatsheet

To update graph in README.md

docker-compose run php bash -c "bin/console workflow:dump backup | dot -Tsvg -o doc/graph.svg"

cloudbackup's Issues

Rclone - Allow to continue workflow by ignoring lines from stderr with a regex

• Add a field in BackupConfiguration to set a regex to ignore errors dumped on stderr
• If the field is valued and the error output is empty after ignoring lines, the command should be considered as success

It will allow to backup object storages with files name > 255 chars on a filesystem that does not support this, and consider the backup as a success even if few files were not backuped

Command "rebuild-index" is deprecated, Use "repair index" instead

Command "rebuild-index" is deprecated, Use "repair index" instead

Size check - Throw an error if the minimum expected backup size is way lighter than real

Sometime we create a backup configuration with empty database, repository, ... so we set to 0 octet, but when we go live we do not remember to set a real value. And the data will growth overtime, so if a backup is not full for some reason, we may not be alerted if the minimum expected size is not updated.

• Checkbox to ignore the error on the BackupConfiguration
• Visual error on dashboard and backup
• How much lighter should throw an error ? Should it be configurable on the BackupConfiguration ?

Velero restic backup check - Get latest backup

When a backup of the same volume is done, it saves the host path. So if the pod start on 2 different nodes, it will create two different snapshots in the history, so --latest 1 may return more than 1 result.

We should sort the resulted array based on the timestamp, with jq or in php

restic snapshots --latest 1 --json | jq 'sort_by(.time) | reverse'

Rclone - Check the file integrity with check/cryptcheck

We should check the remote integrity with https://rclone.org/commands/rclone_check/ or
https://rclone.org/commands/rclone_cryptcheck/
This should be a select choice between never/daily/weekly in the BackupConfiguration.

Rclone "Forget" step throws errors on non-existent backup directory

The forget command rclone delete --rmdirs --min-age "${KEEP_DAILY}d" "${REMOTE_STORAGE_BACKUP}" --config "${RCLONE_CONFIG}" throws errors when ${REMOTE_STORAGE_BACKUP} is non-existent.

Error executing backup - rclone delete - 2023/06/01 14:00:08 ERROR : : error listing: directory not found
2023/06/01 14:00:08 ERROR : Attempt 1/3 failed with 2 errors and: directory not found
2023/06/01 14:00:08 ERROR : : error listing: directory not found
2023/06/01 14:00:08 ERROR : Attempt 2/3 failed with 2 errors and: directory not found
2023/06/01 14:00:08 ERROR : : error listing: directory not found
2023/06/01 14:00:08 ERROR : Attempt 3/3 failed with 2 errors and: directory not found
2023/06/01 14:00:08 Failed to delete with 2 errors: last error was: directory not found

Check before executing the command with rclone lsd and expect to find a backup directory in stdout ?

Display rclone backup size on dashboard and backup history

https://rclone.org/commands/rclone_size/

Flag --last has been deprecated

restic snapshots --json --last -q - Flag --last has been deprecated, use --latest 1

Rclone "Forget" step throws errors on non empty directories

2023/08/31 22:27:42 Failed to delete: remove /data/xxx/rclone/xxx/backup/2023-09-01/documents: directory not empty

We should remove files with rclone delete without --rmdirs arguments, then use rclone rmdirs --leave-root to clean empty directories left.

Velero restic check - find a specific volume

All velero restic volume backup from a same namespace are saved in one restic repository. So if we have multiple pvc, restic snapshots --latest may return more than one result (at leat one by pvc).

We want to filter the result based on the volume name, ex: restic snapshots --latest 1 --tag volume=php-uploads-pvc --json

Add custom restic argument in the BackupConfiguration ?
Add jq or php filter ?

Rclone flags - Allow to add flags to Rclone sync command in the backup configuration

restic: restic stats should filter on tags

Just like restic snapshots, restic stats should filter on tags