cyborgbackup / cyborgbackup

CyBorgBackup backup system based on Borg

CyBorgBackup

CyBorgBackup is a Web and API Interface to manage BorgBackup solution on multiple servers.

It is implemented using Python, Django and AngularJS.

The complete documentation can be found at <https://cyborgbackup.readthedocs.io>

Discussion

• Discord - Real-time chat hosted by the CyBorgBackup community
• GitHub Discussions - Discussion form hosted by GitHub

Build Status

Type	Branch	Status
GitHub	master
Github	develop
CodeCov	master
ReadTheDocs	master

Features

• uses BorgBackup
• SSH Connection
• Scheduled jobs
• Local or Remote Borg Repository
• Catalog based on Borg Archives
• Restore Test
• Archive Size statistics
• Client and Repository preparation
• VM Backup Modules
• E-mail notification
• Auto-prune
• Logs system
• REST API

Installation

Please see documentation for instructions on installing CyBorgBackup.

Providing Feedback

The best platform for general feedback, assistance, and other discussion is our Github Discussions. To report a bug or request a specific feature, please open a GitHub issue.

cyborgbackup's Issues

travis: test on py37

can be done on the xenial VM.

Issues with both docker and standalone instalation

Hi,

I exploring Cyborg. I tried setting up both Docker and Standalone ( using DEB ) on a Ubuntu 20 server.

I am able to install but in Docker version adding new host is grayed out ad I am unable to add any new clients. So not able to proceed.

In standalone version, user created using below string, but unable to login to webUI.

echo "from django.contrib.auth import get_user_model; User = get_user_model(); User.objects.create_superuser('[email protected]', 'admin')" | python3 /cyborgbackup/manage.py shell

Please guide me how can i get this working.

Thanks,
Vijay

Client Preparation Warning.

Hello i installed cyborgbackup 0.4 with the deb file on Debian10.

I have a Problem with the repo preparation.

The Log:
Identity added: /tmp/cyborgbackup_4587_4dnd9_0y/credential_cyborgbackup_ssh_key (root@borgbackup01)
Warning: Permanently added 'borgbackup.linux4you.at,85.235.67.38' (ECDSA) to the list of known hosts
Warning: Permanently added 'borgbackup.linux4you.at,85.235.67.38' (ECDSA) to the list of known hosts
Check Curl
Check Borg
Push repository status to CyBorgBackup
Client Preparation Warning. Repository initialized but unable to update repository object on CyBorgBackup.
Launch this query : curl -f -s -X PATCH -H 'Content-type: application/json' -H 'Authorization: Token b3f6449d22b50025238b9b5a854214ceb00fd567' --data '{"ready": true}' http://127.0.0.1:8000/api/v1/repositories/2/

The Curl Query has no effect.
I have no Port 8000 binding on my system.

Best Regards

How to specify ssh port?

The remote server for storing backups is not on the standard SSH port. How do I specify a custom port for the repository?

I tried:
[email protected]:2222:path/to/dir
[email protected]:2222:path:/to/dir
ssh://[email protected]:2222//path/to/dir

But none of the above methods gave results. In the first two cases it tries to connect to port 22, in the last one - syntax error.

how to add an existing repository that does not have an repository key

as the title of the topic indicates, can we add an existing repository that was created without a repository key.
Thank you

seen borgbase project?

did you have a look at it yet?

maybe there are opportunities for collaboration?

Use https instead of git in requirements.txt

When building the docker image behind a relatively restrictive proxy, the command git clone -q git://github.com/GaetanF/django-elasticsearch-dsl-drf throws an error.

Simply using https://github.com/GaetanF/django-elasticsearch-dsl-drf.git instead fixes the issue.

Can' login to UI

In debug console I this:

ERROR TypeError: /name="csrfmiddlewaretoken" value="([0-9a-zA-Z]+)"/.exec(...) is null
authenticate djangopassword-strategy.ts:182

Prepare Repository Error

Check Curl
Check Borg
grep: /borg/README: No such file or directory
Initialize Repository.
borg init --encryption=repokey
By default repositories initialized with this version will produce security
errors if written to with an older version (up to and including Borg 1.0.8).
If you want to use these older versions, you can disable the check by runnning:
borg upgrade --disable-tam '/borg'
See https://borgbackup.readthedocs.io/en/stable/changes.html#pre-1-0-9-manifest-spoofing-vulnerability for details about the security implications.
Push repository status to CyBorgBackup
{"id":2,"uuid":"293efe22-2837-4156-b76a-e31d214cf34a","url":"/api/v1/repositories/2/","name":"Backup Storage","path":"[email protected]:/borg","repository_key":"XXXXX","original_size":0,"compressed_size":0,"deduplicated_size":0,"ready":true,"enabled":true,"created":"2023-04-13T16:01:26.904601Z","modified":"2023-04-13T17:29:34.250240Z
Repository Preparation OK
bash: line 0: exit: rm: numeric argument required

[BUG] Policy Type - mysql - extra vars missing

Hello,

• Documentation: https://cyborgbackup.readthedocs.io/_/downloads/en/latest/pdf/
• Point: 3.3.5 Policy Type
• Section: mysql
• Quote

You can to specify user,password and database/s in extra vars:

However there are no extra vars available in GUI for Mysql, these exist for example for Piped Backup but not for Mysql

• Funny thing is that I can click on PipedBackup,
• Open dev tool,
• Copy the div with Extra-Vars,
• Switch in select to Mysql Policy
• Define the policy as documentation says,
• Save it and let it run.

Result:

I've made sure that backup content is fine and it is fine.

So this extra vars is defo a bug.

cyborg start (more a st***d question?)

Hi,
I've installed cyborg successfully. But it doesn't start. No info / documentation about how to start it or fixing issues. Nginx displays just a 502 gateway error after install.

cyborgbackup/cyborgbackup:latest not on the Docker Hub?

The instructions at https://cyborgbackup.readthedocs.io/en/latest/user/install.html#install refer to wget https://raw.githubusercontent.com/cyborgbackup/cyborgbackup/master/docker-compose.yml which has cyborgbackup/cyborgbackup:latest. As far as I can tell, this does not exist on the Docker Hub, and therefore I got this error message:

I switched it to milkywan/cyborgbackup:latest per https://hub.docker.com/r/milkywan/cyborgbackup and managed to get the app up and running, but I think it's an old version from 2018.

consider using an organization account instead of personal account

depending on how you want to work on this project (rather as a one-man show or as a team of contributors), it might make sense to move the repo to an organization account (if you want to make it more attractive for contributors).

Default Password

Greetings..
Installed Debian package and got web page, but do not know how to set an email / password to log in.
Thank you in advance
Mark..

Docker: ModuleNotFoundError: No module named 'cyborgback`

as title, i have this exception whit docker-compose up command

REST Api Authentication broker

Accessing api/v1/me/ without Authentication:
{ "detail": "Authentication credentials were not provided." }

With Authentication:

<!DOCTYPE html>
<html lang="en">
    <head>
        <meta http-equiv="content-type" content="text/html; charset=utf-8">
            <meta name="robots" content="NONE,NOARCHIVE">
                <title>WrappedAttributeError
          at /api/v1/me/</title>
...
...
                    <h1>WrappedAttributeError
       at /api/v1/me/</h1>
                    <pre class="exception_value">&#39;User&#39; object has no attribute &#39;username&#39;</pre>
                    <table class="meta">
                        <tr>
                            <th>Request Method:</th>
                            <td>GET</td>
                        </tr>
                        <tr>
                            <th>Request URL:</th>
                            <td>http://omv:8000/api/v1/me/</td>
                        </tr>
                        <tr>
                            <th>Django Version:</th>
                            <td>2.2.19</td>
                        </tr>
                        <tr>
                            <th>Exception Type:</th>
                            <td>WrappedAttributeError</td>
                        </tr>
                        <tr>
                            <th>Exception Value:</th>
                            <td>
                                <pre>&#39;User&#39; object has no attribute &#39;username&#39;</pre>
                            </td>
                        </tr>
                        <tr>
                            <th>Exception Location:</th>
                            <td>/cyborgbackup/cyborgbackup/api/authentication.py in authenticate, line 18</td>
                        </tr>

Cyborgbackup-api-log:
user_auth_tuple = authenticator.authenticate(self) File "/cyborgbackup/cyborgbackup/api/authentication.py", line 18, in authenticate username = ret[0].username if ret[0] else '<none>' rest_framework.request.WrappedAttributeError: 'User' object has no attribute 'username'

Docker images fail on celery after docker-compose up

I used the documentation available here https://cyborgbackup.readthedocs.io/en/latest/user/install.html and the fixes in #36 to build the Docker images. Unfortunately, they do not completely come up correctly. The following repeats without any end:

celery_1 | 2020-10-12 02:50:39,225 DEBUG cyborgbackup.main.utils.task_manager Running CyBorgBackup task manager.
celery_1 | [2020-10-12 02:50:39,225: DEBUG/ForkPoolWorker-1] Running CyBorgBackup task manager.
celery_1 | 2020-10-12 02:50:39,231 DEBUG cyborgbackup.main.scheduler Starting Scheduler
celery_1 | [2020-10-12 02:50:39,231: DEBUG/ForkPoolWorker-1] Starting Scheduler
celery_1 | 2020-10-12 02:50:46,942 DEBUG cyborgbackup.main.tasks Compute Borg Size Report
celery_1 | [2020-10-12 02:50:46,942: DEBUG/ForkPoolWorker-1] Compute Borg Size Report
celery_1 | 2020-10-12 02:50:56,942 DEBUG cyborgbackup.main.tasks Compute Borg Size Report
celery_1 | [2020-10-12 02:50:56,942: DEBUG/ForkPoolWorker-1] Compute Borg Size Report
celery_1 | 2020-10-12 02:50:56,963 DEBUG cyborgbackup.main.tasks Last scheduler run was: 2020-10-12 02:50:26.941330+00:00
celery_1 | [2020-10-12 02:50:56,963: DEBUG/ForkPoolWorker-1] Last scheduler run was: 2020-10-12 02:50:26.941330+00:00
celery_1 | 2020-10-12 02:50:59,223 DEBUG cyborgbackup.main.utils.task_manager Running CyBorgBackup task manager.
celery_1 | [2020-10-12 02:50:59,223: DEBUG/ForkPoolWorker-1] Running CyBorgBackup task manager.
celery_1 | 2020-10-12 02:50:59,228 DEBUG cyborgbackup.main.scheduler Starting Scheduler
celery_1 | [2020-10-12 02:50:59,228: DEBUG/ForkPoolWorker-1] Starting Scheduler

Thank you in advance.

consider moving to a src/ code layout

having the sw package in the workdir causes misc. sometimes strange issues, so better move that to below src/.

pypa/packaging.python.org#320 see links there.

proxy_pass nginx always redirect base

Hi

First of all, thanks for your great work.

I want to set nginx in front of docker port (bind 127.0.0.1) with proxy_pass.
But I didn't find the right configuration for it to work.
Always redirect to my domain name base on "auth.html".
But I want to stay on my domain name base with "cyborgbackup" after like that:

https://myndd/cyborgbackup/

I've try to redirect all to "static" but didn't work too.

For example:
https://myndd/cyborgbackup/
redirect:
https://myndd/auth.html
Result naturely 404.

I've try lot of option but don't find the good one:

proxy_redirect off;
proxy_redirect http://cyborgbackup http://$host;
proxy_set_header Host $host;
rewrite /cyborgbackup/(.*) /$1 break;
proxy_pass http://cyborgbackup/;
proxy_pass http://cyborgbackup;
proxy_pass http://cyborgbackup/$1;

(upstream cyborgbackup 8000)

If you've an idea :)

Thanks and have a good day!

[Api] Catalog State Down

Hello,

I've managed to make the project up and running, but I feel like I'm stuck on last step maybe (?).

I cannot make the Catalog go green no matter what I do:

The catalog job has no log entry at all (the log window and file are empty)

Catalog in GUI is empty

I didn't managed to make the Backup Job yet run succesfully, as I need to get some externally reachable ftp server yet, but can it be that the catalog keeps being empty because the client fails calling the repo (as mentioned, the target ftp is only available locally for containers now), so celery` sees it, client ofc not.

Debian10 deb installation service status

False please delete

Docker Volumes for Repository

Hi,

I am missing the point where to save the Backup-Data to?

I am right, that when I add a repository with path /mnt/test-repo , thats is should be saved to one of the docker containers? Or better, where I need to mount a docker-volume for the repositorys?
This part is missing?

I also dont really get, how the whole process works....
When I understand it right, it works like:

1. CyBorg initiate a backup job -> it connect with ssh to the client which is defined in the policy.
2. there it will run borg locally? but for my understanding, it needs to connect again to a remote ssh host?

Or it pipes output or mount remote sshfs or what?

I am currently running borgmatic, but I like a webinterface to browse the backups.
Maybe somebody get light up how CyBorgBackup works?

EDIT:
Ok, what I have found in the Docs:

A repository must be defined to start backup. The path must be in URI format as the following : user@fqdn:path

Really? Again to a SSH Host? Is the Client connecting to that place? Or how does it work?
Would it not be nice, if CyBorgBackup can handle this task?

Regards,
Thomas

User Documentation/Quick Start

Hello,

I find the documentation under https://cyborgbackup.readthedocs.io/en/latest/user/quickstart.html# quite odd, to be honest.
That is not a Quick Start, but a look deep into the engine room.
Is there a User-Documentation?
I already fail getting something similar to the screenshots https://cyborgbackup.readthedocs.io/en/latest/user/screenshots.html.
When accessing host:8000 I only get the status page (System Status)

Greetings,
Hendrik

mysqldump & special characters in the password

If the MySQL database user password contains special characters, the backup process is not possible.

Log:
bash: line 1: VGNmggFy8v9KL!: command not found
Usage: mysqldump [OPTIONS] database [tables]
OR mysqldump [OPTIONS] --databases DB1 [DB2 DB3...]
OR mysqldump [OPTIONS] --all-databases
OR mysqldump [OPTIONS] --system=[SYSTEMOPTIONS]]
For more options, use mysqldump --help
Remote: Host key verification failed
Connection closed by remote host. Is borg working on the server?

The password must be inside quotes.
Like that:

mysqldump -u username -p"password" dbname....

Dependencies problems

Hello,

When I install the last release(v0.4), on Debian 10.3, I had some dependencies errors

• elasticsearch is not available in the apt repo
• the release note said you use now systemd, but supervisor's package is require.
• git is require for one python dependency but is not in requirements

FYI

repo and client preparation script

Bonsoir,

j'ai quelques question à propos de la configuration et de l'utilisation de cyborgbackup.
J'ai procédé à l'installation de cyberbackup sur une machine physique sous ubuntu 20.04 en utilisant la méthode docker et l'installation est pleinement fonctionnelle.
J'ai accès à l'API et à l'UI.

L'ip de ma machine physique est "192.168.1.100", le hostname est "saveborg" et les noms des utilisateurs de cette machine sont "root" et"masterbalby".
Le but de l'opération est de sauvegarder le répertoire "/etc" de ma machine physique dans un repo BORG situé sur la meme machine dans "/home/masterbalby/BORG".

Si j'ai bien compris le fonctionnement de cyborgbackup le process est le suivant pour créer une nouvelle sauvegarde :

1. je renseigne la page setup
2. je crée un repo
3. j'ajoute un client (la machine à sauvegarder)
4. je crée un schedule
5. et pour finir je crée le policy
6. je lance ensuite le policy est normalement la sauvegarde s'effectue.

Mais voila ça ne se passe pas comme prévu...

Tout d'abord l'étape 1 :
Quelles sont les champs obligatoires dans mon cas ?
J'ai renseigné le champs "Ssh password" par mon mot de passe (qui est le meme pour l'utilisateur "root" et "masterbalby").
A quoi servent les champs "Ssh Key" et "Url" ?
Sont-ils importants ?

Ensuite l'étape 2 :
Je remplis de la manière suivante la page "Add Repository":

• champs "Name" : test_repo
• champs "Path" : [email protected]:/home/masterbalby/BORG
• champs "Repository Key" : ABCD123456
• je sélectionne "Enabled"
• pour finir je clique sur "SAVE"
  Le problème est que rien ne se passe dans "/home/masterbalby/BORG", aucune création de repo.
  J'ai vu sur le wiki que le repo était créé par le script "prepare_repository".
  J'ai donc inspecté le script et ai remarqué que le script recherchait d'abord si curl et borg étaient présents dans le container api. Par défaut curl est présent mais pas borg. Le script essaye alors de télécharger borg sur le github de borgbackup.
  Mais une erreur (je crois) dans le script empèche le téléchargement.

Voici la ligne en cause (la 11) :
binary=$(curl -sL $ttps://api.github.com/repos/borgbackup/borg/releases/latest | grep "browser_download_url" | grep 'borg-linux64"' | cut -d : -f 2,3 | tr -d \")

Si on enlève le " juste après borg-linux64 le script arrive désormais à télécharger borg et à l'installer.

Voici la ligne 11 modifiée :
binary=$(curl -sL $ttps://api.github.com/repos/borgbackup/borg/releases/latest | grep "browser_download_url" | grep 'borg-linux64' | cut -d : -f 2,3 | tr -d \")

De cette manière borg peut s'installer dans le container.
Malheureusement si je recrée le repo il ne sa passe toujours rien.
Est-ce que lorsque l'on click sur "SAVE" le repo est créé tout de suite ?

Etape 3 :
Je remplis de la manière suivante la page "add client":

• champs "Hostname" : "saveborg"
• je séléctionne "enabled"
• et je valide avec "SAVE"
  D'après la doc l'IP et la version de borg du client sont renseignés par le script "preparation_client".
  Mais de la meme manière que à l'etape 2 rien ne se passe.

Je ne suis pas encore passé aux étapes suivantes. Je préfère d'abord arriver à créer mon repo et à configurer le client.
N'arrivant pas à effectuer ces étapes j'ai essayé de créer le repo et le client à partir de l'API.
J'arrive à renseigner les deux formulaires mais le repo ne se crée toujours pas et pour le client je renseigne son adresse IP et une fois validé, dans la liste des clients le client apparait sans adresse IP.

Voila j'en ai terminé pour ce soir.
Je sais que ce post est assez long mais j'ai essayé d'être le plus clair possible.

Par avance merci beaucoup pour l'aide que vous pourrez m'apporter.

---

Good evening,

I have some questions about configuring and using cyborgbackup.
I proceeded to install cyborgbackup on a physical machine running ubuntu 20.04 using the docker method and the installation is fully functional.
I have access to the API and the UI.

The ip of my physical machine is "192.168.1.100", the hostname is "saveborg" and the user names of this machine are "root" and "masterbalby".
The purpose of the operation is to save the directory "/etc" of my physical machine in a BORG repo located on the same machine in "/home/masterbalby/BORG".

If I understood well the operation of cyborgbackup the process is the following to create a new backup:

1. I inform the setup page
2. I create a repo
3. I add a client
4. I create a schedule
5. and finally I create the policy
6. I then launch the policy and normally the backup is done.

But it doesn't go as planned...

First of all step 1 :
What are the mandatory fields in my case ?
I filled the field "Ssh password" with my password (which is the same for the user "root" and "masterbalby").
What are the "Ssh Key" and "Url" fields for ?
Are they important ?

Then step 2 :
I fill the "Add Repository" page in the following way:

• fields "Name": test_repo
• fields "Path" : [email protected]:/home/masterbalby/BORG
• "Repository Key" fields: ABCD123456
• I select "Enabled"
• to finish I click on "SAVE".

The problem is that nothing happens in "/home/masterbalby/BORG", no repo creation.
I saw on the wiki that the repo was created by the "prepare_repository" script.
So I inspected the script and noticed that the script first looked to see if curl and borg were present in the api container. By default curl is present but not borg. The script then tries to download borg to the borgbackup github.
But an error (I think) in the script prevents the download.

Here is the line in question (the 11) :
binary=$(curl -sL $ttps://api.github.com/repos/borgbackup/borg/releases/latest | grep "browser_download_url" | grep 'borg-linux64"' | cut -d : -f 2,3 | tr -d \\")

If we remove the " just after borg-linux64 the script now manages to download borg and install it.

Here is the modified line 11 :
binary=$(curl -sL $ttps://api.github.com/repos/borgbackup/borg/releases/latest | grep "browser_download_url" | grep 'borg-linux64' | cut -d : -f 2,3 | tr -d \\")

In this way borg can be installed in the container.
Unfortunately if I recreate the repo it still doesn't work.
When I click on "SAVE" is the repo created right away?

Step 3:
I fill in the "add client" page as follows:

• fields "Hostname" : "saveborg"
• I select "enabled".
• and I validate with "SAVE"
  According to the doc the IP and the borg version of the client are filled by the script "preparation_client".
  But in the same way as in step 2 nothing happens.

I haven't gone to the next steps yet. I prefer to create my repo and configure the client first.
Since I can't do these steps I tried to create the repo and the client from the API.
I manage to fill in both forms but the repo is still not created and for the client I fill in his IP address and once validated, in the client list the client appears without an IP address.

That's it for tonight.
I know this post is quite long but I tried to be as clear as possible.

In advance thank you very much for the help you can give me.

VM Backup script

Hello @GaetanF, thanks for your exceptional work. I am looking into VM module docs, and I like to contribute to your project. My VM backup process is as follow:

• call OpenNebula API and create a disk snapshot
• call 3PAR API and export snapshot via FiberChannel to the backup server
• discover new LUN on the backup server
• do dd if | borg create...
• flush LUN from the backup server
• call 3PAR API and unexport snapshot
• call OpenNebula API and delete the snapshot

In VM module docs you write, that script has to return data to stdout, and you pass it to borg create, which is fine, but I need to post backup tasks, flush LUN, unexport, delete...

Would you point me to source code, where I should do extension?

Thanks!

error in prepare client

the error is:
Enter passphrase for /tmp/cyborgbackup_72_3zjwwfat/credential_cyborgbackup_ssh_key

i install cyborgbackup use docker-compose
it looks like to need input the ssh_key password,i read the doc,but could not find how to config this,help me ,thanks

link to here from borg's community repo

https://github.com/borgbackup/community see there, make a PR.

prune job failed

Fresh installation (docker), back job and catalog job runs successfull, prune job fails with:

Identity added: /tmp/cyborgbackup_15_kar_w5ge/credential_cyborgbackup_ssh_key ([email protected]
sh: 1: borg: not found

Docker installation process

Hi,
i can't complete the docker installation process.
I receive this error: "No such service: web" after i tape "docker-compose exec web /bin/bash".

Regards,
Giacomo

Authentification sur l'api

Bonjour,

Quelle est le paramètre à utiliser pour s'authentifier sur l'api de cyborg afin d'effectuer des actions automatisées ?
J'ai beau lire et relire la documentation je ne trouve aucune information sur ce sujet.

Prune Job Failed

Log:

Identity added: /tmp/cyborgbackup_181_mfs80m86/credential_cyborgbackup_ssh_key ([email protected])
sh: 1: borg: not found

Borg is installed on both the client and the repository. All previous tasks have been completed successfully.

Unable to connect to the interface

Hello everyone and bravo for this project.
I'm trying to install cyborgbackup using the method from source code and docker.
I think the installation is going well but when I try to connect to the interface from http://localhost:8000, I get to the following page and not the authentication page
.
A little help would be welcome.
Cédric

Client Preparation Warning. behind a firewall

Hello,

I've the same kind of error than #32 , but this is because it's the host who's doing is preparation (which is in another site) who try to do a curl on the backup server internal network.
There are not on the same network, and the Firewall can authorize ssh going out but not TCP connexion going in.

It's possible to push the client characteristics through initial the ssh connexion ?

Yes I suppose that I can expose the API port through an nginx with an SSL certificate, their is a token..., but I would like not... :/

[IDEA] container web has to be restarted a 2nd time after restarting the project

Hello,

web container is breaking when you start the project, restart it. It is mainly due to the fact that postgreSQL DB is not ready to receive connections. A manual restart of web container is required:

docker-compose logs web

ouput:

Attaching to cyborgbackup_web_1
web_1                | Traceback (most recent call last):
web_1                |   File "/usr/local/lib/python3.7/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
web_1                |     self.connect()
web_1                |   File "/usr/local/lib/python3.7/site-packages/django/db/backends/base/base.py", line 195, in connect
web_1                |     self.connection = self.get_new_connection(conn_params)
web_1                |   File "/usr/local/lib/python3.7/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
web_1                |     connection = Database.connect(**conn_params)
web_1                |   File "/usr/local/lib/python3.7/site-packages/psycopg2/__init__.py", line 127, in connect
web_1                |     conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
web_1                | psycopg2.OperationalError: could not connect to server: Connection refused
web_1                |  Is the server running on host "postgres" (172.24.0.2) and accepting
web_1                |  TCP/IP connections on port 5432?
web_1                | The above exception was the direct cause of the following exception:
web_1                | Traceback (most recent call last):
web_1                |   File "/cyborgbackup/manage.py", line 22, in <module>
web_1                |     execute_from_command_line(sys.argv)
web_1                |   File "/usr/local/lib/python3.7/site-packages/django/core/management/__init__.py", line 381, in execute_from_command_line
web_1                |     utility.execute()
web_1                |   File "/usr/local/lib/python3.7/site-packages/django/core/management/__init__.py", line 375, in execute
web_1                |     self.fetch_command(subcommand).run_from_argv(self.argv)
web_1                |   File "/usr/local/lib/python3.7/site-packages/django/core/management/base.py", line 323, in run_from_argv
web_1                |     self.execute(*args, **cmd_options)
web_1                |   File "/usr/local/lib/python3.7/site-packages/django/core/management/base.py", line 364, in execute
web_1                |     output = self.handle(*args, **options)
web_1                |   File "/usr/local/lib/python3.7/site-packages/django/core/management/base.py", line 83, in wrapped
web_1                |     res = handle_func(*args, **kwargs)
web_1                |   File "/usr/local/lib/python3.7/site-packages/django/core/management/commands/migrate.py", line 87, in handle
web_1                |     executor = MigrationExecutor(connection, self.migration_progress_callback)
web_1                |   File "/usr/local/lib/python3.7/site-packages/django/db/migrations/executor.py", line 18, in __init__
web_1                |     self.loader = MigrationLoader(self.connection)
web_1                |   File "/usr/local/lib/python3.7/site-packages/django/db/migrations/loader.py", line 49, in __init__
web_1                |     self.build_graph()
web_1                |   File "/usr/local/lib/python3.7/site-packages/django/db/migrations/loader.py", line 212, in build_graph
web_1                |     self.applied_migrations = recorder.applied_migrations()
web_1                |   File "/usr/local/lib/python3.7/site-packages/django/db/migrations/recorder.py", line 73, in applied_migrations
web_1                |     if self.has_table():
web_1                |   File "/usr/local/lib/python3.7/site-packages/django/db/migrations/recorder.py", line 56, in has_table
web_1                |     return self.Migration._meta.db_table in self.connection.introspection.table_names(self.connection.cursor())
web_1                |   File "/usr/local/lib/python3.7/site-packages/django/db/backends/base/base.py", line 256, in cursor
web_1                |     return self._cursor()
web_1                |   File "/usr/local/lib/python3.7/site-packages/django/db/backends/base/base.py", line 233, in _cursor
web_1                |     self.ensure_connection()
web_1                |   File "/usr/local/lib/python3.7/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
web_1                |     self.connect()
web_1                |   File "/usr/local/lib/python3.7/site-packages/django/db/utils.py", line 89, in __exit__
web_1                |     raise dj_exc_value.with_traceback(traceback) from exc_value
web_1                |   File "/usr/local/lib/python3.7/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
web_1                |     self.connect()
web_1                |   File "/usr/local/lib/python3.7/site-packages/django/db/backends/base/base.py", line 195, in connect
web_1                |     self.connection = self.get_new_connection(conn_params)
web_1                |   File "/usr/local/lib/python3.7/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
web_1                |     connection = Database.connect(**conn_params)
web_1                |   File "/usr/local/lib/python3.7/site-packages/psycopg2/__init__.py", line 127, in connect
web_1                |     conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
web_1                | django.db.utils.OperationalError: could not connect to server: Connection refused
web_1                |  Is the server running on host "postgres" (172.24.0.2) and accepting
web_1                |  TCP/IP connections on port 5432?

In docker-compose.yml

  web:
    image: cyborgbackup/cyborgbackup:latest
    command: bash -c "python /cyborgbackup/manage.py migrate --noinput && python /cyborgbackup/manage.py runserver 0.0.0.0:8000"
    env_file:
      - .env
    ports:
      - "8000:8000"
    depends_on:
      - postgres
      - rabbitmq

could be modified to :

  web:
    image: cyborgbackup/cyborgbackup:latest
    command: bash -c "python /cyborgbackup/manage.py migrate --noinput && python /cyborgbackup/manage.py runserver 0.0.0.0:8000"
    env_file:
      - .env
    ports:
      - "8000:8000"
    depends_on:
      - postgres
      - rabbitmq
    restart: unless-stopped

Thanks for this awesome project
Regards

Broken install instructions

Hi,
The latest changes you made in Update Docs and ScreenShots have broken the install process when following the instructions at https://cyborgbackup.readthedocs.io/en/latest/user/install.html#install

In-particular changing the filename to "docker-compose.full.yml" results in an error when running

$ docker-compose up
ERROR: 
        Can't find a suitable configuration file in this directory or any
        parent. Are you in the right directory?

        Supported filenames: docker-compose.yml, docker-compose.yaml

Then if I manually rename the filename to "docker-compose.yml" I get the following error

$ docker-compose up
ERROR: Service 'ui' depends on service 'web' which is undefined.

Thanks

decide about license, make it consistent

setup.py says apache license, LICENSE says GPL.

borgbackup is BSD, btw.

Documentation of the architecture / use cases

I read the current docs, but I did not really understand how CyBorgBackup works in a setup of N repositories/destinations and M nodes/sources:

• Is CyBorgBackup installed on each node and manages when and how it backups?
• Is CyBorgBackup installed on the destination host containing the repositories?
• Is CyBorgBackup installed on a separate host?
• Does it fully control everything (e.g. create SSH keys, distribute it to the nodes, configure borg) or do I have to do stuff myself?
• Can it be used as an additional "just get me some information without battling the CLI"-web interface in combination with existing borg and borgmatic stuff?

I think it would be nice to have some kind of walkthrough :-). Because the screenshots and the general idea of having some kind of UI are nice, but in the current state of the docs I would not consider to switch from my borgmatic setup.

consider adding flake8 style checker

it can be invoked e.g. via a separate tox env and makes sure your code follows the usual python coding style conventions. it also warns about some other stuff.

Docker: Port 5432 (postgres) already in use

Hi,
i tried to install cyborgbackup with docker compose but it fails, because the container postgres port 5432 is already in use. I changed the Port in the docker compose yaml. After that localhost:8000 says Catalog has a exclamation mark and i can't login.

Any advice?

Also after executing "docker-compose up" the console shows the whole time logs of the containers. So i need to exit it with CTRL + C and all containers are shutting down. I started them manually with portainer. After that i tried to execute

"docker-compose exec web /bin/bash"

but it says

"service "web" is not running container # 1"

so i can't finish the Docker install instruction :/

consider using setuptools_scm

it can help you a lot with versioning (based on git tags) and maintaining the manifest (based on what's committed to git).

connexion SSH serveur, repo, client

Bonjour,

Je reviens vers vous pour la dernière fois en espérant arriver cette fois à faire fonctionner mon instance de cyborgbackup.
Pour rappel j'ai les plus grandes difficultés à paramétrer cyborgbackup pour qu'il puisse dialoguer avec le repo et le client.

Pour mes tests j'ai installé cyborgbackup version docker sur une Ubuntu 20.04.
J'ai accés à l'UI avec l'adresse "http://localhost:8088" ou "http://192.168.1.100:8088".
J'ai accés à l'API avec l'adresse "http://localhost:8000" ou http://192.168.1.100:8000".
J'ai décidé que le serveur, le repo et le client seront sur la même machine pour faciliter mes tests.
Cette machine s'appelle "cyborgbackup".
L'adresse ip de cette machine est "192.168.1.100"
Le login de cette machine est "masterbalby".
Chemin du repo "/home/masterbalby/BORG".
Chemin à sauvegarder sur le client "/home/masterbalby/A_SAUVER".

Après de nombreux tests je sais que je renseigne mal certains champs de la page "Settings" pour interconnecter le serveur, le repo et le client.
Dans votre réponse précédante vous indiquiez que "CyBorgBackup se repose uniquement sur les clés SSH pour se connecter aux différents clients/repositories.".
C'est de là que provient certainement mon problème.
Pour configurer l'authentification par clé ssh j'ai lancé sur mon "serveur" avec la session "masterbalby" la commande "ssh-keygen -t rsa", j'ai indiqué la passphrase et tout ça a généré dans "/home/masterbalby/.ssh" ma clé privée "id_rsa" et ma clé publique "id_rsa.pub".
Comme tout est connecté par ssh et que le serveur, le repo et le client sont sur la même machine j'ai ajouté le contenu de ma clé publique "/home/masterbalby/.ssh/id_rsa.pub" dans le fichier "/home/masterbalby/.ssh/authorized_keys".
Je peux désormais me connecter en ssh sans mot de passe de mon serveur vers ma machine (qui est la même) en utilisant la commande "ssh [email protected]" et ma passphrase renseignée lors du "ssh-keygen".

Etape suivant : Paramétrer correctement l'UI
Je cite "Dans les settings, SSH Password et SSH Key correspondent à la passphrase de la clé SSH et son contenu.".
J'ai donc dans le champs "SSh key" coller le contenu de ma clé privée "/home/masterbalby/.ssh/id_rsa" de la forme :

-----BEGIN OPENSSH PRIVATE KEY-----
b3BlbnNzaC1rZXktdjEAAAAACmFlczI1Ni1jdHIAAAAGYmNyeXB0AAAAGAAAABCPTI83VW
0SwK7tYIIi7zMvAAAAEAAAAAEAAAGXAAAAB3NzaC1yc2EAAAADAQABAAABgQDgbNgACluL
....
...
...
hWw3GRkRaQwVGkSMTY+bSxZcGDInMMMOC4d6UIdVyJetC1ea/2h8x2u3nMe2hIexQul3EI
5wgHxYp9gMtBQetTrPOXyk/7rpryADyq0GOJvlMDrma12T4L
-----END OPENSSH PRIVATE KEY-----

Dans le champs "SSh password" j'ai renseigné la passphrase de la clé SSH (ssh-keygen).
Pour le champs "URL" j'ai mis "http://192.168.1.100:8000".

Pour la page "add repo" dans le champs "path" j'ai mis "[email protected]:/home/masterbalby/BORG" (login@ip:/chemin/vers/repo).
Et dans le champs "Repository Key" j'ai indiqué la clé souhaitée pour la création du repo.

Pour la page "add client" j'ai juste indiqué dans le champs "hostname" le hostname de mon client à savoir "cyborgbackup" (nom de la machine sur laquelle est installée pour rappel le serveur, le repo et le client.)

Ensuite je crée un schedule, une policy que je lance et rien ne se passe.
Sur la page "Job" apparaissent 4 lignes en erreur (une pour la préparation du repo, une pour la préparation du client, une pour le backup job et une pour le catalog job).
Mais à chaque fois les fenêtres de log sont vide.

Je suis sur le point s'abandonner à contre coeur , mais je m'accroche à trouver une solution car cyborgbackup correspond parfaitement à ce que je recherche pour la sauvegarde de mes différents serveurs.

Par avance merci pour votre aide
Cédric

Restoration Docker/UI

Hello,

I don't see this issue not like a true one, sorry :(
So my question is : "I'm using CyborgBackup-UI (developp version) with docker, I would like to do some restoration through the UI, with docker server (or api) version I've to choose ?"

Thanks ;)
Antoine.

Docker behind reverse proxy

I'm running the containers behind traefik.
When opening the UI over the loadbalancer url (https://mydomain.local, CyBorg Server Address: http://host:8000), i can't login:

Http failure response for http://host:8000/api/login/: 0 Unknown Error

When changing https://mydomain.local to http://host:8088 it's working

Edit: http is working

Error on Create [object Object]

New install of Cyborgbackup and Cyborgbackup-ui on debian 12
Installed using the deb packages.
when I try to add or create anything I get
Error on Create
[object Object]

would like to how to debug

How to backup separate directories?

CyBorgBackup interface login

Hi
My machine is. VPS / Debian 11 .The installation was flawless.I get an error when I try to log in
Http failure response for http://1.1.1.1:8000/api/login/: 0 Unknown Error

nginx config :

/etc/nginx/sites-available/cyborg.

map $http_upgrade $connection_upgrade {
    default upgrade;
    ''      close;
}
upstream uwsgi_cyborgbackup {
    server localhost:8150;
    }

upstream daphne_cyborgbackup {
    server localhost:8151;
}

server {
    listen 8088;
    listen [::]:8088;

    server_name mydomain.com;
    keepalive_timeout 65;
    client_max_body_size 10M;

    root /usr/share/cyborgbackup-ui/dist;

    location / {
        try_files $uri $uri/ /index.html;
    }

    location /api {
        uwsgi_read_timeout 120s;
        uwsgi_pass uwsgi_cyborgbackup;
        include /etc/nginx/uwsgi_params;
    }

    location /static {
        alias /opt/cyborgbackup/cyborgbackup/static;
    }

    location /websocket {
    proxy_pass http://daphne_cyborgbackup;
    proxy_http_version 1.1;
        proxy_buffering off;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto https;
        proxy_set_header Host $http_host;
        proxy_redirect off;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection $connection_upgrade;
    }
}

Is it the fault of the wrong address Specific CyBorg server?

Ps,Same problem after installing on Windows

ERROR/MainProcess "failed to resolve broker hostname"

what can i do with this problem?
test on debian, ubuntu, Kali, error in -docker-compose up

celery_beat_1 ERROR/MainProcess] beat: Connection error: failed to resolve broker hostname. Trying again in xx.x seconds...

celery_1 ERROR/MainProcess] consumer: Cannot connect to amqp://cyborgbackup:**@rabbitmq:5672/cyborgbackup: failed to resolve broker hostname.

Release 0.5

I have decided to redevelop the UI interface of CyBorgBackup to facilitate feature addition on next release.

CyBorgBackup main engine and UI will be separated in two packages and in two GitHub repository.

This release come with some new features and bugfix.

Bugs :

• Policy next schedule not updated after Schedule object update
• Catalog job waiting archive name from the Stdout of the backup job
• Move root of UI (Bug #18)
• Dependencies fix (Bug #26 #24 )

Features :

• JWT Authentication
• UI can connect to multiple CyBorgServer to administrate them from a central management
• Restore Job
• Job cleaning
• UI Theming
• Proxmox Job Policy

This issue will be updated until release.