galeone / bacup

An easy-to-use backup tool designed for servers.

Hi 👋

I write code, I write articles, I wrote a book (I like TensorFlow).

It looks like I write all day 🤔

bacup's Issues

What and why

I want to start this project using Rust, so I have a good excuse to learn the language by having a project to complete.

The idea

Backing up a whole VPS is stupid, backing up only the data might be a good idea, but the data we want to backup depends on the services hosted on the VPS; the problem is not trivial.

The obvious way of using rsync + systemd timers or whatever just bores me, and it's old and not trivially configurable (since everything depends on the services we want to backup).

What to backup

What the user wants. There might be thousands of different scenarios, like:

• I have a database with PostgreSQL, I'd like to execute the pg_dumpall periodically and backup this dump.
• Same stuff with MySQL/MariaDB
• Redis? I need to copy the file
• Sqlite? Like Redis
• What about the configuration files? I usually need them.
• What if I have different services interacting with each other?
• What if I have a webserver? I should back up the web server configuration (e.g. nginx.conf and all the other parts of the configuration)
• What if I have a PHP website? A Python server?
• A Go application?

There are TONS of use cases, and every use case is completely user-dependent (e.g. I like to configure my system as I want, and my systems are almost completely customized. All the standard backup tools just don't work for me).

Where to backup?

It depends. Do you like the cloud? I don't. But there's some good cheap storage on "the cloud", so you might be interested in backing up your stuff there.

• AWS
• Google Cloud Storage
• Other decent cloud providers
• My self-hosted server
• Are you a terrible person and you want to backup everything on git? (yeah GitHub has infinite free storage for small files...)

The self-hosted server can be trivial or can be something nice, a bacup client and a bacup server might be nice to have.

How to backup

It should be trivial.

Every time I configure a new server, I should just:

• download the bacup client
• configure it
• start it as a background service and forgot about it

The configuration part should be trivial. I should be able to:

• define with a list the services I want to backup and how I want to backup them, something like

  foo:
     cloud: aws
     address: whatever
     authentication: the_token
  
  dbname:
     type: postgres
     what: cluster
     when: daily
     incremental: true
     where: foo

and if PostgreSQL is in some well-known location, then the pg_dump_all -f filename.sql can be executed every day, to create the backup file and once created upload it to the configured cloud provider.

With the same simplicity I should be able to backup: files (invoking rsync in the background if possible, why not), folders, and invoke the correct backup tool for the specified service (if it has it, like PostgreSQL).

So this is more or less the idea.

Any comment/suggestion is welcome.

The design of the software architecture will be in a different issue - this issue is only to brainstorm about the idea, the simplicity of usage, how it should work from the user point of view (yeah it's like a user-story, although I don't "love" all these agile things).

Build error

on the latest stable rust version:

error[E0308]: mismatched types
   --> /home/debian/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-cron-scheduler-0.7.6/src/job/mod.rs:346:41
    |
346 |                       .checked_add_signed(time::Duration::seconds(
    |  ______________________------------------_^
    | |                      |
    | |                      arguments to this method are incorrect
347 | |                         instant.duration_since(Instant::now()).as_secs() as i64,
348 | |                     ))
    | |_____________________^ expected `TimeDelta`, found `Duration`
    |
note: method defined here
   --> /home/debian/.cargo/registry/src/index.crates.io-6f17d22bba15001f/chrono-0.4.35/src/datetime/mod.rs:442:12
    |
442 |     pub fn checked_add_signed(self, rhs: TimeDelta) -> Option<DateTime<Tz>> {
    |            ^^^^^^^^^^^^^^^^^^

error[E0308]: mismatched types
   --> /home/debian/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-cron-scheduler-0.7.6/src/job/mod.rs:444:41
    |
444 |                     .checked_add_signed(time::Duration::seconds(duration.as_secs() as i64))
    |                      ------------------ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expected `TimeDelta`, found `Duration`
    |                      |
    |                      arguments to this method are incorrect
    |
note: method defined here
   --> /home/debian/.cargo/registry/src/index.crates.io-6f17d22bba15001f/chrono-0.4.35/src/datetime/mod.rs:442:12
    |
442 |     pub fn checked_add_signed(self, rhs: TimeDelta) -> Option<DateTime<Tz>> {
    |            ^^^^^^^^^^^^^^^^^^

error[E0308]: mismatched types
   --> /home/debian/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-cron-scheduler-0.7.6/src/job/mod.rs:568:62
    |
568 |                         .and_then(|nt| nt.checked_add_signed(time::Duration::seconds(r as i64)))
    |                                           ------------------ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expected `TimeDelta`, found `Duration`
    |                                           |
    |                                           arguments to this method are incorrect
    |
note: method defined here
   --> /home/debian/.cargo/registry/src/index.crates.io-6f17d22bba15001f/chrono-0.4.35/src/datetime/mod.rs:442:12
    |
442 |     pub fn checked_add_signed(self, rhs: TimeDelta) -> Option<DateTime<Tz>> {
    |            ^^^^^^^^^^^^^^^^^^

error[E0308]: mismatched types
   --> /home/debian/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-cron-scheduler-0.7.6/src/scheduler.rs:167:67
    |
167 | ...                   nt.checked_add_signed(time::Duration::seconds(r as i64))
    |                          ------------------ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expected `TimeDelta`, found `Duration`
    |                          |
    |                          arguments to this method are incorrect
    |
note: method defined here
   --> /home/debian/.cargo/registry/src/index.crates.io-6f17d22bba15001f/chrono-0.4.35/src/datetime/mod.rs:442:12
    |
442 |     pub fn checked_add_signed(self, rhs: TimeDelta) -> Option<DateTime<Tz>> {
    |            ^^^^^^^^^^^^^^^^^^

For more information about this error, try `rustc --explain E0308`.
error: could not compile `tokio-cron-scheduler` (lib) due to 4 previous errors
warning: build failed, waiting for other jobs to finish...

Missing remote: localhost

Perhaps I have already configured some network filesystem and I have a working mount point (sshfs, nfs, samba, whatever), everything bacup has to do is just copy the files in the correct local location.

Missing remotes: git

It's a very strange way of doing backup, but it's something I've already seen (and done, shame on me).

If a local folder is a git repository, then we can basically do:

git checkout <backup-branch> # optional?
git add . -A
git commit -m 'backup'
git push origin

Using SSH is mandatory, hence we can re-use the private key control part of the SSH remote + adding the which(git) and be sure that git is installed in the system, and create a new type that implements the Uploader trait and it behaves similarly to the SSH service.

The support for backing up compressed files and compressed folders can be added too, although it's horrific loading archives on Git (especially for backing up). But anyway, it's a feature we can offer just in case.

Missing services: docker-based service?

Let's say we have a containerized service made in this way:

• webserver (Nginx)
• backend (PHP)
• database1 (PostgreSQL)
• database2 (Redis)
• frontend (HTML,CSS, js, ...)

All these services are inside a docker container, and they are configurable through configuration files often mounted inside a volume.

Doing the backup of the configuration files is easy: we can just use the "folders" service.

But what about the data of the different databases? We could use the folders service and copy and archive all of them, but if the database is in use (and it's often the case), there's a non-zero chance of copying some file that's in a nonstable state.

Hence, the possibility of invoking the correct commands on the container should be provided, in order to ask the running DB to do the dump, store it inside the volume (accessible from the host), and then back up the correctly created dump.

An interface for being compatible with docker should be defined.

Support for the first cloud provider and general structure

I have to start from somewhere:

• Implement the AWS support
• Implement the configuration file parsing
• Implement the generic project structure: client/server application (currently develop only the server - that's the daemon that runs the backup. While the client currently might remain empty, but ideally it should be a handy tool that completely restores the services backed-up using backup)

Missing services: MySql/MariaDB

Another very common database used server-side is MariaDB/MySQL.

Similar to the PostgreSQL type, this new type must implement the Service trait and only depend on the system commands for doing the dump. Do not depend on libraries DB specific.

Missing remotes: Google Cloud Storage

The support for Google Cloud storage is nice to have.

The new type should implement the Uploader trait and behave similarly to AWSBucket.

Missing remotes: Google Drive

Google Drive support as a remote would be nice!

Missing documentation

Currently, the code is not well documented, any kind of documentation is gold.

Improve logging

The logging is still poor, need to add an info message when:

• the backup starts to "dump"
• the execution of the dump is completed
• if there's compression when the compression starts
• if there's compression when the compression ends

Missing CI

We have some tests, but we don't run them in the CI.

A TravisCI or something new like GitHub actions must be implemented.

NOTE: some tests require third-party services to be running in the background (e.g. testing the dump of a database, requires the database running and the possibility of calling the DB utils from CLI).

Systemd configuration?

Since this tool is designed for servers, a folder misc/system (or other scripts) that contains a service file would be really handy.

Keep only last "days" "N backups"

This is somehow a must-have feature.
Especially if we are using some free-tier remote that has limited storage