Does backup pruning happens properly and automatically? about borg-backup

[kelson42]

It seems it does not:

stats__backup__2020-11-07T03:14:09   Sat, 2020-11-07 03:14:32 [53a2c790db9c9b4c05087ba7b1e0fe8200530076e7d06feaee47f0e538d23d2e]
stats__backup__2020-11-08T03:12:44   Sun, 2020-11-08 03:12:53 [a039ba0643ce47897213d8fa6ed5931b59baa457b470377ca837ecc8a275be5d]
stats__backup__2020-11-10T03:12:34   Tue, 2020-11-10 03:12:47 [3a2c444e6cebe45868ff3688f031750a460e6b84d05d60241cb739f12a3a9c2f]
stats__backup__2020-11-11T03:12:22   Wed, 2020-11-11 03:12:30 [3033a687eccbed04614e7cc05e1465178a16766a8aa759eb44d915d15b684eba]
stats__backup__2020-11-12T03:12:28   Thu, 2020-11-12 03:12:52 [5621dc7940505f7f68db92a9e5e809ee247ae2f32156bdc92465dacd6b47796b]
stats__backup__2020-11-13T03:12:14   Fri, 2020-11-13 03:12:24 [1b378be83a80555101b0e49105f1816b57e5aa11b9fac206ac0d73025605bca9]
stats__backup__2020-11-14T03:13:55   Sat, 2020-11-14 03:14:27 [8e7232e9464457a5176e5c3e745f01e184243dd402b0f49dd8d9d43c2fa89fbd]
stats__backup__2020-11-15T03:12:18   Sun, 2020-11-15 03:12:35 [6b4bfd9854b5aac4548ef541e53c8588e72545d7bcb14de4fc91224fd4f48019]
stats__backup__2020-11-16T03:13:14   Mon, 2020-11-16 03:13:33 [5b0e46ab05127aaaeb77fef95d3182121ca0c7fffe1a15ea7a929910ca39d48b]
stats__backup__2020-11-17T03:12:21   Tue, 2020-11-17 03:12:41 [58b4e9e8c556bf18103490adf57860a6b35fb1dbdd2ca2f38255ec81de92da63]

[florentk]

@kelson42 actually, today I got this :

stats__backup__2020-11-08T03:12:44   Sun, 2020-11-08 03:12:53 [a039ba0643ce47897213d8fa6ed5931b59baa457b470377ca837ecc8a275be5d]
stats__backup__2020-11-11T03:12:22   Wed, 2020-11-11 03:12:30 [3033a687eccbed04614e7cc05e1465178a16766a8aa759eb44d915d15b684eba]
stats__backup__2020-11-12T03:12:28   Thu, 2020-11-12 03:12:52 [5621dc7940505f7f68db92a9e5e809ee247ae2f32156bdc92465dacd6b47796b]
stats__backup__2020-11-13T03:12:14   Fri, 2020-11-13 03:12:24 [1b378be83a80555101b0e49105f1816b57e5aa11b9fac206ac0d73025605bca9]
stats__backup__2020-11-14T03:13:55   Sat, 2020-11-14 03:14:27 [8e7232e9464457a5176e5c3e745f01e184243dd402b0f49dd8d9d43c2fa89fbd]
stats__backup__2020-11-15T03:12:18   Sun, 2020-11-15 03:12:35 [6b4bfd9854b5aac4548ef541e53c8588e72545d7bcb14de4fc91224fd4f48019]
stats__backup__2020-11-16T03:13:14   Mon, 2020-11-16 03:13:33 [5b0e46ab05127aaaeb77fef95d3182121ca0c7fffe1a15ea7a929910ca39d48b]
stats__backup__2020-11-17T03:12:21   Tue, 2020-11-17 03:12:41 [58b4e9e8c556bf18103490adf57860a6b35fb1dbdd2ca2f38255ec81de92da63]
stats__backup__2020-11-18T03:13:40   Wed, 2020-11-18 03:14:01 [281b266a2dcbb50c6f3227dd63f0397c48b6031b9cb1185ef93ef7eaff8e8ed9]
stats__backup__2020-11-19T03:14:21   Thu, 2020-11-19 03:14:43 [c0d7aec2693fead98a992ef27a1ce65610b01aa8e8e056ecceaf505161d99011]
stats__backup__2020-11-20T03:12:24   Fri, 2020-11-20 03:12:30 [2d7a59da36038fc7f24ecca3c0f880fc76b73b09c378920e0dd418bf836fec9c]

stats__backup__2020-11-07T03:14:09 and stats__backup__2020-11-10T03:12:34 have been deleted.

[kelson42]

@florentk Seems still wrong. Why the backup of the 2020-11-12 is there?

[florentk]

@kelson42 our default configuration has 'keep whithin' at 48h, that's why it keeps two more.

We can see that when I simulate a prune with and without --keep-within option :

root@86fab9ad11dd:/config# borg prune --keep-within 48H -d 7 -w 4 -m 12 -y 1 -n --list [email protected]:repo
Keeping archive: stats__backup__2020-11-20T03:12:24   Fri, 2020-11-20 03:12:30 [2d7a59da36038fc7f24ecca3c0f880fc76b73b09c378920e0dd418bf836fec9c]
Keeping archive: stats__backup__2020-11-19T03:14:21   Thu, 2020-11-19 03:14:43 [c0d7aec2693fead98a992ef27a1ce65610b01aa8e8e056ecceaf505161d99011]
Keeping archive: stats__backup__2020-11-18T03:13:40   Wed, 2020-11-18 03:14:01 [281b266a2dcbb50c6f3227dd63f0397c48b6031b9cb1185ef93ef7eaff8e8ed9]
Keeping archive: stats__backup__2020-11-17T03:12:21   Tue, 2020-11-17 03:12:41 [58b4e9e8c556bf18103490adf57860a6b35fb1dbdd2ca2f38255ec81de92da63]
Keeping archive: stats__backup__2020-11-16T03:13:14   Mon, 2020-11-16 03:13:33 [5b0e46ab05127aaaeb77fef95d3182121ca0c7fffe1a15ea7a929910ca39d48b]
Keeping archive: stats__backup__2020-11-15T03:12:18   Sun, 2020-11-15 03:12:35 [6b4bfd9854b5aac4548ef541e53c8588e72545d7bcb14de4fc91224fd4f48019]
Keeping archive: stats__backup__2020-11-14T03:13:55   Sat, 2020-11-14 03:14:27 [8e7232e9464457a5176e5c3e745f01e184243dd402b0f49dd8d9d43c2fa89fbd]
Keeping archive: stats__backup__2020-11-13T03:12:14   Fri, 2020-11-13 03:12:24 [1b378be83a80555101b0e49105f1816b57e5aa11b9fac206ac0d73025605bca9]
Keeping archive: stats__backup__2020-11-12T03:12:28   Thu, 2020-11-12 03:12:52 [5621dc7940505f7f68db92a9e5e809ee247ae2f32156bdc92465dacd6b47796b]
Would prune:     stats__backup__2020-11-11T03:12:22   Wed, 2020-11-11 03:12:30 [3033a687eccbed04614e7cc05e1465178a16766a8aa759eb44d915d15b684eba]
Keeping archive: stats__backup__2020-11-08T03:12:44   Sun, 2020-11-08 03:12:53 [a039ba0643ce47897213d8fa6ed5931b59baa457b470377ca837ecc8a275be5d]

root@86fab9ad11dd:/config# borg prune  -d 7 -w 4 -m 12 -y 1 -n --list [email protected]:repo
Keeping archive: stats__backup__2020-11-20T03:12:24   Fri, 2020-11-20 03:12:30 [2d7a59da36038fc7f24ecca3c0f880fc76b73b09c378920e0dd418bf836fec9c]
Keeping archive: stats__backup__2020-11-19T03:14:21   Thu, 2020-11-19 03:14:43 [c0d7aec2693fead98a992ef27a1ce65610b01aa8e8e056ecceaf505161d99011]
Keeping archive: stats__backup__2020-11-18T03:13:40   Wed, 2020-11-18 03:14:01 [281b266a2dcbb50c6f3227dd63f0397c48b6031b9cb1185ef93ef7eaff8e8ed9]
Keeping archive: stats__backup__2020-11-17T03:12:21   Tue, 2020-11-17 03:12:41 [58b4e9e8c556bf18103490adf57860a6b35fb1dbdd2ca2f38255ec81de92da63]
Keeping archive: stats__backup__2020-11-16T03:13:14   Mon, 2020-11-16 03:13:33 [5b0e46ab05127aaaeb77fef95d3182121ca0c7fffe1a15ea7a929910ca39d48b]
Keeping archive: stats__backup__2020-11-15T03:12:18   Sun, 2020-11-15 03:12:35 [6b4bfd9854b5aac4548ef541e53c8588e72545d7bcb14de4fc91224fd4f48019]
Keeping archive: stats__backup__2020-11-14T03:13:55   Sat, 2020-11-14 03:14:27 [8e7232e9464457a5176e5c3e745f01e184243dd402b0f49dd8d9d43c2fa89fbd]
Would prune:     stats__backup__2020-11-13T03:12:14   Fri, 2020-11-13 03:12:24 [1b378be83a80555101b0e49105f1816b57e5aa11b9fac206ac0d73025605bca9]
Would prune:     stats__backup__2020-11-12T03:12:28   Thu, 2020-11-12 03:12:52 [5621dc7940505f7f68db92a9e5e809ee247ae2f32156bdc92465dacd6b47796b]
Would prune:     stats__backup__2020-11-11T03:12:22   Wed, 2020-11-11 03:12:30 [3033a687eccbed04614e7cc05e1465178a16766a8aa759eb44d915d15b684eba]
Keeping archive: stats__backup__2020-11-08T03:12:44   Sun, 2020-11-08 03:12:53 [a039ba0643ce47897213d8fa6ed5931b59baa457b470377ca837ecc8a275be5d]

[kelson42]

We need to have per default a backup for the:

• the 7 last days
• 5 last weeks
• 12 last months
• one per year before

[rgaudin]

Apparently fixed.

[kelson42]

@florentk I have to reopen this ticket as this is really not clear how this could work. See the documentation: https://docs.borgbase.com/faq/#why-can-i-still-prune-or-delete-archives-with-active-append-only-mode

[florentk]

@kelson42 I understand that in "Append only" mode, when you make a prune, it doesn't really do it until you do it with a full mode. This is strange that there is no error and that it no longer appears in the list. But it will be necessary to do it manually from time to time with another key to save space. Then it would be to see how to properly integrate then into our tool

[kelson42]

@florentk concretly, what should be exactly run to effectively free the sapce of the pruned backups?

[florentk]

@kelson42 borgmatic -c /root/.config/borgmatic/config.yaml --verbosity 1 --files with a correct SSH config.

[kelson42]

@florentk This is to prune only one reposity I guess?

[florentk]

@kelson42 yes

[kelson42]

@florentk We need a strategy to do that more or less automatically for all repos.

[kelson42]

@florentk @rgaudin It a bit sad to discover this problem so lately, in particular after me asking specifically to check the pruning. Anyway, now we need to fix it. It seems to me that there is no way for the backuper to effectively prune the old revisions with the append-only key. Changing the SSH keys to allow write operation, even temporarily, sounds cumbersome and dangerous. The best solution I can imagine is to have a third prune mode running with the master token. This mode should see the list of repositories in Bitwarden and for each do the necessary to get the prune done. What do you think?

[rgaudin]

Yes the behavior is very deceiving and that's disappointing. I want to dig a bit further to understand how and why borgmatic could remove pruned-but-not-actually-removed archives from the list that it reports with that append-only key.
This sounds like a mandatory first step so we can document this behavior properly in the README and not deceive our own users accidentally.

Next, I agree with you about the prune-command strategy using another set of key which would not be shared with the backuper account.

[florentk]

@kelson42 @rgaudin I agree with this, acutaly, we do not provide a key for full access when we create the BorgBase repository. Could we have just one for all repos and saved in the "master" BitWarden account? Or do you must have a different full keys for each repos ?