Add configuration option for specifying custom location for .snapshots directory. about snapper

[montvid]

I just wanted to ask why snapper puts snapshots in the snaped subvolume? Now I want to delete the subvolume & restore from a snapshot but can't because the snapshots are in the subvolume?!?!

[GreatEmerald]

This would be useful, especially given the Btrfs upstream recommendation to create "flat" Btrfs hierarchies. That is, / containing nothing but subvolumes, like one for /home, one for the rest of /, etc., and making it mount the / subvolume by using btrfs subvolume set-default. That makes it easier to apply full subvolume restores, as nested subvolumes are fairly tricky in that situation. For more information see this: https://btrfs.wiki.kernel.org/index.php/SysadminGuide#Managing_snapshots

[keithwissing]

I would like this configuration option because I do not want my snapshots to be in a subdirectory of the volume I am snapshotting. The volumes I am snapshotting are frequently rsync'd to other places, and I don't want rsync to copy all the snapshots and I don't want to have to remember to exclude the .snapshots directory every time.

[pnahratow]

I intuitively planned the exact layout described in
https://btrfs.wiki.kernel.org/index.php/SysadminGuide#Managing_Snapshots

Even if there were no additional gain from this feature I'd want it just because I want to dictate my directory layouts myself.

I hope this feature gets implemented

For others landing on this page, you might want to look here
https://github.com/mmehnert/btrfs-snapshot-rotation
https://btrfs.wiki.kernel.org/index.php/Incremental_Backup

[sysfu]

A lot of people have already shown interest in it. So instead of - or along with - saying "+1 I want it", why don't you send in a pull request?

I agree with this sentiment and your arguments for it. I'm aware of the thankless hours that many FLOSS devs put into these projects so I'm far from demanding when I make feature requests.

If I had the coding skills to submit a pull, I would have. The next best thing I could offer is a bounty for the feature to become implemented. At the time this was originally opened, I don't think we had as many tools for raising software bounties as we do today. It was almost ten years ago!

At least the developer has been kind enough to keep the issue open instead of closing it and locking comments. That's a lot more than I can say for some other github project devs that shut down discussion at the drop of a hat.

That said, I'm in the process of creating a bountysource item for this feature request, maybe if we can get some money behind this it will be worth the devs time to implement. I'm thinking those interested in this feature should raise $300-500 for this, what do you guys think?

[dpantel]

Just want to +1 this.

I like the idea of having snapshots in a separate subvolume, not nested under the snapper target. This makes it easier to pool different btrfs snapshotting utils into a single destination. That way I can know with certainty that all of my available snapshots are in FS_ROOT/@snapshots, for example, no matter which util created them (snapper, apt-btrfs-snapshots, manual, etc.)

In response to @rbellamy's comment regarding snapper needing the nesting to know the target path... since snapper creates a meta file for each snapshot anyway (info.xml), the path and other pertinent data can be stored in there.

i.e., given a destination directory, the file structure could be:

SNAPPER_TARGET/unique_id/data. The info.xml can store snapshot type, number, path, etc. info.

[hirak99]

As it stands right now snapper does not work for me. Both (1) the file structure, and (2) rollback mechanism are not compatible with what I want. I can get around the rollback mechanism, but file structure is particularly vexing. I can get around it with hacks (similar to this one), but it is tedious for multiple subvolumes, interferes with deleting configs, and is something end users should not have to do.

Since this is not implemented yet despite the clear interest (and reasoning) seen in the comments here, I suppose this doesn't seem to be a priority?

My guess is that this was a design decision early on, as evidenced by the rollback mechanism which works by changing the default subvolume that only works for root - and it's now there's no easy way to take it back?

Since it doesn't work for me, I might end up implementing an alternative myself. I wanted to check once before doing that.

[rdslw]

I also agree that this feature would be great. Flat btrfs, no need to configure every utility (rsync/find/locate/etc), having one place for all snapshots, and simply using my way of directory/volumes structure.

[hirak99]

As stated earlier, this didn't unfortunately work for me. So I hacked together an alternative system for myself, which does work for me nicely.

It's simpler, and a lot more hacakble (for now).

To give some context, here's my example config that I use -

CONFIGS: list[Config] = [
    Config(source='/',
           dest_prefix='/.snapshots/@root-',
           on_pacman=True,
           keep_daily=3,
           keep_weekly=3,
           keep_monthly=2),
    Config(source='/home',
           dest_prefix='/.snapshots/@home-',
           keep_hourly=3,
           keep_daily=5),
]

If anyone else is interested I can polish it up and publish it.

[aschnell]

You are right that the location of snapshots cannot be configured.

From the mkinitcpio-btrfs documentation I read that the snapshots must follow the convention "__snapshot/[snapshot name]". Would "__snapshot/[snapshot num]/snapshot" even work?
Since snapper stores additional metadata with each snapshots the extra directory is very useful.

In general I prefer a name starting with a dot. The path .snapshots is also used by e.g. rsnapshot and AFAIR ext4. Maybe mkinitcpio-btrfs could get an option to use different paths.

[sysfu]

I was able to get mkintcpio-btrfs to utilize the snapper snapshots by modifying the line in /etc/defaults/btrfs_advanced like so: BTRFS_DIR_SNAPSHOT="/__active/.snapshots" ...and recreating initrd with "mkinitcpio -p linux".

It seemed a little kludgy to me however so I thought I'd ask here about perhaps gettings snapper to place the .snapshots dir in /__snapshot.

Also, I have the btrfs root mounted via /etc/fstab at /var/lib/btrfs, and I tried blasting away the /__active/.snapshots subvolume and creating a symlink instead /__active/.snapshots => /var/lib/btrfs/__rollback.

This seems to have failed with these errors in /var/log/snapper.log
2014-02-27 02:24:25 ERR libsnapper(1786) FileUtils.cc(SDir):84 - open failed path://.snapshots (Too many levels of symbolic links) 2014-02-27 02:24:25 ERR libsnapper(1786) Snapshot.cc(initialize):348 - reading failed

[rbellamy]

I spent a bit of time reading the code to see if there was a quick and dirty way to patch things to support this. It seems that it's a bit more challenging than "just pointing snapper to the /_snapshots subvolume."

First of all, the code just presumes that the snapshots folder is a child of the configured subvolume. So it can always use a relative path from that... so throughout the code you've got access points that make this assumption and "just work."

Secondly, how would you handle multiple, deeply embedded subvolumes, where the path isn't contiguous? For instance.

Current	Option 1	Option 2
/home/rbellamy/.snapshots	/_snapshots/home/rbellamy	/_snapshots/rbellamy
/var/lib/libvirt/.snapshots	/_snapshots/var/lib/libvirt	/_snapshots/libvirt
/var/lib/libvirt/images/.snapshots	/_snapshots/var/lib/libvirt/images	/_snapshots/images

• Option 1: it's virtually impossible to determine directory from subvolume just by looking at the path.
• Option 2: there's no guarantee that the directory will be unique, unlike a path-based scheme.

[rbellamy]

In a recent post by Robert White he mentions a very compelling reason for snapshots being located outside the normal filesystem. He does this not by attacking the idea of in-filesystem snapshots, instead he's explaining his preferred structure.

Basically the gist of it is that he keeps his snapshots outside the filesystem to avoid undesired snapshot directory traversal. Or as he puts it, "duplicate files from the snapshot are not interesting."

He's much more concise and precise than this poor representation, but as I see it, with an in-filesystem snapshot directory, how do you tell search tools (like find) that they're supposed to completely ignore the .snapshots directory? There are ways, like modifying /etc/updatedb.conf and such, but it's a form of blacklisting, which is inherently fragile, and requires maintenance - for instance, for grep, find and any other command line tool, you're faced with having to create aliases for each that explicitly excludes .snapshots.

[aschnell]

This would be useful, especially given the Btrfs upstream recommendation to create "flat" Btrfs hierarchies. That is, / containing nothing but subvolumes, like one for /home, one for the rest of /, etc.

If you want a "flat" btrfs hierarchies (/ only contains subvolumes) that is first of all a request for your system installer and not snapper.

[aschnell]

Basically the gist of it is that he keeps his snapshots outside the filesystem to avoid undesired snapshot directory traversal.

Hiding the snapshots from other tasks, e.g. updatedb, is not always possible. E.g. when snapper compares two snapshots they must be mounted and visible to snapper so other tasks can also see them.

Namespaces might help here but I haven't used them so far.

[aschnell]

First of all, the code just presumes that the snapshots folder is a child of the configured subvolume.

That is true and has a reason: A snapshot can only be made in the same btrfs filesystem. E.g. if /home/rbellamy is a different btrfs filesystem than / you cannot make a snapshot of /home/rbellamy in /.snapshots/rbellamy (unless you mount /home/rbellamy at /.snapshots but then the snapshot is also visible at /home/rbellamy).

[GreatEmerald]

If you want a "flat" btrfs hierarchies (/ only contains subvolumes) that is first of all a request for your system installer and not snapper.

It would be... Except in the case of Gentoo there is no installer, and openSUSE's YaST apparently already supports flat Btrfs hierarchies too, just not enabled by default (see openFATE #314606).

[nvsystems]

One of my backup methods is rsyncing everything to another drive or another machine. Having snapshots in / will make it sync this too. Now I need to make special exclusions for this.

But even when working around this, a simple find / -iname "somefilename" will then start searching all snapshots, which I really do not want it too. This absolutely should be configurable.

[mstarostik]

This is my one big concern with snapper. As mentioned above, having to explicitly exclude .snapshots from every tool that traverses a subvolume's directory tree is error prone and sometimes hard or even impossible. In case of the subvolume hosting e.g. a web root, forgetting to exclude .snapshots might even impact security as it might be used to circumvent other path based access restrictions (admin fail on multiple accounts there, but still...)
Apart from that and purely as a matter of taste, the current location feels like unnecessarily cluttering up "my" directory tree in an ugly way.

There should be a simple option to specify an absolute path to the snapshot hierarchy per snapper config. It is the admin's responsibility to ensure this path resides on the same btrfs as the subvolume to snapshot.

[MountainX]

This is still a needed feature, although of course there are some limitations on where the custom location can be. For example, snapshots have to be on the same BTRFS volume.

[DerYupp]

Same for me. This is the showstopper for using Snapper since years now. It's clear, that both have to be located on the same filesystem. But someone changing it knows about this.
PS: possibly best option would be to keep configuration in ./.snapshot but locate the snapshots themselves separately.

[zilexa]

This is why I prefer:
/mnt/disk/data (subvolume in subvolid5)
/mnt/disk/.snapshots (a folder for snapshots)
Please allow this.

[ajgringo619]

As an avid user of snapper, I didn't even realize that this was a snapper limitation; thought it was part of the BTRFS design. I completely agree that snapshot location should be configurable by the user, within BTRFS's parameters, of course.

[MountainX]

Since it doesn't work for me, I might end up implementing an alternative myself. I wanted to check once before doing that.

For me, btrbk is a great alternative. It allows specifying custom locations for .snapshots directory. The only downside I have noticed is that the pacman hook functionality is not as good as snapper.

[marczz]

@hirak99 I suppose that the alternative system just mentioned is yabsnap, it would be nice if it is tested on other distribution than archlinux. Also snapper is very well tested and provides operations that are not yet be implemented in yabsnap like diff, cleanup (very useful), mount.
Also snapper is used with other tools, I personally use baksnapper, if you maintain a compatibility with snapper, it could be not to difficult to use yabsnap.

Adding this feature to snapper, or forking it would be easier for the users.

When yabsnap will be more mature I will try it, if it become more closer to snapper features.

[hirak99]

That's fair. What's cleanup? Do you mean delete?

Edit: You probably mean auto deletion algorithm. Automatic cleanup is supported in yabsnap. It is determined based on the trigger and config. User and on-installation backup have a max count specified in the config. Scheduled backups ensure you have m hourly, n daily, p weekly etc.; it deletes any extra snaps to maintain the counts that you specify in config.

Automatic mount and support for other OS are good features that I've been thinking about too. Some of what you asked are easy to add, and I feel will be valuable additions. But maturity and external tooling can only come with time. If you want, you can specify features you would like in the yabsnap issues.

[marczz]

What's cleanup? Do you mean delete?

I mean snapper cleanup and the support for the cleanup algorithms "number" and "timeline".

But I learn from your reply that they already exist, but it is good to have a command to apply an algorithm on demand. Most of my snappers snapshots are not scheduled, but either triggered by an event or done manually.

[hirak99]

Gotcha. Edited my previous comment.

[hirak99]

Being able to apply a custom cleanup on demand makes sense to me.

[netzego]

@aschnell +1 this feature is really missing. could you let us know if you are open to this? since you answered here 2015 the last time.

@hirak99 please don't advertise your own software here. we are trying to get this feature into snapper and not into what-ever-script.sh. thank you very much.

[hirak99]

@netzego You're free to wait however long to use whatever, of course.

You don't speak for everyone in your "we", and after waiting for 9 years some people deserve to know that there are alternatives which were born literally on this thread, due to this specific request.

Anyway - I've moved on, good luck.

I'm unsubscribing from this issue. If anyone has comments on yabsnap then discuss it there.

[hirak99]

Here's a constructive suggestion to bring feature in Snapper.

A lot of people have already shown interest in it. So instead of - or along with - saying "+1 I want it", why don't you send in a pull request?

Can't speak for all developers, but most of them are human - typically working on projects like this out of passion.
If something matters to a small number of people (compared to the entire userbase), and it requires heavy and risky refactoring, it is natural that they will prioritize their limited time on other stuff.

So (assuming you are able to code), why not send in a pull request to Snapper to add it?

Most developers will welcome pull requests - it takes less time to review and guide than implement.
It also sets you up for future contributions in the same repo once your pull is accepted. And what better place to start than a feature that not only is desired by a lot of folks, but which you personally want as well!

[sysfu]

It looks like bountysource.com pulled their cryptocurrency funding option. If anyone can suggest a good place to get a software bounty going on this issue, please inform.