Weekly Snapshots? about sanoid

[jonathanvaughn]

I for one would still love to see a weekly option, since to get weeklies right now I have to keep a month's worth of dailies ...

[jimsalterjrs]

Hm. Well, it's an interesting thought - but now you're asking Sanoid to manage crontabs alongside normal user handling of crontabs, with a lot of potential for misunderstanding and screwups on both sides.

TBH @STRML I've been thinking about your use case, and the more I think about it the more I think it really needs a fairly major engine redesign, going from a periodically-activated-by-cron model to a daemon model. The way Sanoid runs now, it caches media-heavy operations like full zfs list -rt snap to avoid thrashing disks, and otherwise just sort of eats the overhead of being constantly started since it's only a few milliseconds every minute at worst (aside from the periodic destruction and rebuild of the cache). It's a nice design because it's simple, and it doesn't need a watchdog to look out for it crashing since it's just started anew anyway.

However, some users with more heavily loaded systems are seeing difficult-to-debug issues with too many snapshots being taken on the same interval, and in particular your desired use case - much, much more frequent intervals - would itself tend to put a lot more load on the system than it really should.

I think what we're looking at here is a Sanoid 2.0 with a --daemonize mode, which instead of keeping a simple cache that gets rebuilt every time the filesystem is changed, keeps an actual running tally of all system snapshots (that it has created), which is only rebuilt if or when the daemon itself is reloaded. That also relieves you in particular of the need to run some kind of microcron, since Sanoid itself can handle scheduling with great precision inside the daemon (I've been working with micro-timing recently over in my network-testing repo, so it's fresh on my mind).

It's probably also possible to make a syntax work well for user-defined intervals with desired "perfect" times to take snapshots that can be overridden if snapshots are overdue, too; it's just a little more brain-break-y than I was originally willing to deal with when I wrote Sanoid in the first place.

How do you feel about "partially commissioning" a major rewrite as described? I think the only other option is to implement your request of super-frequent intervals as a completely separate daemon. That would certainly be faster and easier than rewriting Sanoid itself, but TBH I'd like to take the extra time and make the whole platform benefit from it (and that would also make it easier for you to take advantage of ongoing updates to the platform in other respects).

We can and should talk about the nitty-gritty of the commissioning part off-list, but I figured since you brought it up here, it wouldn't hurt to discuss the technical aspects of it in public. =)

[redmop]

We are getting OT, so I'll email you.

[BioSehnsucht]

I see a PR was made some time ago to implement this, any chance of it being merged in the near future?

[jimsalterjrs]

Yep, I'll put this on the to-do list.

[cooper75]

Great, Thanks!

[cooper75]

I have been thinking about this a little more. The way templates are handled now, options are replaced. What if they were stacked? That way i could define multiple daily, weekly, monthly snaps. The templates might need to be redefined/restricted to handling only one type of snapshot, and need a name/label/prefix. Something like this?:

[template_dailysnap11a]
; take a snapshot @ 11am
name=11am
type=daily # hourly,daily,weekly,monthly, yearly
daily_hour=11
daily_min=01
autoprune=yes
count=2

This way you could define multiple snapshots for each class (hourly,daily. .etc) and then attach them to backup_templates or the host definition. This would give people the ability to do things like
2 11am snapshots
2 2pm snapshots
26 Friday snapshots
1st and 15 of the month.
7 daily
and on and on. . .

I realize this is a pretty big enhancement/feature request, but i think it would be valuable and give users the ability to do/define any snapshot scheme they want.

[jimsalterjrs]

Too complex and, frankly, strikes me as completely unnecessary, sorry. #wontfix on that idea.

[STRML]

The way zfs-auto-snapshot does it is quite nice, in fact. It sets cron jobs in /etc/cron.{d,hourly,daily,weekly,monthly} and then you set a keep=x param on that. It's not as flexible of a syntax, especially for multiple jobs, but we could do something really nice with a custom syntax.

This syntax could remove all hard-coded periods and replace it with a single structure. For example, I imagine reimplementing the existing periods like so:

<hourly>
schedule = 0 * * * *

<daily>
schedule = 59 23 * * *

<monthly>
schedule = 0 0 1 * *

<yearly>
schedule = 0 0 1 1 *

Then specifying them as:

[template_weekly]

keep_hourly = 48
keep_daily = 90
keep_monthly = 6
keep_yearly = 0

The purpose of the keep_ prefix is such that period names can be arbitrary without fear of collision.

Then, it could be possible to define the following useful periods without any modifications to sanoid:

<minutely>
schedule = * * * * *

<weekly>
schedule = * * * * 1

Or even something crazier in combination with https://github.com/homer6/frequent-cron:

<frequent>
schedule = */10 * * * *

[STRML]

How do you feel about "partially commissioning" a major rewrite as described

I hereby.... 🤣

Yes, I'd support this (and emailed you yesterday to that end), although I am a bit wary of what complexity may be introduced by something as ambitious as a full rewrite. The syntax I described above didn't suggest using cron to actually run the jobs (although it could!) but instead just as a flexible, terse, familiar syntax for the mechanism that already exists. I still see sanoid being invoke via a * * * * * cron.d job.

How risky - or lengthy - a rewrite would be is a question I think only you can answer. If it's possible to keep it simple, IMO that's better.

[redmop]

@STRML, zfSnap pretty much does the separate cron thing you suggested. It can also work on @reboot and pre/post apt/yum.

[STRML]

We're evaluating znapzend as well, which has done very well even at very high frequency (e.g. 5s).

[redmop]

I'm using znapzend right now, and can recommend it. Editing the config is a little clunky, but it works well.

[STRML]

Yeah I agree it's clunky. Out of curiosity, have you done/seen any scripts that wrap admin actions (such as apt, upstart, or a a deployment) around snapshots & can clean them up after some time?

[jimsalterjrs]

@redmop have you switched from sanoid to znapzend? If so would you mind sharing what the deciding factors were? Off list is fine if you prefer.

[emacsomancer]

I moved from znapzend to sanoid, but I miss weekly snapshots.

[phreaker0]

@jimsalterjrs do you already started with adding weekly snapshots or are you fine if i prepare a PR?

[jimsalterjrs]

PR please! :)

…

On November 8, 2017 11:39:24 AM phreaker0 ***@***.***> wrote: @jimsalterjrs do you already started with adding weekly snapshots or are you fine if i prepare a PR? -- You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub: #8 (comment)

[kryptus36]

Another me too - please merge the weekly snap option.

[ThatGuy435]

I'm not savvy with git enough to fork and merge, so count this as another request for the PR to be merged into the main codebase.