lightningd can crash if time moves backwards about lightning

[cdecker]

Maybe we can push this issue upstream to make strict monotonicity optional by adding an additional flag to struct timers. My fear is that this could possibly result in multiple executions of the same timers?

[rustyrussell]

gwillen [email protected] writes:

In the ccan timer module, there's an assertion that can crash if time moves backwards. I know there are different perspectives on this, but it seems like lightningd can probably recover from slight non-monotonicity of time, so it's worth thinking about whether crashing is the best outcome in this case.

Ouch, thanks!

So you know what happened? NTP jump or manual setting?

lightningd: ccan/ccan/timer/timer.c:307: timers_expire: Assertion `now >= timers->base' failed.
lightningd(24325): FATAL SIGNAL 6 RECEIVED
Fatal signal 6. Log dumped in crash.log
Aborted

Yuck. Saying that time shouldn't go backwards is fairly fundamental,
and a useful sanity check. Lots of things assume time makes progress.
And a library can't report soft errors.

OTOH, real life always wins over my personal preferences :)

One option is to switch to timemono, which really can't go backwards.
But timemono is completely detached from any wallclock time, so it's
only useful for relative timers.

Which implies an API change to always use relative time (this actually
matches with all non-test usage I can find).

I'll look harder.

Thanks!
Rusty.

[gwillen]

This was a purely artificial example -- I accidentally jumped the time when debugging some time-handling code in an unrelated project. So it's not really "real life".

Although I'm curious what you mean by "a library can't report soft errors", since I am generally a fan of what seems like the opposite philosophy, "a library should generally never crash the binary it's executing within." But I think it's fine if we decide that lightningd crashes are acceptable in weird conditions, we just have to be aware (and perhaps document) that this is possible since it's possibly surprising.

[cdecker]

It happens from time to time, I noticed it 2-3 times when coming back from suspend. It's likely that the clock drifts substantially and then NTP kicks in between two calls to timer_expire. So I wouldn't exactly call this a rare event :-)