CLN (temporarily) lost connection to bitcoind and crashed about lightning

[cdecker]

Well, your bitcoind is extremely slow, that's what changed ;-)

[cdecker]

Thanks for sharing, I was definitely not aware that bitcoind will happily stop responding when processing a block, that's annoying indeed.

[hMsats]

It happened again today

[cdecker]

I'm not sure what we should be doing for this issue. The "crash" is us
very loudly and explicitly exiting, after noticing that we can no
longer interact with the bitcoin network, i.e., we're flying blind,
potentially yelling into the void.

This is our way of hopefully attracting the operator's attention, as
there is nothing we can do on our side. We could increase the timeout,
however escalating to an operator and asking them to fix the lack of
connectivity to the network is the right thing to do here.

This fix here is for the operator to address the connectivity problem
between CLN and the bitcoind that is acting as a backend, there is
nothing for CLN to do here.

[hMsats]

Well, I don't recall ever having this issue before upgrading to v24.05 and nothing else changed on my system. Don't know if that could be a hint ...

[hMsats]

Fair :-) next time it happens, I will try to have a look in debug.log what bitcoind is doing around the time of the crash.

[hMsats]

@cdecker just to close this issue from my side: I had a look at another crash and came to the conclusion that when blocks come in too fast, my bitcoind is in trouble. In this case 3 blocks came in within 3 minutes so 10 times as fast as on average (but I know that's still normal):

Highed  Timestamp	        Txs     Size(KB)	Weight (KWU)
849683  2024-06-27 14:46:02     390     1861.678        3993.01
849682  2024-06-27 14:44:11     596     1842.971        3993.242
849681  2024-06-27 14:43:30     1889    1595.995        3993.163

I have increased my cpu-limit (a lot more is running on my server). To be honest, I still don't get why one wouldn't close CLN with the error message something like "bitcoind didn't respond, closing CLN" instead of what looks like a CLN crash.

I get the impression bitcoind first processes blocks before responding to rpc calls (in between blocks) but I'm not sure.