Alert when blocks are mined too quickly about infra

[geigerzaehler]

How about not alerting on a high block import rate for the stateless nodes (i.e. the miner and the rpc server)? This would solve the problem and I cannot think of a case where there is something wrong and the block rate is high on stateless nodes but not on the stateful validators.

[geigerzaehler]

We’re seeing false positives for this when miners are restarted and sync very quickly. We’re not using substrate_sub_libp2p_is_major_syncing correctly. If this metric goes to 0 we still have a high import rate for some time. Maybe we need to consider using the substrate_block_height{status="sync_target"}

[CodeSandwich]

This is going to be tricky. I agree that substrate_sub_libp2p_is_major_syncing isn't the right gauge to look at, it's 1 only when there's really a lot to import.

The sync_target is going to be rather tricky. AFAIK it's different from best whenever there's anything to sync, so there's no clear way to tell being bootstrapped from bigger regular syncing.

Maybe we should look at the block timestamps? We could attach the Prometheus timestamp to every data point. If in the window any of the block timestamps is older than the Prometheus timestamp by more than let's say 15 minutes, the whole window is considered to be witnessing some serious syncing and not eligible for raising an alert.

The possible downside is that if a block takes more than 15 minutes to mine, it will have the block timestamp already 15 minutes old when broadcasting. It will effectively limit low import rate alert for 11-minute window to being raised for no longer than 4 minutes.

[CodeSandwich]

That's a very reasonable life simplification. The only case when we may keep getting spam will be during restarting of the validator nodes, but that's rare and we'll be expecting that.

[geigerzaehler]

The only case when we may keep getting spam will be during restarting of the validator nodes

Since the validator nodes are stateful they will not sync extensively. The will start from the chain they shut down with.