Restoring state-sync snapshot takes a long time about iavl

[tac0turtle]

there are a couple issue in iavl to assist with this. Here is one pr that aims to increase the speed. #664.

This issue could be to how large your current state is? ill transfer this issue to iavl so it can be focused on where the actual problem may lie. Lots of the compacting is being cleaned up as we speak.

[PolkachuIntern]

Our state-sync scripts run every Sunday, so this issue is rather timely. Nois testnet state-sync took a bit over an hour today. In contrast, Stargaze mainnet state-sync took about 10 minutes. I am not knowledgable enough to articulate the reasons behind it. As Simon mentioned, Evmos and Sei state-sync is also slow and at times unreliable.

[kaisbaccour]

Would it make sense to make the inventory of the number of application.db KV entries for all these networks and some other very busy networks to say if the rule stands or no?

[jhernandezb]

Adding profiles:
cpu.pb.gz
mem.pb.gz

Binary can be built using:
https://github.com/noislabs/full-node/tree/main

One thing to mention is that stargaze still has fast node disabled by default (will be changed to enabled in our next upgrade).

Seems that fast node is contributing to more cpu cycles and garbage collection

[tac0turtle]

with iavlv1 this should be significantly faster. Would you want to test with this version?