Bloom Filter Error Rate about plutus

[pimoussefrnl]

@MisterTeo You can't, the database provided is already serialized with 0.1% (if i'm not mistaken).
Mine was modified before the new pull requests so I have serialized myself the database from January with an error rate of 0.0000001%. (I have tested the bloomfilter before it was already integrated so my code is the original one from Fraccaman) I think @Isaacdelly need to push a new database only.

[Finn-Sionnach]

It's not all of them, (but it is more than I would have expected) figure something with the bloom filter and the .1 error rate?

[budiony]

I suppose this is related to the bloom filter, I have the same issue.

[MisterTeo]

Same here. Hundred. All 0 balance.

[Simkas45]

all address saved to txt file from 0 balance

[pimoussefrnl]

That is due to the bloom filter, if you look at the README in the database, the filter has been created with a 0.1% error rate which is way too much. The error rate of 0.0001% produces already about 4/5 wallets per 4 hours (empty wallet). The bloom filter should be done with 0.0000001% error rate at least (this is the one I'm using, Thanks to Fraccaman). If I remember correctly, 0.0001% the Filter was about 75 Mb, with a correction of 0.0000001%the filter was about 120 Mb.

bloom = BloomFilter(max_elements=10000, error_rate=0.1)
To:
bloom = BloomFilter(max_elements=25000000, error_rate=0.0000001)

[MisterTeo]

you can help him and upload your database... :)

[pimoussefrnl]

I can't, my file is not a .pickle and it was not created the same way he did. I don't have the file right now to compare. I will look into it when I have it.

[Isaacdelly]

It was because the bloom filter had a 0.1 error rate. If I tried to make the error rate smaller, then the serialized file would get bigger and then couldn't get uploaded to GitHub because it exceeded 50MB. And bloom filters can't be partitioned into smaller files like we do with a set, so 0.1 was the smallest value I could upload.

I went back to using a set in this update and i'll try to see if there is a way to upload a bloom filter with a smaller error rate that's under 50MB.