Choosing insertion batch size for MinHashLSH \w Redis backend about datasketch

[ae-foster]

Hey @alexeyev . In general, you should make the batch sizes as large as possible. In fact, using insertion_session automatically does some batching, see https://github.com/ekzhu/datasketch/blob/master/datasketch/storage.py#L226 . You could try running without your own batching and see whether the in-built batching is sufficient for your purposes. This literal value of 50000 needs to be changed in future to something user-specifiable.

Secondly, on the question of performance, the additional time you see almost certainly results from additional network I/O which is inevitable when using Redis. Large batch sizes mitigate this as much as possible. It can be instructive to profile your insertion code to confirm this. I once looked into creating Redis dump files without running a live Redis server (see https://github.com/ae-foster/rdbgenerate) but this didn't seem promising.

[alexeyev]

Also -- am I doing it wrong or does adding redis backend slow down indexing process dramatically?

[ekzhu]

Yes, in my experience redis insertion can be slow due to the serialization and deserialization, even when the database is running on the same machine.

A possible solution to this problem is to implement a way to initialize a Redis-backed LSH from a Dictionary-backed LSH. So you can insert into a Dictionary-backed LSH first (fast), and when you are done, you can (asynchronously) create the Redis-backed LSH.

This way you can start using the Dictionary-backed LSH right-away.