Threaded add_items issue about hnswlib

[yurymalkov]

I see. Thanks!
It seems there are only two options to solve this:

1. use single-threaded construction.
2. setting high ef/efConstruction values, so the search will be almost exact.

There is a potential fix that can stabilize the randomness to some extent - setting the element levels before the actual insertion (it would require updating bindings), but it will not solve the problem completely.
I think that hnsw in faiss (e.g. https://github.com/facebookresearch/faiss/blob/master/benchs/bench_hnsw.py) works that way. You can try it (although it is generally slower than hnswlib at fixed accuracy).

[yurymalkov]

Hi @sumsuddin,
Can you please provide a demo script to understand what is going on?

[sumsuddin]

I can't share the private data that I was working on. But here is a randomly generated numpy array that I saved in a file. I attached the saved file here so that you can investigate.

# Generating sample data
#data = np.float32(np.random.random((num_elements, dim)))
#np.savetxt('data.txt', data)
data = np.loadtxt('data.txt')

For this specific random number combination (attached file) I get following two different recall accuracy randomly in different run.

Recall for two batches: 0.99990000000000001 (this happens rarely)
Recall for two batches: 1.0 (I mostly get this one)

Increasing the item size makes the issue more obvious in my experiments.
num_elements = 100000

I guess you can find easier ways to regenerate the issue.
Thanks for your time.

Python version : Python 2.7.6
OS: Ubuntu 14.04.5 LTS

data.txt