Build index in unormalized inner product with lower performance? about hnswlib

[fcjy]

@searchivarius Of course! I had sent it.

[searchivarius]

@fcjy do you have the same issue with NMSLIB implementation?
PS: HNSW doesn't always work. I have an example where it doesn't work with the cosine similarity and with the inner product.

[fcjy]

@fcjy do you have the same issue with NMSLIB implementation?
PS: HNSW doesn't always work. I have an example where it doesn't work with the cosine similarity and with the inner product.

Thank you for your reply!
I had tried NMSLIB, it is better than hnswlib. it's recall rate is 72%, cost 7ms for once search, with the same parameters.
Maybe faiss::IndexIVFFlat is more suitable for my situation.

[yurymalkov]

@fcjy Can you please share a dataset with the problem?
Also, did you tried post=2 in nmslib's HNSW and setting EXTEND_USE_EXTENDED_NEIGHB_AT_CONSTR to 1 in nmslib's code https://github.com/nmslib/nmslib/blob/master/similarity_search/src/method/hnsw.cc#L52?

I have some ideas why this might happen. One way to solve the problems is using https://github.com/stanis-morozov/ip-nsw code/ (C++), an extension of HNSW done specifically for inner product space.

[fcjy]

@yurymalkov Of course! I had sent an email to you(yurymalkov_mail_ru), please check it.
And I will try the two methods you mentioned today.
Thanks!

[yurymalkov]

@fcjy Thank you! I'll test it in a few days.

[fcjy]

I do a transform, let the MIPS problem change to NNS problem, such as chapter 2.1 in this paper, and then run libhnsw, that get a very nice result!

[searchivarius]

@fcjy could you share your data with me to? (leo at boytsov.info). A one to 10 million sample will be enough. Many thanks!

[wuwenjunwwj]

hi, I have encountered the same problem。I have a dataset with 128 dimension and I build a index(M = 200, ef_construction = 1000) in inner product space，search topK = 5000, it's call rate is 70%。but when I build index with normalized vector with the same parametor，it‘s rate is about 98% 。Do you know the reason now and have any suggestions ？

[searchivarius]

@wuwenjunwwj the inner product search is likely more challenging. It's not the same search problem. Plus, for topK=5000 it's difficult to get accurate results with k-NN search overall.

[searchivarius]

@fcjy do you still have that old data set? I tried to download it recently, but clearly it's not there any more :-)