Indexing and retrieval arguments for ColBERT quantization about colbert

[okhat]

@surajnair04 It's ready, beta version: https://github.com/stanford-futuredata/ColBERT/tree/binarization

[okhat]

Hi Suraj! Thanks for the question.

I'll get back to you after a few meetings, but I wanted to mention a limitation of the "beta" quantization branch. FAISS indexing shouldn't be done over the compressed vectors; the double compression leads to very poor FAISS quality in that case.

Either FAISS indexing has to be executed over the full index uncompressed (and then you can delete that big index!), or it will have to use the code we'll soon push that does FAISS indexing during the initial colbert.index step!

I haven't read your commands yet btw...

[surajnair04]

Hi @okhat, thanks for your prompt response!
I think I understand what's happening

• the initial colbert.index step in the beta quantization branch compress the vectors
• colbert.faiss_index step would then further compress the vector as part of the underlying indexing strategy

This double compression leads to a drop in the ranking quality. Is this correct?

So, the alternative is to wait for the beta branch to stabilize at which point colbert.index would contain the FAISS indexing?
Do you have a sense of when the release is planned?
I'm quite eager to try it out :)

The commands are pretty much the same as in the master branch (with the exception of --compress argument for colbert.index)

[okhat]

This double compression leads to a drop in the ranking quality. Is this correct?

Exactly. If you have the uncompressed index, you can create the FAISS object on top of that. Then this is resolved.

Otherwise, we're planning to update this in less than 2 weeks.

The commands are pretty much the same as in the master branch (with the exception of --compress argument for colbert.index)

Yup sounds right to me

[surajnair04]

Exactly. If you have the uncompressed index, you can create the FAISS object on top of that. Then this is resolved.

Otherwise, we're planning to update this in less than 2 weeks.

I believe using an uncompressed index would make sense for smaller document collections from a storage perspective.
However, I am interested in storing the compressed vectors in FAISS so it makes sense to wait.

Thanks for your help, @okhat!

[surajnair04]

@okhat, Thanks a lot! I will try it out and open new issues if I encounter any.
Closing this one for now.