Question on how are we storing in FAISS? about colbert

[okhat]

Hey @deepankar27 !

In our faiss representations, we store a flat matrix indeed. So we'll store 15 embeddings. If you had two documents, one with 15 tokens and the other with 20 tokens, we'll feed faiss a matrix with shape (15+20, 128).

Hope this helps! Let me know if you have any other questions.

[deepankar27]

Hello, @okhat thanks a lot for the response. I understood your explanation. I believe that's why we are wrote this code in :

ColBERT/colbert/ranking/faiss_index.py

Line 76 in 49d7fb6

Q_faiss = Q.view(num_queries * embeddings_per_query, dim).cpu().contiguous()

This lead to only few last questions:

1. If my understanding is correct then are we storing these embeddings against emb2pid?
   self.emb2pid[offset_doclens: offset_doclens + dlength] = pid_offset + pid
   

   ColBERT/colbert/ranking/faiss_index.py

   Line 57 in 49d7fb6

   self.emb2pid[offset_doclens: offset_doclens + dlength] = pid_offset + pid

2. Are we performing two level of quantization for Query embeddings as well?
3. How about storing the non stopwords tokens only, that way we would be storing only the tokens which are representing the context and size/ shape could be smaller.

[okhat]

Dropping stopwords is fine in practice, indeed!

emb2pid is just a mapping from each embedding to its document of origin.

Queries are not compressed with FAISS.

[deepankar27]

@okhat Thanks and appreciate your responses 👏