Issue with indexing BGE-M3 (large dimensionality vectors) about ragatouille

[NohTow]

I did some additional experiments which confirmed that the issue is in the compression process.
I computed the similarity between original vectors and decompression of the compressed version (np.diag(embs[:100] @ self.decompress(ResidualCodec.Embeddings(codes, residuals))[:100].T.cpu())).
This value is, for the original ColBERTv2 model 0.9995, whereas, for the BGE model, it is 0.010826.

In an attempt to understand why, I reversed every modification done to make BGE compatible with RAGatouille and identified one that strongly affect the score: the final normalization (after the linear layer) is L2 in the original ColBERT and L1 in BGE. Using L2 allows to increase the reconstruction similarity to 0.394.

More surprising is that, by reducing the number of bits (setting nbits = 2 manually), the score did not decrease but increased to 0.9473. This means that the model is actually usable in this state, but it would be better to understand why:

• Using L1 destroys the compression results. Although it is not a big deal, BGE has been trained with L1 and using L2 might hurt the results.
• How 2 bits quantization can be better than 8 bits. Again, it is usable like so, but given that BGE is using a very large embedding size, it would be cool to use more bits to encode it