Clarification on PLAID retrieval about colbert

[okhat]

Hey Thibault! Hope you’re well.

I can check but basically we return k, but we compute exact scores for a larger number than k

[okhat]

This sounds right, but @santhnm2 might be able to confirm too

[thibault-formal]

Hey Omar! I hope you are well too!
I see -- so basically

• in the k=10 scenario, you compute 256/4=64 exact scores
• in the k=100 scenario, you compute 1024/4=256 exact scores
  etc.

Just checking that I got things correctly, as I have been working on related stuff :)
Thanks

[santhnm2]

Yes this is correct, this function is where we choose the hyperparameters according to k: https://github.com/stanford-futuredata/ColBERT/blob/main/colbert/searcher.py#L88
And here is where the number of exact scores is computed:

ColBERT/colbert/search/index_storage.py

Line 152 in fc3ce55

pids = pids[torch.topk(approx_scores, k=(config.ndocs // 4)).indices]

[thibault-formal]

Perfect, thank you both for the quick answer!

[thibault-formal]

Hi again,

I have another (unrelated) question regarding PLAID: did you evaluate the performance on the BEIR benchmark? Could there be a performance drop (OOD) due to the approximation?

EDIT: I saw the Lotte results (apparently no drop) but I wonder if it's also true on BEIR

Thanks