args.milliseconds is always empty in ranking.py about colbert

[okhat]

Thanks---will fix in a little bit!

In any case, this is not the file you want to be timing. This script uses on-the-fly ranking which is (by nature) very slow.

Instead, you're probably more interested in the timing code of retrieval.py, which is much faster.

[aviclu]

@okhat Thanks for your quick reply!
In this case, how would you suggest running the test.py script in order to properly reproduce the validation results in your paper, including timing and latency results?

[okhat]

You can use colbert.index and colbert.retrieve/colbert.rerank if you're interested in low latency.

I just remove the offending line from colbert.test too if you don't want indexing or fast retrieval.

[aviclu]

Thanks! So, what is the most effective way to reproduce the results from Tables 1 and 2 from your paper?

[okhat]

For re-ranking, you need to use colbert.index and then colbert.rerank.

For end-to-end retrieval, you'll need colbert.index, colbert.index_faiss, and then colbert.retrieve.

[okhat]

Looks resolved?

Closing but feel free to re-open if I can help with other issues!