Performance on BM25 retrieval baseline about multidoc2dial

[songfeng]

Thank you for the questions!

• For the retrieval results reported in the papers, they are all passage retrieval results (i.e., "Pid_Prec@n") not document retrieval (i.e., "Doc_Prec@n" in the output). The passage results you got is quite comparable to the last three columns in Table 4.
• run_eval_rag_re.sh only provides the retrieval results. For text generation evaluation scores (F1, EM, BL in Table 4) , please refer to run_eval_rag_e2e.sh
• D^token-*-ft means that we use finetuned-DPR encoders for the document index (run_kb_index.sh) and the biencoder for Retriever Module in RAG (run_converter.sh).

[sutakori]

Thanks for your reply!
So if I am not mistaken, Table 5 is from run_eval_rag_re.sh, and Table 4&6 are from run_eval_rag_e2e.sh, with task set as grounding&generation, is that right?
I mistakenly thought D^token-ft as DPR and D^token-rr-cls-ft as RAG, and so they are all RAG? And I am still confusing of the difference between D^token-ft and the *-rr-*.

[songfeng]

• Table 4 and 6 contain retrieval evaluation (run_eval_rag_re.sh) and text generation evaluation (run_eval_rag_e2e.sh) results by RAG models. Table 5 is DPR retrieval results, not RAG.
• -rr- corresponds to reranking the retrieved passages by RAG retriever based on the retrieval results by only the current turn, where the embedding of the current turn is based on [CLS] (rr-cls) or pooled (rr-pl). Please see Paper Section 3.2 and code as a reference.

[sutakori]

Ok, I've got it, thank you for your prompt reply!