question about data preprocessing about multidoc2dial

[songfeng]

Thank you for the question! If you download the data manually and save it locally, then you can specify DATA_DIR here. You should be fine. There is also sample code for loading the dataset in here.

We updated the test split of MultiDoc2Dial dataset recently, which caused some issue using Huggingface load_dataset. @sivasankalpp Could you please follow up on this? Thank you very much!

[sivasankalpp]

Hi @gouqi666, happy to help! Can you share the command you tried to download the dataset?

[songfeng]

Hi @gouqi666, happy to help! Can you share the command you tried to download the dataset?

Hi @sivasankalpp , I meant that the huggingface multdoc2dial. I think there's need to run their cmd to update dataset_infos.json and then test if load_dataset works with the latest multidoc2dial download. This should resolve the data loading issue reported here.

[gouqi666]

And in model_convert.py --> retriever = RagRetriever(model.config, question_encoder_tokenizer, generator_tokenizer) shows ConnectionError: Couldn't reach https://raw.githubusercontent.com/huggingface/datasets/1.16.1/datasets/wiki_dpr/wiki_dpr.py.

[songfeng]

The link works for me.