URL invalid for some datasets: about crossfit

[cherry979988]

Thank you for raising this! Bill (@yuchenlin) and I will try to find some workaround.

[cherry979988]

Hi Sewon,

I'm trying to reproduce this issue but my scripts are working as expected. Could you please provide some extra information for us? Thank you.

1. What are the error messages you're getting?
2. Could you double-check if your huggingface dataset has version 1.4.0 and could you please try the scripts again after clearing the cache?

Attaching my logs for reference.

[shmsw25]

Hi @cherry979988, thank you for your help. Yes, I double-checked that the HF datasets version is 1.4.0, and the error is keep occurring after clearing the cache. Error messages are saved here.

P.S. I think if you have downloaded the data once, the data is saved as a cache. Perhaps that is why you were not able to reproduce the error?

[cherry979988]

Hi @shmsw25

Thank you for providing the logs. I am able to reproduce the errors.

My guess is that the dataset owners updated their files, and the checksums in HF datasets is not yet updated, so we're getting this checksum error.

A temporary solution will be using ignore_verifications=True when loading datasets (e.g., dataset = load_dataset("kilt_tasks", "wow", ignore_verifications=True)). However, this will probably leads to differences in few-shot sampling. I'll discuss with Bill and see if there is a better solution...

[shmsw25]

Got it, thank you for taking a look at this!

[slyviacassell]

@cherry979988 Would you mind sharing your cache of the following for the unavailable network?

• jeopardy
• kilt_wow
• definite_pronoun_resolution
• wiki_auto