Error when executing create_thread_prompts.py about llama2lang

[ErikTromp]

Your output folder from translation looks fine - we go from source language (in oasst1) to target (eu in your case) so that's what you see. Do the separate files contain actual (JSON) content? Because it seems that your create_thread_prompts.py is simply not picking up any data, see: Generating train split: 0 examples [00:00, ? examples/s]

This could be due to the loading from disk in Arrow format - you can try uploading your translated oasst1 dataset to Huggingface first and then using that to create the threads.

[ErikTromp]

No the code was wrong but I fixed it now. If you pull main and make sure to update how you call the scripts (check the readme) you should be good to go from where you left off :)

[elBlacksmith]

Your output folder from translation looks fine - we go from source language (in oasst1) to target (eu in your case) so that's what you see. Do the separate files contain actual (JSON) content? Because it seems that your create_thread_prompts.py is simply not picking up any data, see: Generating train split: 0 examples [00:00, ? examples/s]

This could be due to the loading from disk in Arrow format - you can try uploading your translated oasst1 dataset to Huggingface first and then using that to create the threads.

Hi thanks for the quick reply. :)

The json files are populated, I don´t see any of 0kb and for what I see they contain phrases in basque. I guess it´s a good signal.

I used the combine_checkpoints.py to upload the files to my public HugginFace repository: https://huggingface.co/datasets/Jaimefebe/Eus02 but I am getting the same error.

If the problem is on my lack of knowledge, please don´t hesitate to tell me.

I just opened this ticket because ChatGPT 3.5 wasn´t really helpful with that error and I thought that maybe it was a bug that could interest you, but if it´s a some newbie error maybe ChatGPT 4 can help me.

[elBlacksmith]

No the code was wrong but I fixed it now. If you pull main and make sure to update how you call the scripts (check the readme) you should be good to go from where you left off :)

Cool! I'll try it as soon as possible.

Thanks!