Is this method implement only in the data parallel ? is there any pipeline parallel just like the model parallel ? about datatrove

[hynky1999]

Hey, I am heaving a bit hard time understanding the issue, could you elaborate more ?
What method do you have in mind ?

[WenhaoZhang-Git]

Hey, I am heaving a bit hard time understanding the issue, could you elaborate more ? What method do you have in mind ?

Thank u answer, i mean, the execute method, the PipelineExecutor class method.

[guipenedo]

Hi, to clarify, datatrove is a data processing library, and not a distributed training framework. If you want a distributed training framework I recommend you look into nanotron

[WenhaoZhang-Git]

Hi, to clarify, datatrove is a data processing library, and not a distributed training framework. If you want a distributed training framework I recommend you look into nanotron

thank u reply

[WenhaoZhang-Git]

Hi, to clarify, datatrove is a data processing library, and not a distributed training framework. If you want a distributed training framework I recommend you look into nanotron

Thank u reply again, is there any data processing library or method for processing structured data such as the tabular data in the book ?

[guipenedo]

What book are you referring to?

[WenhaoZhang-Git]

What book are you referring to?

I have some the pdf type of book that include table data, but i got 'textified_table' derived from the 'table_in_book', when i convert the book.pdf to book.txt, i just wanna the pure text of the book, the no structure 'textified_table' that is some kinda of noisy data. maybe just convert pdf to md. how to deal with this kinda problem in current public book dataset for pretraining of llm , do you know ? thank u a lot, i very appreciate for your reply

[guipenedo]

We don't currently have any fix for this, our text extraction is intended for when you have an html document

[WenhaoZhang-Git]

We don't currently have any fix for this, our text extraction is intended for when you have an html document

Appreciate u candor