Deleting Conda/Python as a dependency entirely to dramatically decrease "latency to step" about llm.c

[karpathy]

FineWeb100B is 1010 files total, these are raw .bin shards of 100M tokens each

• Each is of size 191MB
• Zipped, each is 134MB

134MB * 1010 files = 135340MB ~= 135GB

[banyan-god]

Have you played with this streaming parameter ?
load_dataset("HuggingFaceFW/fineweb", name="CC-MAIN-2024-10", split='train', streaming=False,num_proc=28)
I was going to use it but i have already downloaded 500GB of files

[karpathy]

(I used streaming originally but then started getting some errors in the tokenization workers when a request randomly fails, so I took it out)

[banyan-god]

I do something like this not very efficient as i am encoding it on the fly but I am planning to implement a thread that tokenizes and buffers it so it is available readily . https://github.com/banyan-god/llama2.c/blob/master/finewebllama2.py