Impact of repo-level code concat about deepseek-coder

[guoday]

In our experiment, repo-level code would slightly hurt performance of benchmark. However, it will enhance long context modeling and repo-level completion.

[feiyuvl]

@guoday Thanx for your reply. Another question, how many epochs do you train on the 2T tokens dataset?

[guoday]

4-5 epochs

[feiyuvl]

So the total training tokens is about 10T? Amazing

[guoday]

No. The total data is about 400-500B. We train the model for 4-5 epochs, resulting 2T.

[feiyuvl]

This amount is equal to the starcoder dataset. Do you possibly resuse the stack dataset, and apply a more advanced data filter alogrithm, or do you download more code data than stack, and implment more stricted data cleaning algorithms?

[feiyuvl]

Deepseek use the repo-level files, so it can't resuse the stack dataset. Which is the key contrbution to this amazing improvment over starcoder, code quaility filtering, second pretraining or something else?

[guoday]

The primary contributions come from many aspects, including data collection, deduplication, filtering, the total number of tokens for pre-training, and the methods of pre-training. The optimization of each process is essential for enhancing the model. We will analyze this further in our later technical reports.

[feiyuvl]

Thank you, looking forward to your technical reports

[wentinghome]

Hey, thanks for the great work! can you share the script of dependency parsing and repo level dedup?