duplicated substring removal in suffix_array.py about text-dedup

[ChenghaoMou]

This is actually what been used in the original paper, you can find this on their readme file https://github.com/google-research/deduplicate-text-datasets:

In our paper we suggest just taking all of these duplicate sequences that have been identified and completely striking them from the dataset. This somewhat breaks the flow of text, for example if previously had an example "Alice wanted to go to the store" and we deduplicated at the level of 10 characters, we might completely strike " to go to the " and be left with "Alice wantedstore". In practice we have found this doesn't break the language model because we remove relatively little text, and so these breaks don't cause harm.

However, it might not be the case for your own dataset, or you have different use case of those duplicated strings, in which case you will have to modify the code.