[Bug]: Inconsistent document count when loading custom dataset with ColumnCorpus about flair

[helpmefindaname]

Hi @anna-shopova
I think the changing results are due to the sampling of missing splits:
The default behavour randomly samples a dev dataset and a test dataset with 10% of the samples.
When I pass sample_missing_splits=False, I always get the same 5 examples as result.

corpus: Corpus = ColumnCorpus(data_folder, columns,
                              train_file='test.txt',
                              comment_symbol='# ',
                              document_separator_token="-DOCSTART-", 
                              sample_missing_splits=False)

gives

{'# id': '1163testb\n', '# topic': 'SOCCER\n'} Sentence[1]: "-DOCSTART-"
{'# id': '1164testb\n', '# topic': 'RUGBY\n'} Sentence[1]: "-DOCSTART-"
{'# id': '1165testb\n', '# topic': 'SOCCER\n'} Sentence[1]: "-DOCSTART-"
{'# id': '1166testb\n', '# topic': 'FREESTYLE\n'} Sentence[1]: "-DOCSTART-"
{'# id': '1167testb\n', '# topic': 'SOCCER\n'} Sentence[1]: "-DOCSTART-"
5
1163testb SOCCER
1164testb RUGBY
1165testb SOCCER
1166testb FREESTYLE
1167testb SOCCER
5

[anna-shopova]

With this code I have tested how to extract the metadata of the documents from the corpus. However, if I have train/dev/test files, this does not occur. From my side we could close the issue as I have the dataset splits during training and saw that if they are not defined, they are randomly generated.