rachnog / rap_generation Goto Github PK

Hi!
In the Transformers_Generation_Training_EN.ipynb file, there is mask_tokens function that prepares an input batch of tokenized text to corresponding inputs and labels. The case where we replace selected words with [MASK] token seems clear to me, but later there is this piece of code where we select 10% of words to replace them with random ones. So I was wondering is there a typo here or we should use 0.5? Thanks!

indices_random = torch.bernoulli(torch.full(labels.shape, 0.5)).bool() & masked_indices & ~indices_replaced random_words = torch.randint(len(tokenizer), labels.shape, dtype=torch.long) inputs[indices_random] = random_words[indices_random]