Mask值 about guwenbert

[Ethan-yt]

from transformers import BertTokenizer
tokenizer = BertTokenizer.from_pretrained("ethanyt/guwenbert-base")
tokenizer('言下法未[PAD]')
>>>  {'input_ids': [0, 55, 34, 76, 89, 1, 2], 'token_type_ids': [0, 0, 0, 0, 0, 0, 0], 'attention_mask': [1, 1, 1, 1, 1, 1, 1]}

[Ethan-yt]

In [8]: tokenizer(['言下法未', '言下'], padding=True)
Out[8]: {'input_ids': [[0, 55, 34, 76, 89, 2], [0, 55, 34, 2, 1, 1]], 'token_type_ids': [[0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0]], 'attention_mask': [[1, 1, 1, 1, 1, 1], [1, 1, 1, 1, 0, 0]]}

[Ethan-yt]

In [16]: tokenizer.get_special_tokens_mask([0, 55, 34, 76, 89, 1, 1, 1, 1], already_has_special_tokens=True)
Out[16]: [1, 0, 0, 0, 0, 1, 1, 1, 1]

[Ethan-yt]

不知道我的这几段代码有没有回答你的问题

[YangHan-Morningstar]

好的，我明白了，谢谢