gpt_tokenize: unknown token '? about embedai

[mark420524]

This looks like a common issue with Python 3.8 . You can upgrade to Python 3.10 and it should work

I upgrade python to 3.11 , it works. thanks

[RamiSJ12]

For the important_tokens which contain several actual words (like frankie_and_bennys), you can replace underscore with the space and feed them normally, Or add them as a special token. I prefer the first option because this way you can use pre-trained embedding for their subtokens. For the ones which aren't actual words (like cb17dy), you must add them as special tokens.

from transformers import GPT2TokenizerFast tokenizer = GPT2TokenizerFast.from_pretrained("gpt2") your_string = '[PRED] name [SUB] frankie and bennys frankie_and_bennys [PRED] cb17dy' SPECIAL_TOKENS = { "bos_token": "<|endoftext|>", "eos_token": "<|endoftext|>", "pad_token": "[PAD]", "additional_special_tokens": ["[SYS]", "[USR]", "[KG]", "[SUB]", "[PRED]", "[OBJ]", "[TRIPLE]", "[SEP]", "[Q]","[DOM]", 'frankie_and_bennys', 'cb17dy'] } tokenizer.add_special_tokens(SPECIAL_TOKENS) print(tokenizer(your_string)['input_ids']) print(tokenizer.convert_ids_to_tokens(tokenizer(your_string)['input_ids']))

[Anil-matcha]

This looks like a common issue with Python 3.8 . You can upgrade to Python 3.10 and it should work