Special token handling breaks idempotency of sentencepiece due to extra spaces about tokenizers

[ArthurZucker]

Do you have a reproducer?
I'd love to fix it, but I'm not sure this is still happening

[ArthurZucker]

Llama based tokenizer don't have this issue anymore and was fixed by the metaspace refactoring.

[ArthurZucker]

Are you using legacy=False (mistral does not)

[ArthurZucker]

Also the snipper shared:

from transformers import LlamaTokenizer
model_id = "lmsys/vicuna-13b-delta-v1.1"
tokenizer = LlamaTokenizer.from_pretrained(model_id, add_bos_token = False, )
message = "<s>hello</s>"
decoded = tokenizer.decode(tokenizer(message)['input_ids'])
print(decoded, decoded == message)

this is on transformers side. Not tokenizers. I'll open a PR right away, it's super weird that it was not caught up until now