Newly uploaded image_t2x.json would return assertation error about next-gpt

[ChocoWu]

Hi, @jwzhi
I guess you are using the updated Vicuna (not v0) because I also encountered the issue with the updated tokenizer where it tokenizes “[IMG0] [IMG1] [IMG2] [IMG3]” into “32002, 259, 32003, 259, 32004, 259, 32005” instead of the expected “32002, 32003, 32004, 32005”. But, when trying to tokenize “[IMG0]”, you will get the expected results, “32002”.
Unfortunately, I haven't been able to identify the cause of this issue yet.
So, I recommend either reverting to Vicuna-v0 or making modifications to the code to address this problem.