Where do classes get added as special tokens? about donut

[NielsRogge]

Hi,

Thanks for updating that :)

Regarding uploading the checkpoints, I can open up PRs on your repos. I'll open a PR on the Transformers repository today to add the model to the library. Will update you.

[NielsRogge]

Hi @gwkrsrch,

I've opened PRs on all 8 repos. Feel free to review and merge them :)

[gwkrsrch]

Hi, @NielsRogge

For (1),
7e45119 will resolve the issue. Thank you for pointing the issue out :)
As you found, such categorical special tokens (from values of the dictionaries) are not automatically added to the vocabulary.
Those should be manually added if needed. This will help improve performance a bit, but it is optional.

For (2),
the authors are pleased with the huggingface update. As you suggested, we would like to transfer them to hf.co/naver-clova-ix/*.
I tagged the original files with the official branch. As far as I understand, this update will enable the model repositories to support both huggingface and donut-python.
donut-python will use revision= "official" instead, to help the huggingface update.

I found 5 donut repositories at hf.co/nielsr/*. (Note that there are 8 donut model repositories in hf.co/naver-clova-ix/*)
If you make PRs to the main branches, I will accept them (or is it better to update the files on my own?).

Kind regards,
Geewook Kim

[gwkrsrch]

Great! I will wait for the updates :)
Please let me know when the PRs to the model repos are ready.

[NielsRogge]

Hi,

I'll open PRs on the repos today. When these are merged, I can also merge the Donut PR linked above.

Kind regards,

Niels

[gwkrsrch]

Hi @NielsRogge,

I have just checked that all model repositories got the PR and merged it successfully. And I see the huggingface update is now available. Awesome :)

I will close this issue since the related updates are completed and merged.