Adding VideoMAE to HuggingFace Transformers about videomae

[yztongzhan]

Hi @NielsRogge! Thanks for your suggestions! We have created a org for https://huggingface.co/videomae. I want to know how to upload our models correctly?

[yztongzhan]

Hi @NielsRogge! Is there any update?

[NielsRogge]

Hi @yztongzhan,

I just worked a bit further on it, I've implemented VideoMAEForPreTraining now as well, which includes the decoder and loss computation. The PR is now ready for review and will be reviewed by my colleagues.

Also, would it be possible to create an organization on the hub for Multimedia Computing Group, Nanjing University, with a short name (rather than just VideoMAE)? Cause otherwise people will have to do:

from transformers import VideoMAEModel

model = VideoMAEForVideoClassification.from_pretrained("VideoMAE/videomae-base-finetuned-kinetics")

for instance, which means they have to type quite a lot of videomae 😂 also, if there would be newer models coming out that are also part of the research of the same organization (such as AdaMixer), it makes sense to upload them to the same organization on the hub.

Regards,

Niels

[wanglimin]

Hi @NielsRogge ,

Thanks for your update. We have created an organization account on the hub:

https://huggingface.co/MCG-NJU

You can use this organization for storing our model checkpoints. BTW, you could also include our other repo such as AdaMixer and MixFormer.

Best,
Limin

[wanglimin]

@NielsRogge Any update？

[NielsRogge]

Hi @wanglimin,

the model will soon be added to the library. I'll transfer the weights to the MCG-NJYU organization today.

Are you interested in collaborating on a script for easy fine-tuning?

[NielsRogge]

I've currently transferred 3 models: https://huggingface.co/models?other=videomae.

To make the model names not too long, I would use the following names:

model_names = [
        # Kinetics-400 checkpoints (short = pretrained only for 800 epochs instead of 1600)
        "videomae-base-short",
        "videomae-base-short-finetuned-kinetics",
        "videomae-base",
        "videomae-base-finetuned-kinetics",
        "videomae-large",
        "videomae-large-finetuned-kinetics",
        # Something-Something-v2 checkpoints (short = pretrained only for 800 epochs instead of 2400)
        "videomae-base-short-ssv2",
        "videomae-base-short-finetuned-ssv2",
        "videomae-base-ssv2",
        "videomae-base-finetuned-ssv2",
    ]

Is that ok for you? Also, are you interested in adding model cards to the repos on the hub? Each model has its own git repo, and the model card is just a README (Markdown file).

[NielsRogge]

Hi @wanglimin,

VideoMAE has been added to the library! https://huggingface.co/docs/transformers/main/en/model_doc/videomae

Checkpoints are on the hub: https://huggingface.co/models?other=videomae

[yztongzhan]

Hi @NielsRogge! Thanks again for your efforts! We will add these links in README.

[wanglimin]

@NielsRogge , Thanks a lot for your help!