Missing keys in demo notebook about videotransformer-pytorch

[Simcs]

Thank you for the answer!
After following @realgump 's suggestion, I could load the ViViT parameters normally.

For those who want to know the detailed instructions, I did the following steps.

1. import ViViT model from original repo

from video transformer import ViViT

2. set weights_from param to 'kinetics' when initializing ViViT model

    pretrain_pth = './vivit_model.pth'
    num_frames = num_frames * 2
    frame_interval = frame_interval // 2
    model = ViViT(num_frames=num_frames,
                  img_size=224,
                  patch_size=16,
                  embed_dims=768,
                  in_channels=3,
                  attention_type='fact_encoder',
                  return_cls_token=True,
                  weights_from='kinetics',
                  pretrain_pth=pretrain_pth)

3. remove the remaining initialization code

# comment out init_from_pretrain_() method
cls_head = ClassificationHead(num_classes=num_class, in_channels=768)
# msg_trans = init_from_pretrain_(model, pretrain_pth, init_module='transformer')
msg_cls = init_from_pretrain_(cls_head, pretrain_pth, init_module='cls_head')

Thanks again for the quick response!

[realgump]

Have you set weights_from? I have met the same error and fixed it after set weights_from.

[mx-mark]

@Simcs Sorry for that, it should be the reason as @realgump said where we have updated the weights initialization function. But we have not keep path with it in notebook.

[zz0126]

我想问问您成功加载vivit-b预训练权重了吗？