Performance of VIT-B/32 is worse than RN50 on CC3M about open_clip

[carlini]

ViT-B performed worse for us on CC than a RN50. I suspect (but can not prove) this is because there's not enough data and vision transformers appear more data hungry than resnets. I don't have the accuracy off hand, but this looks comparable to what we were seeing.

[rwightman]

This is expected and your numbers appear reasonable. In training quite a few models at the lower end recently, the ViT-B models (even the smaller ones) will underperform similar sized ResNet models for smaller data. This includes up to at least the 12-15M sample range as I was unable to push ViT-B-32 past RN50 on cc12m or yfcc15m. I feel the crossover point is probably in the 40-100M sample range but have not verified that.

One could possibly work around this by using a pretrained backbone for the vision tower. There is partial support for this right now in some (preliminary) support for timm models...

One could modify a model config such as this to enable the timm_model_pretrained flag https://github.com/mlfoundations/open_clip/blob/main/src/open_clip/model_configs/timm-vit_base_patch32_224.json

You'd then be starting with a vision tower pretrained on imagenet. It significantly speeds up reaching decent zero shot and eval rseults BUT I'd caution against using an imagenet pretrained backbone and doing zero_shot eval on Imagenet, you'd probably want an alternate zero shot test dataset

[rwightman]

Moving to discussion for future reference