kamalkraj / vision-transformer

Vision Transformer using TensorFlow 2.0

Vision Transformer

Abstract

Abstract: While the Transformer architecture has become the de-facto standard for natural language processing tasks, its applications to computer vision remain limited. In vision, attention is either applied in conjunction with convolutional networks, or used to replace certain components of convolutional networks while keeping their overall structure in place. We show that this reliance on CNNs is not necessary and a pure transformer can perform very well on image classification tasks when applied directly to sequences of image patches. When pre-trained on large amounts of data and transferred to multiple recognition benchmarks (ImageNet, CIFAR-100, VTAB, etc), Vision Transformer attains excellent results compared to state-of-the-art convolutional networks while requiring substantially fewer computational resources to train.

Vision Transformer implemented using TF 2.0.

example usage can found in cifar.ipynb

PyTorch version here

Citations

@inproceedings{
    anonymous2021an,
    title={An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale},
    author={Anonymous},
    booktitle={Submitted to International Conference on Learning Representations},
    year={2021},
    url={https://openreview.net/forum?id=YicbFdNTTy},
    note={under review}
}

vision-transformer's Issues

Why tf.function to wrap call

Hi, Thanks for your works!!

I have a question about this line of code:

Vision-Transformer/model.py

Line 146 in 1ee5e62

@tf.function

Why wrapping call to tf.function?

Why only use the first patch? Thanks

I don't understand in line 158 of model.py:
x = self.to_cls_token(x[:, 0])
If the first dimension of x is batch, then the 2nd dimension 0 should be patch, as the dimension of x should be [batch, patch, feature]. Does it mean only the first patch is used? Could anybody help me on this? Thanks a lot.

Do you have plan to improve the test accuracy?

Hello!!
Im beginner.
Do you have plan to improve the test accuracy?

I tried to run for 300 epochs. The result is below.
epoch 300: train loss 0.10614. train accuracy 0.96536
epoch 300: test loss 5.57218. test accuracy 0.48910
Thank you.