This repository implements Vision Transformer on a synthetic dataset of mnist colored numbers on textures/solid background .
![PatchEmbedding](https://private-user-images.githubusercontent.com/144267687/302552693-e8eef8af-2670-48c5-8825-8465d2c38ec0.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjQ2OTAwNDksIm5iZiI6MTcyNDY4OTc0OSwicGF0aCI6Ii8xNDQyNjc2ODcvMzAyNTUyNjkzLWU4ZWVmOGFmLTI2NzAtNDhjNS04ODI1LTg0NjVkMmMzOGVjMC5wbmc_WC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBVkNPRFlMU0E1M1BRSzRaQSUyRjIwMjQwODI2JTJGdXMtZWFzdC0xJTJGczMlMkZhd3M0X3JlcXVlc3QmWC1BbXotRGF0ZT0yMDI0MDgyNlQxNjI5MDlaJlgtQW16LUV4cGlyZXM9MzAwJlgtQW16LVNpZ25hdHVyZT0xMWUxNTYyMDQ5ZjI1ZDE2ZThhMTdhNjQ1MmU3NjNmYzU0MTQ2NDc4ZDBkZDcwMDNiMDQyYTRhNTgyNTNjOTc4JlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCZhY3Rvcl9pZD0wJmtleV9pZD0wJnJlcG9faWQ9MCJ9.7Xwxftzdh40O257tSLVUZc5nfP9ZJRPcee0hF70mbJU)
![Attention Block](https://private-user-images.githubusercontent.com/144267687/302552706-beaef101-c918-485c-aa71-9b1cb41ded50.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjQ2OTAwNDksIm5iZiI6MTcyNDY4OTc0OSwicGF0aCI6Ii8xNDQyNjc2ODcvMzAyNTUyNzA2LWJlYWVmMTAxLWM5MTgtNDg1Yy1hYTcxLTliMWNiNDFkZWQ1MC5wbmc_WC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBVkNPRFlMU0E1M1BRSzRaQSUyRjIwMjQwODI2JTJGdXMtZWFzdC0xJTJGczMlMkZhd3M0X3JlcXVlc3QmWC1BbXotRGF0ZT0yMDI0MDgyNlQxNjI5MDlaJlgtQW16LUV4cGlyZXM9MzAwJlgtQW16LVNpZ25hdHVyZT02ZmJlZjQxMTg4ODlmMDNjNWZkNDJlNjZlNGMzNjBmNThkMjdiM2E0OGM4NjE1YWViYTM2MDMwZGY5YjhlNTE3JlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCZhY3Rvcl9pZD0wJmtleV9pZD0wJnJlcG9faWQ9MCJ9.lZSAyCFfs40Vs4zlZ5t7eCr-5OgqfukopA8AUS-M6c4)
![Building Vision Transformer](https://private-user-images.githubusercontent.com/144267687/302552723-29e9c030-7a23-4ce3-b78d-46a5029e6fcd.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjQ2OTAwNDksIm5iZiI6MTcyNDY4OTc0OSwicGF0aCI6Ii8xNDQyNjc2ODcvMzAyNTUyNzIzLTI5ZTljMDMwLTdhMjMtNGNlMy1iNzhkLTQ2YTUwMjllNmZjZC5wbmc_WC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBVkNPRFlMU0E1M1BRSzRaQSUyRjIwMjQwODI2JTJGdXMtZWFzdC0xJTJGczMlMkZhd3M0X3JlcXVlc3QmWC1BbXotRGF0ZT0yMDI0MDgyNlQxNjI5MDlaJlgtQW16LUV4cGlyZXM9MzAwJlgtQW16LVNpZ25hdHVyZT01MGExMjMwOThmZGJhZmNkMzYyNGVlODVmNThhM2MyNjQ1YTZkZDM3YTUzMGZlMWVlMjE5ZjhhNjkyNjdhODZiJlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCZhY3Rvcl9pZD0wJmtleV9pZD0wJnJlcG9faWQ9MCJ9.AjrgGCHzKCSXt2XKcYzobW7aN61V_ny3sDw2WdFQGRA)
![](https://private-user-images.githubusercontent.com/144267687/278376773-6a60022e-b6f6-4037-9839-c62fa7cabaf2.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjQ2OTAwNDksIm5iZiI6MTcyNDY4OTc0OSwicGF0aCI6Ii8xNDQyNjc2ODcvMjc4Mzc2NzczLTZhNjAwMjJlLWI2ZjYtNDAzNy05ODM5LWM2MmZhN2NhYmFmMi5wbmc_WC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBVkNPRFlMU0E1M1BRSzRaQSUyRjIwMjQwODI2JTJGdXMtZWFzdC0xJTJGczMlMkZhd3M0X3JlcXVlc3QmWC1BbXotRGF0ZT0yMDI0MDgyNlQxNjI5MDlaJlgtQW16LUV4cGlyZXM9MzAwJlgtQW16LVNpZ25hdHVyZT0yODk1YTRjOTNjYzYxZWQ1YmYyYTU4ZTQ5ZjQyMGQ2MzkzOTFhZmY1MGIxNTM3MWE5ZDVmZjg3ZjJkYWUxY2I2JlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCZhY3Rvcl9pZD0wJmtleV9pZD0wJnJlcG9faWQ9MCJ9.oNeBl4uGN67S3VIwF5gyue6ZCePW460hpjMN4rwc06M)
For setting up the mnist dataset: Follow - https://github.com/explainingai-code/Pytorch-VAE#data-preparation
Download Quarter RGB resolution texture data from ALOT Homepage In case you want to train on higher resolution, you can download that as well and but you would have to create new imdb.json Rest of the code should work fine as long as you create valid json files.
Download imdb.json from Drive Verify the data directory has the following structure after textures download
VIT-Pytorch/data/textures/{texture_number}
*.png
VIT-Pytorch/data/train/images/{0/1/.../9}
*.png
VIT-Pytorch/data/test/images/{0/1/.../9}
*.png
VIT-Pytorch/data/imdb.json
- Create a new conda environment with python 3.8 then run below commands
git clone https://github.com/explainingai-code/VIT-Pytorch.git
cd VIT-Pytorch
pip install -r requirements.txt
python -m tools.train
for training vitpython -m tools.inference
for running inference, attention visualizations and positional embedding plots
config/default.yaml
- Allows you to play with different aspects of VIT
Outputs will be saved according to the configuration present in yaml files.
For every run a folder of task_name
key in config will be created
- Best Model checkpoint in
task_name
directory
During inference the following output will be saved
- Attention map visualization for sample of test set in
task_name/output
- Positional embedding similarity plots in
task_name/output/position_plot.png
Following is a sample attention map that you should get
![](https://private-user-images.githubusercontent.com/144267687/278374219-76d63318-478f-41aa-a321-e907c29e26fb.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjQ2OTAwNDksIm5iZiI6MTcyNDY4OTc0OSwicGF0aCI6Ii8xNDQyNjc2ODcvMjc4Mzc0MjE5LTc2ZDYzMzE4LTQ3OGYtNDFhYS1hMzIxLWU5MDdjMjllMjZmYi5wbmc_WC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBVkNPRFlMU0E1M1BRSzRaQSUyRjIwMjQwODI2JTJGdXMtZWFzdC0xJTJGczMlMkZhd3M0X3JlcXVlc3QmWC1BbXotRGF0ZT0yMDI0MDgyNlQxNjI5MDlaJlgtQW16LUV4cGlyZXM9MzAwJlgtQW16LVNpZ25hdHVyZT00YWU1OWNkMjk2NzY4NTVjNDAwNjU3ZjE1ZWRlNTFjMDdiNmNhZDM3MmMyNDUyZTQ4M2FjZmQwZDEzZWQyMGJmJlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCZhY3Rvcl9pZD0wJmtleV9pZD0wJnJlcG9faWQ9MCJ9.RKM9uA0nhmu-o5-8L8Nkgj7DJ_0mTpbHA8gPK5_8aEg)
Here is a positional embedding similarity plot you should get
![](https://private-user-images.githubusercontent.com/144267687/278374254-0ba9afa7-1d5d-4cbc-8b9d-0a4367e9f9ed.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjQ2OTAwNDksIm5iZiI6MTcyNDY4OTc0OSwicGF0aCI6Ii8xNDQyNjc2ODcvMjc4Mzc0MjU0LTBiYTlhZmE3LTFkNWQtNGNiYy04YjlkLTBhNDM2N2U5ZjllZC5wbmc_WC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBVkNPRFlMU0E1M1BRSzRaQSUyRjIwMjQwODI2JTJGdXMtZWFzdC0xJTJGczMlMkZhd3M0X3JlcXVlc3QmWC1BbXotRGF0ZT0yMDI0MDgyNlQxNjI5MDlaJlgtQW16LUV4cGlyZXM9MzAwJlgtQW16LVNpZ25hdHVyZT1jMDgxNGY1Mjk5M2VhMDdmMjM5Zjk4NzJjNDZjNjhmYTM2ODRhNGQ0YzI3Yjg0NWY3NTljN2M5MzYzZmE4YmI4JlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCZhY3Rvcl9pZD0wJmtleV9pZD0wJnJlcG9faWQ9MCJ9.9-_UvpHykDFFiWrEyef7ymlSes1y7NT1aKBSscgkmNI)
@misc{dosovitskiy2021image,
title={An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale},
author={Alexey Dosovitskiy and Lucas Beyer and Alexander Kolesnikov and Dirk Weissenborn and Xiaohua Zhai and Thomas Unterthiner and Mostafa Dehghani and Matthias Minderer and Georg Heigold and Sylvain Gelly and Jakob Uszkoreit and Neil Houlsby},
year={2021},
eprint={2010.11929},
archivePrefix={arXiv},
primaryClass={cs.CV}
}