trellixvulnteam / deepformer_dcrz

Learning the deep secret of Transformers

Deep Vision Transformers with K-FAC optimizer & gabor attention

The KFAC code was originally forked from https://github.com/gpauloski/kfac_pytorch

The VIT model was referenced from https://github.com/jeonsworld/ViT-pytorch

DeepFormer leverages several excellent libraries(hiddenlayer, torchstat, torchsummary...) to visualize the secret of deep transformers.

Requirements

This code is validated to run with PyTorch v1.3

Installation

$ git clone https://github.com/closest-git/DeepFormer.git
$ cd 
$ pip install .

Usage

Citation

@article{chen2021iterative,
  title={An iterative K-FAC algorithm for Deep Learning},
  author={Chen, Yingshi},
  journal={arXiv preprint arXiv:2101.00218},
  year={2021}
}