lightonai / dfa-scales-to-modern-deep-learning

Study on the applicability of Direct Feedback Alignment to neural view synthesis, recommender systems, geometric learning, and natural language processing.

Direct Feedback Alignment Scales to Modern Deep Learning Tasks and Architectures

Code for our paper Direct Feedback Alignment Scales to Modern Deep Learning Tasks and Architectures.

We study the applicability of Direct Feedback Alignment (DFA) to neural view synthesis, recommender systems, geometric learning, and natural language processing. At variance with common beliefs, we show that challenging tasks can be tackled in the absence of weight transport.

Reproducing our results

• Instructions for reproduction are given within each task folder, in the associated README.md file.

Requirements

• A requirements.txt file is available at the root of this repository, specifying the required packages for all of our experiments;
• Our DFA implementation, TinyDFA, is pip-installable: from the TinyDFA folder, run pip install .;
• tsnecuda may require installation from source: see the tsne-cuda repository for details;
• Neural rendering datasets can be found on the NeRF website—other datasets will be automatically fetched.

Citation

If you found this code and findings useful in your research, please consider citing:

@article{launay2020dfascaling,
  title={Direct Feedback Alignment Scales to Modern Deep Learning Tasks and Architectures},
  author={Launay, Julien and Iacopo, Poli and Francois, Boniface and Krzakala, Florent},
  journal={arXiv preprint arXiv:2006.12878},
  year={2020}
}

About LightOn/LightOn Cloud

LightOn develops a light-based technology required in large scale artificial intelligence computations. Our ambition is to significantly reduce the time and energy required to make sense of the world around us.

Please visit https://cloud.lighton.ai/ for more information on accessing our technology.

dfa-scales-to-modern-deep-learning's Issues

Need better instructions for installing dependencies

Congratulations on your paper. I was trying to play around a bit and found that the dependencies are not easy to install (at least on arm64)

1. The requirements file is missing an = sign on line that refers to seaborn.
2. The package sentencepiece==1.9.0 is not available with pip on arm64.
3. torch==1.5.0+cu101 is not available with pip

The first two problems are easily fixable and the third one too, but having encountered three hiccups in installing just half of the dependencies listed in requirements.txt, I decided to file this issue. Please add better instructions for installing dependencies, or if possible, please provide docker images (arm64 compatible) to solve all hassles in one go.

Where do I get `bpe_models/wikitext-2.bpe.32000`?

I'm trying to run train_lm.py to replicate Table 5 in your paper. For the simple baseline experiment python train_lm.py --gpu_id 0 --beta2 0.98 I am missing the file bpe_models/wikitext-2.bpe.32000. How do I download or generate this file? Also the default --dataset is 'wikitext103', not 'wikitext2', is this compatible with bpe_models/wikitext-2.bpe.32000?

Thanks! And cool project!