dreadlord1984 / network-speed-and-compression

Network acceleration methods

Network-acceleration

• https://github.com/yihui-he/channel-pruning

Network Compression

• Learning both Weights and Connections for Efficient Neural Network.
• Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding.
• SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model size.
• 8-Bit Approximations for Parallelism in Deep Learning.
• Neural Networks with Few Multiplications.
• Compression of Deep Convolutional Neural Networks for Fast and Low Power Mobile Applications.
• Hardware-oriented Approximation of Convolutional Neural Networks.
• Reduced-Precision Strategies for Bounded Memory in Deep Neural Nets.
• Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations.
• DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients.
• Deep Learning with Limited Numerical Precision.
• Dynamic Network Surgery for Efficient DNNs.
• Understanding the Impact of Precision Quantization on the Accuracy and Energy of Neural Networks.
• Variational Dropout Sparsifies Deep Neural Networks https://github.com/ars-ashuha/variational-dropout-sparsifies-dnn
• Soft Weight-Sharing for Neural Network Compression
• LCNN: Lookup-based Convolutional Neural Network
• Bayesian Compression for Deep Learning
• ThiNet: A Filter Level Pruning Method for Deep Neural Network Compression

Network quantization

• https://petewarden.com/2016/05/03/how-to-quantize-neural-networks-with-tensorflow/
• https://github.com/aaron-xichen/pytorch-playground#quantization
• Quantizing deep convolutional networks for efficient inference: A whitepaper
• https://nervanasystems.github.io/distiller/quantization/

Binary networks

• XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks.
• Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1.
• BMXNet: An Open-Source Binary Neural Network Implementation Based on MXNet
• Optimize Deep Convolutional Neural Network with Ternarized Weights and High Accuracy
• Trained Ternary Quantization [2017]
• Training wide residual networks for deployment using a single bit for each weight [2018]
  • https://github.com/szagoruyko/binary-wide-resnet

Convolution decomposition

• Flattened convolutional neural networks for feedforward acceleration [2015]
• Speeding-up Convolutional Neural Networks Using Fine-tuned CP-Decomposition [2015]
  • https://github.com/vadim-v-lebedev/cp-decomposition
  • https://github.com/jacobgil/pytorch-tensor-decompositions
  • https://medium.com/@keremturgutlu/tensor-decomposition-fast-cnn-in-your-pocket-f03e9b2a6788
• Compression of Deep Convolutional Neural Networks for Fast and Low Power Mobile Applications [2016]

Pruning

• Pruning Convolutional Neural Networks for Resource Efficient Inference [2017]
  • https://jacobgil.github.io/deeplearning/pruning-deep-learning
• Pruning in tensorflow
  • https://github.com/ex4sperans/pruning_with_tensorflow

Frameworks

• https://github.com/Tencent/PocketFlow
• https://code.fb.com/ml-applications/qnnpack/
• https://developer.nvidia.com/tensorrt
• https://github.com/NervanaSystems/distiller
• https://github.com/tensorflow/tensorflow/tree/master/tensorflow/contrib/quantize

Network models for mobile devices

• Mobilenet
• CondenseNet
• NASNet
  • https://github.com/tensorflow/models/tree/master/research/slim/nets/nasnet
• ShuffleNet
• SqueezeNet
• FD-MobileNet[2018]
• ProxylessNAS[2019]
  • https://github.com/MIT-HAN-LAB/ProxylessNAS
• MnasNet[2018]
• ESPNetv2[2018]
  • https://github.com/sacmehta/ESPNetv2
• SqueezeNext [2018]

Training smaller model based on existing one

• Data-Free Knowledge Distillation for Deep Neural Networks (https://arxiv.org/pdf/1710.07535.pdf) [2017]
  • https://github.com/iRapha/replayed_distillation
• Stealing Machine Learning Models via Prediction APIs (https://arxiv.org/pdf/1609.02943.pdf) [2016]

Face recognition

• MobiFace [2018]

To look at:

https://github.com/wpf535236337/real-time-network
https://github.com/dkozlov/awesome-knowledge-distillation
https://github.com/memoiry/Awesome-model-compression-and-acceleration
https://github.com/ljk628/ML-Systems/blob/master/dl_cnn.md
https://github.com/songhan/SqueezeNet-Deep-Compression
https://github.com/jiaxiang-wu/quantized-cnn
https://github.com/andyhahaha/Convolutional-Neural-Network-Compression-Survey
https://github.com/Zhouaojun/Efficient-Deep-Learning
https://github.com/ZFTurbo/Keras-inference-time-optimizer
https://github.com/becauseofAI/MobileFace