vita-group / bnn_nobn

[CVPRW 21] "BNN - BN = ? Training Binary Neural Networks without Batch Normalization", Tianlong Chen, Zhenyu Zhang, Xu Ouyang, Zechun Liu, Zhiqiang Shen, Zhangyang Wang

BNN - BN = ? Training Binary Neural Networks without Batch Normalization

Codes for this paper BNN - BN = ? Training Binary Neural Networks without Batch Normalization. [CVPR BiVision Workshop 2021]

Tianlong Chen, Zhenyu Zhang, Xu Ouyang, Zechun Liu, Zhiqiang Shen, Zhangyang Wang.

Overview

Batch normalization (BN) is a key facilitator and considered essential for state-of-the-art binary neural networks (BNN). However, the BN layer is costly to calculate and is typically implemented with non-binary parameters, leaving a hurdle for the efficient implementation of BNN training. It also introduces undesirable dependence between samples within each batch.

Inspired by the latest advance on Batch Normalization Free (BN-Free) training, we extend their framework to training BNNs, and for the first time demonstrate that BNs can be completely removed from BNN training and inference regimes. By plugging in and customizing techniques including adaptive gradient clipping, scale weight standardization, and specialized bottleneck block, a BN-free BNN is capable of maintaining competitive accuracy compared to its BN-based counterpart. Experimental results can be found in our paper.

BN-Free Binary Neural Networks

Reproduce

Environment

pytorch == 1.5.0
torchvision == 0.6.0
timm == 0.4.5

Training on ImageNet

./script/imagenet_reactnet_A_bf.sh (BN-Free ReActNet-A)
./script/imagenet_reactnet_A_bn.sh (with BN ReActNet-A)
./script/imagenet_reactnet_A_none.sh (without BN ReActNet-A)

Citation

@misc{chen2021bnn,
      title={"BNN - BN = ?": Training Binary Neural Networks without Batch Normalization}, 
      author={Tianlong Chen and Zhenyu Zhang and Xu Ouyang and Zechun Liu and Zhiqiang Shen and Zhangyang Wang},
      year={2021},
      eprint={2104.08215},
      archivePrefix={arXiv},
      primaryClass={cs.LG}
}

Acknowledgement

https://github.com/liuzechun/ReActNet

https://github.com/liuzechun/Bi-Real-net

https://github.com/vballoli/nfnets-pytorch

https://github.com/deepmind/deepmind-research/tree/master/nfnets

bnn_nobn's Issues

where is the weight?

Binary weights

Hi, author. Thanks for the nice work!

有个问题想请教下作者。看到HardBinaryConv这段代码表示一个二值卷积，但是打印了权重，发现大概是0.005,0.006等等这样的数。看到他代码给二值化卷积torch.sign(real_weights)前面乘了个scaling_factor，这样好像就又变成32位浮点数了。我不知道我这样的理解对不对

CIFAR reactnet18 performance cannot be reproduced

Thanks for your work, but i cannot reproduce the performance of reactnet18 on CIFAR10 and CIFAR100, could you share your pth files of reactnet18 and the teacher networks?

How BNN_NoBN performance for simple NN.

Dear author, Thanks for the interesting topic. I would like to know how BNN_NoBN performance in simple NN : For example LeNET and VGG. I am looking for this topic for my hardware accelerator. If you don't mind could you share the paper and your code with simple DNN? Many thanks.