walkerning / nics_fix_pytorch Goto Github PK

你好，请教一下，这边有看到对forward的处理主要是get_fix_forward函数里面，没有看到关于backward的相关的特殊处理，请问定点训练里面，这部分是怎么做的呢？有相关的文章吗？

Currently, when doing parallel training across multiple GPUs (e.g. DataParallel), the fixed-point config for every module is unprotectedly shared by multiple threads of different GPUs. This will cause undefined behavior for activation_fix, and batch norm buffer update for running_mean and running_var, as different GPU is doing this according to different split of data, and concurrently write/read the config variables. (They also do not exist on the same devices...)

If the config is registered as buffer, this issue will be solved. @Gaffey

And, to keep the multi-gpu training as close as possible to single-gpu training, maybe [https://hangzhang.org/PyTorch-Encoding/nn.html#encoding.nn.SyncBatchNorm](synchronized batch norm) should be used, also synchronized quantization of activation should be used? (Weights can be quantized separately, as they are the same and not in-place updated on any gpus)

To minimize the discrepancy of fix-point training and actually deployment, the bn and convolution should be merged together before quantization. @Gaffey