Cannot get correct result in eval_coco OR eval_voc about bidet

[Wuziyi616]

Hi! Thank you for your attention to our work. Actually we discovered in our experiments that, large learning rate is very important for training binary detectors. And we guess it's because large lr can avoid local minima for the binary neural network. In addition, if you use large lr, you need to use a large batch size to stabilize the training of the network. This might be why your loss goes to NaN when using batch_size=16 and lr=1e-3.

I also tried lr=1e-4 with batch_size=32 before, indeed the training process is more stable, but the performance is bad. If I remember correctly, we achieved ~60% mAP on VOC using this lr scheduler (lr starts from 1e-4 and decay to 1e-5 and 1e-6 when the loss stops decreasing).

So, from my perspective, I recommend you to use multiple GPUs to train on batch_size=32, because large initial lr is very important if you want to get good performance. If you can't find a way to train on large batch_size, maybe you can use some tricks like lr warmup to stabilize the early stage of training. But I'm sure whether the mAP will be as good as reported in our paper.

[hadhe145]

Thank you very much for your explanation, I will try again!