Multiple loss.backward() for each minibatch during training about cnnimageretrieval-pytorch

[filipradenovic]

1. As far as I know it sums all the gradients, take a look at this discussion for more insights about PyTorch and gradients. A really important thing is to do optimizer.zero_grad() before the loop in which you do loss.backward(), because the gradients need to be cleared before accumulating them for a new batch.
2. First of all, both implementations would yield the same result. But yes, there is a reason why we do it the way we do. It is a trade-off between performance and memory usage. As it is implemented right now, it is the most memory efficient (so that you can do training with larger images as well, if necessary). Doing only one loss.backward() per optimizer.step() would be the most memory hungry, but also the fastest. Code can be simply adjusted if you want to do that. You can also implement it to split each batch into smaller sub-batches, so that they nearly fill up the memory.
3. During validation there is no backward pass or gradient computation, it is a simple evaluation of the network so it does not require a lot of memory. Because of that, we do not need to split the loss computations, we can simply compute it for the whole batch.

Hope this helps!

[itzsid]

Thanks for the quick replies @filipradenovic. This is helpful.