kd_loss implementation issue about ld

[HikariTJU]

I remeber that .mean(1) is equal to reduction='batch_mean‘ ？

[ZaberKo]

I remeber that .mean(1) is equal to reduction='batch_mean‘ ？

Here is the source code of F.kl_div:
https://github.com/pytorch/pytorch/blob/defa0d3a2d230e5d731d5c443c1b9beda2e7fd93/torch/nn/functional.py#L2949-L2958

And the problem here is that the kd_loss is subsequently averaged by @weighted_loss wrapper.

[HikariTJU]

So batch_mean equals .mean(0)?

[ZaberKo]

So batch_mean equals .mean(0)?

No. "batchmean" means .sum()/batch_size, i.e., .sum(1).mean()

[HikariTJU]

OK, I get your point, you mean mathmatically .sum(1) is the correct implementation and .mean(1)=.sum(1)/16
That's true, but how is it related to batchmean?

[ZaberKo]

OK, I get your point, you mean mathmatically .sum(1) is the correct implementation and .mean(1)=.sum(1)/16 That's true, but how is it related to batchmean?

BTW, I also found that loss_ld used weighted sum and was not divided by avg_factor (i.e. sum of weights). Is this a typo or intended behavior for not using normalization?

[ZaberKo]

FYI: I record the factor ratio avg_factor/(self.reg_max+1) during the training. Maybe it will help this discussion.

[HikariTJU]

It's a intended behavior because experiment shows not dividing is better. Don't know the theory behind this though

[ZaberKo]

It's a intended behavior because experiment shows not dividing is better. Don't know the theory behind this though

I see, thanks for the reply.