Internal Covariate Shift example in BatchNorm about annotated_deep_learning_paper_implementations

[vpj]

Sorry I didn't understand this statement

There is no difference in parameters of the l1 outputs' distribution. Maybe there should be another values of mean and variance?

Could you please elaborate? Do you mean to say distribution of L1 outputs don't change? What do you mean by "parameters of the l1 outputs' distribution"

[vklyukin]

Do you mean to say distribution of L1 outputs don't change?

Yes, you are right. I mean that distributions before and after the phrase "Then, after some training steps..." are the same, although the definition of Internal Covariate Shift says that it should change.

What do you mean by "parameters of the l1 outputs' distribution"

By parameters of the distribution I mean mean and variance

[vpj]

No the distribution can change, because the parameters of the layer change. Why do you think the distribution will stay constant?

[vklyukin]

I mean, that they should change, according to the definition of Internal Covariate Shift. And phrase:

Then, after some training steps, it could move to...

means, that the parameters of the distribution should change somehow. But it said that they moved to the N(0.5, 1) which is the same as it was before some training steps (the same mean and variance).

I think, that values of mean/variance after "some training steps" supposed to be differrent from the start ones, because now it's a bit confusing. For instance N(1.5, 1).

[vpj]

Oh crap, really sorry I didn't notice that the example has a mistake. It should be something like N(0.5, 1) changing to N(0.7, 1.1). Will correct it and close the issue.

Thanks again for spotting it.

[vpj]

Fixed the example: 2e54543