LayerNormalization & other normalization techniques about cntkx

[delzac]

Hmmm, i just looked through the scale and bias in LayerNormalization, i don't see anything wrong with it, what's your reservation?

*I think its better to have the conversation here, there might be people with the same issue :)

[haixpham]

scale = Parameter(_INFERRED, init=initial_scale, name='scale')
at https://cntk.ai/pythondocs/_modules/cntk/layers/layers.html#LayerNormalization.

It's predetermined that 'scale' is rank-1, so if input is rank-2 or more, scale will only have the length of the last dimensionality of input.

For example, if
x = C.input((3,100,100)
then
scale.shape == (100, )
while it should be (3, 100, 100).

[delzac]

That's because LayerNormalization was created/invented for regularization of RNNs and not CNNs.

My experience with using LayerNormalization instead of BatchNormalization is that it hurts performance of CNNs.

[haixpham]

I haven't tried LayerNorm CNN yet, but using BatchNorm to train CNN to learn features leads to noisy features - I think because it requires popular batch statistics at runtime. Without normalization the training optimization is too "stochastic" for me.

Btw, is there a way to infer tensor rank when defining a custom layer? Currently I define one layer for each tensor rank.

[delzac]

I have had good experiences with batchnorm on cnns even with small batch sizes of around 16, sometimes even on batch size 4.

If you want to extend LayerNormalization to all three static axes, i think you can do something like this:

scale = Parameter((_INFERRED, _INFERRED, _INFERRED), init=initial_scale, name='scale')

[haixpham]

That's exactly what I did, so I have LayerNormalization1D, 2D, 3D etc...
Would be nice if it's possible to define one layer functional to handle all cases.

[delzac]

I gave some thought about it, i don't think it is doable cause there's no way of knowing the shape of the input tensor at the point of defining the parameter.

[haixpham]

It's not possible using only Python API, unless (IMO) we can define C API similar to BatchNorm, and do resize() & reshape() in C.

[delzac]

Well, if you are okay with not using the @C.BlockFunction decorator in the inner function of your layer function, then yes you can do it using python to check for their shape and dynamically route the correct function to it.

Alternatively, why not create your layernorm like the Dense API where it explicitly ask for either input_rank or map_rank to decide what rank its weight parameter is.

[haixpham]

On that note, can you please tell me what's the purpose of C.BlockFunction?

[delzac]

There are two uses for C.BlockFunction, it wraps all the 'inner' ops and make it appear like a single op when you view it using Netron or when you save it as a graph, i.e. it looks like a single block.

Second, a C.BlockFunction is also a C.Function. A C.Function is used to create a cntk function. It particularly important to first wrap certain blocks of code as a C.Function when you want to use encoder-decoder attention mechanism. You can see this for an example. If you remove the decorator the whole thing will throw an error.

[haixpham]

Thanks mate, very insightful! Look forward to future discussion.

[delzac]

Happy to help. If you have any issues, feel free to reopen or open a new issue!