Positional Encoding Clarification about annotated-transformer

[liangbright]

add sine wave directly to word embedding vector. It is like to attach a name-tag on someone's face..., kind of weird

[guillaume-chevalier]

This might help you: https://github.com/guillaume-chevalier/Linear-Attention-Recurrent-Neural-Network/blob/master/AnnotatedMultiHeadAttention.ipynb

• First, the almost-original pos encoding is plotted, without any random offset.
• Second, the frequencies are changed to more "perfect" or "natural" ones so that it's like counting in binary, and also they are concatenated as features instead of added. I still wonder why the original frequencies were like they were (I'd love to know). I also wonder why they added them instead of concatenating them, here, concatenating make more sense to me.

[gussmith]

By concatenating, the dimension increases and thus the number of parameters.
That's one advantage to keep lower dimensions.

The addition is similar to response of cells in early visual cortex, like in V1 in the brain. Many cell response to a visual stimuli, say an edge, yet the response of every cell is in addition modulated by eye position (angle of eye direction) and by vergence (~focus distance).
Thus depending on where you look, the same visual stimulus will elicit a different response in the neuron. The overall population of cells thus not only encode the visual stimulus in the overall visual field, but also the eye positions (the direction where the eye(s) is/are looking)
Here the positional encoding is a bit like the eye position.