Parallel mode issue about mad

[craffel]

There's no real advantage to using soft decoding at inference time - standard softmax attention typically works as well or better and has the same complexity/capabilities. You only really get a gain (in terms of cost and the ability to online decoding) from monotonic attention if you do hard attention at inference time.

[craffel]

IIRC you could probably do it with a custom kernel, but it's been so long since I thought about it that I don't remember what I had in mind. Otherwise you could try using higher-precision floats, but AFAIK there aren't any other options.

[gn-yh-lim]

what do you mean by a custom kernel? could you please give an example? thanks!

[craffel]

E.g. a custom CUDA kernel. No examples, sorry.

[gn-yh-lim]

thanks for clarifying. Just one last question: during inference, from my understanding, it would be ok to use soft monotonic attention as in training (despite it being more expensive) instead of hard monotonic attention. Could you please confirm if that is true? The only potential problem that I see is the introduction of white noise before the sigmoid layer, would really appreciate it if you could give some insights into this. Thanks again for the great paper!