dannymerkx / next_word_prediction Goto Github PK

I notice that in your work, you have made it possible that parameters are comparable for 2 different models.
However, from what I understand, the same amount of parameter does not equal the same ability of learning.
If that's the case, a simple MLP layer could be stacked to compete with SOTA models in CV or NLP, but as far as we know, this is not true since we do know that some models learns well because of a more complex and efficient structure.
The transformer model itself is just much more powerful comparing to exsiting recurrent models like GRU, BiLSTM even with same amount of parameters. If a new recurrent model in the future could perform like transformer, does the idea of 'human reading style is like attention rather than recurrent' still hold?