dannymerkx / next_word_prediction Goto Github PK
View Code? Open in Web Editor NEWPytorch implementation of next word prediction. Includes my own implementation of Google AI's Transformer architecture
Pytorch implementation of next word prediction. Includes my own implementation of Google AI's Transformer architecture
I notice that in your work, you have made it possible that parameters are comparable for 2 different models.
However, from what I understand, the same amount of parameter does not equal the same ability of learning.
If that's the case, a simple MLP layer could be stacked to compete with SOTA models in CV or NLP, but as far as we know, this is not true since we do know that some models learns well because of a more complex and efficient structure.
The transformer model itself is just much more powerful comparing to exsiting recurrent models like GRU, BiLSTM even with same amount of parameters. If a new recurrent model in the future could perform like transformer, does the idea of 'human reading style is like attention rather than recurrent' still hold?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.