Comments (2)
Hello,
the decoder initializes their hidden states by means of a MLP, which takes a mean of the representation of the input sentence (see e.g. the expression before Eq. 7 from this paper: https://arxiv.org/pdf/1502.03044.pdf). Note that, since we work in minibatches, each sentence from a minibatch can have a different length. Therefore, we use masks, in order to know which are the actual words. MaskedMean()
computes the mean of the annotations, but considering their associated mask.
Similarly, we pass the full sequence of annotations to the decoder, for computing attention on them. But we don't want to give attention to masked annotations. Therefore, we apply the mask on them with MaskLayer
.
from nmt-keras.
Your explanation is very clear, thank you very much again. @lvapeab
from nmt-keras.
Related Issues (20)
- Support for Factored Models ? HOT 1
- consume long time for predicting validation output HOT 3
- Confusion with opennmt-tf HOT 1
- Missing auto setup of required packages for running this library HOT 1
- How to use pretrained word2vec embeddings? HOT 1
- Getting error index out of range when training a Transformer model HOT 10
- Using CPU for inference with GPU-trained model HOT 20
- Evaluating perplexity HOT 4
- Getting error when using Tensorboard HOT 2
- Save perplexity on training and validation sets HOT 5
- Regd Rare Words/OOV Tokens ? HOT 9
- Sampling decoding HOT 1
- Strange behavior with plotting metrics for validation HOT 2
- Issue with ensemble scoring method HOT 3
- AssertionError: Reduction function "Noam" unimplemented! HOT 1
- Data Error ? HOT 6
- Detecting multiple GPUs HOT 9
- Training Error HOT 1
- Conversion to TFJS HOT 1
- Example Colab Fails HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from nmt-keras.