shuxinlin / nn4nlp_project Goto Github PK

After adding <EOS>, you break the LENGTH_UNIT you padded?

We need to use German language of CONLL 2003 dataset.

How to prepare the pre-trained word embedding?

Currently we transform from hidden state to scores of all possible labels by a linear layer. Is this a good model?

If use padding, mask the following:

1. objective function: don't include cross entropy from pair of matching to input padding.
2. attention: when computing alpha_{ij} (doing softmax), don't include padding. (softmax only over non-padding part.)

Currently attention does not involve cell states. Is that a good choice?

think about it carefully again: who attend to who?

Simply batch same-length sentences together. Don't do padding.

Maybe we need to take care of this case later.

Adams learning rate 0.001 or 0.0001

Understand what it does.

Can consider this later

In preprocess.py, in build_vocab(), there is a line of code:
all_words = collections.Counter(all_text).most_common()
It seems that at this moment the "most_common()" part is not really utilized. But could potentially pick the most common N words here.

In preprocessor.py, in minibatch(), in the last two lines with filter:
X_batch = filter(lambda mini_batch: len(mini_batch) == batch_size, X_batch)
Y_batch = filter(lambda mini_batch: len(mini_batch) == batch_size, Y_batch)
They will discard all batches that have sizes different from the user-specified batch size (default 32). I don't think this is the correct thing to do? It would unnecessarily discard many data.

This makes date (e.g. "1996-08-30") being treated differently from numbers (e.g. "83").

Need to think about whether this is what we want.

In preprocessor.py, minibatch(), the argument columns_to_batch is not used.

"an LSTM decoder (1 layer with tanh activation function) with a fixed attention mechanism that deterministically attends to the i-th input token when decoding the i-th output, and hence does not involve learning of any attention parameters"

May allow different dimensions.

I think we need to do that.

I don't think that in our tasks (NER and CCG supertagging) we would need . Can we simply remove it?

Need to use bi-directional LSTM for encoder.

In Preprocessor.entity_dict, '<EOS>' is 1.
In build_vocab() in preprocess.py, '<EOS>' is 2. ('<UNK>' is 1 here.)
Need to check whether this has any side effect.

Thanks for @hainow pointing out the following questions:
In ner.py, in train(),
init_dec_hidden = enc_hidden_out[-1] # -1 instead of 0?

Does it fit our use case that we discard any sentence that has character count larger than self.max_sentence_length? (In preprocessor.py, preprocess().)