artemlos / deeplearning2020 Goto Github PK

I didn't look thoroughly at what needed to be cleaned in the datasets except splitting them into tokens.

Some datasetes might benefit from removing odd characters for instance.

I'm adding a simple TODO list for the LSTM baseline so that it will maybe be easier to keep track of what is left to do. Initially these things are to be implemented in the current LSTMBaseline model:

• Fix so that load_weights() works on the model. Problem is because the custom layer TiedDense does not wrap the kernel variable (the weight matrix of the TiedDense layer) with tf.Variable(). For reference: url explaining the issue.
  Fixed by removing use of kernel and directly using input embedding weight matrix in the matrix multiplication
• Dropout
• l2 penalty
• Gradient clipping norm 10
• Adam optimizer with b_1 = 0
• Perplexity (word-level) -> loss function
• bits per character (char-level) -> loss functions
• Skip connections
• Softmax temperature
• Dynamic evaluation