LSTM language model baseline gap about examples

[Smerity]

Secondary confirmation.

P100, Driver Version: 375.26, cuDNN: 5, CUDA: 8.0

-----------------------------------------------------------------------------------------
| end of epoch   6 | time: 18.46s | valid loss  4.89 | valid ppl   133.20
-----------------------------------------------------------------------------------------
=========================================================================================
| End of training | test loss  4.86 | test ppl   128.70
=========================================================================================

Turning off gradient clipping:

-----------------------------------------------------------------------------------------
| end of epoch   6 | time: 20.51s | valid loss  4.77 | valid ppl   118.37
-----------------------------------------------------------------------------------------
=========================================================================================
| End of training | test loss  4.74 | test ppl   114.18
=========================================================================================

The issue appears caused by the clip_gradient function - though I've not yet found the exact source of the issue in that function. Replacing it with the recently added clip_grad_norm (source) brings us back towards the reported perplexity of 113 while using gradient clipping.

-----------------------------------------------------------------------------------------
| end of epoch   6 | time: 16.87s | valid loss  4.78 | valid ppl   119.09
-----------------------------------------------------------------------------------------
=========================================================================================
| End of training | test loss  4.75 | test ppl   115.49
=========================================================================================

With a different random seed (42), the perplexity is more in line with the reported 113, so the difference is likely superficial.

-----------------------------------------------------------------------------------------
| end of epoch   6 | time: 17.03s | valid loss  4.77 | valid ppl   118.16
-----------------------------------------------------------------------------------------
=========================================================================================
| End of training | test loss  4.74 | test ppl   114.46
=========================================================================================