albarji / neurowriter Goto Github PK

Example: when trying with the Quijote corpus, some of these symbols are being generated:

  9   "zcan",
  9   "Ver",
  9   "onada",
  9   "nis",
  9   "menti",
  9   "memor",
  9   "llen",
  9   "guas",
  9   "enga",
  9   "encias",
  9   "cue",
  9   "áronle",
  9   "amen",
  9   "all",
  8   "yos",
  8   "tencia",
  8   "suje",
  8   "senti",
  8   "ólo",
  8   "neces",
  8   "nadas",
  8   "mun",
  8   "mez",
  8   "laron",
  8   "idor",
  8   "idades",
  8   "empera",
  8   "discre",
  8   "bastan",
  8   "alu",
  7   "vence",
  7   "Ú",
  7   "tentar",
  7   "rosos",

This is probably a result of merging symbols together. In the prune phase the low frequency symbols should be deleted, breaking them down to individual characters.

Instead of using an stacked LSTM model, which is powerful but slow, a viable alternative might be using a Conv+LSTM model. An working example of this architecture can be seen in the Tacotron 2 network, inside the encoder module (https://research.googleblog.com/2017/12/tacotron-2-generating-human-like-speech.html). Main points from this architecture:

• Character-level embeddings of size 512
• 3 layers of Conv, each with 512 filters and shape 5 x 1 (5-chars span) + BatchNormalization + ReLU + Dropout 0.5
• Bidirectional LSTM with 256 units in each direction + ZoneOut 0.1

Except for the ZoneOut, all of these can be implemented here.

Thu CUDNN version of LSTM layers does not work when running on CPU. The software should detect whether we are running just on CPU, and swap those layers by standard LSTM layers.

The attention model proposed in the project https://github.com/minimaxir/textgenrnn seems to work well.

Invalid configuration.
Errors:

• syntax error: (): did not find expected key while parsing a block mapping at line 1 column 1

This paper might contain useful insights for improving this application: https://arxiv.org/abs/1711.09534

Recent results show that while adaptive optimization methods do obtain better minima in the trainining loss function, they are more prone to overfitting, specially in network with more parameters than training data, which might well be the case here.

To avoid this it would be useful to SGD and SGD+Nesterov as optimizers in the hypertuning procedure.

• https://keras.io/optimizers/
• https://arxiv.org/abs/1705.08292

These work very well on sequence tagging tasks: https://arxiv.org/pdf/1702.02098.pdf

Try using the WaveNet parameters used in https://github.com/buriburisuri/speech-to-text-wavenet

https://github.com/fchollet/keras/blob/3dd3e8331677e68e7dec6ed4a1cbf16b7ef19f7f/keras/utils/training_utils.py#L56-L75

Some studies have been carried out on subword decompositions apt for poetry.

• Spanish: http://www.elcastellano.org/ns/edicion/2014/abril/silabas.html

It would be useful to adapt to SubWordTokenizer to allow providing a list of desired subword tokens.

This paper proposes some ideas on generating Haikus in Japanes using different generator models. Of special interest is how a CNN-LSTM model with MaxGlobalPooling is used.

http://www.anlp.jp/proceedings/annual_meeting/2017/pdf_dir/B7-5.pdf

https://keras.io/layers/recurrent/#cudnngru

The use of a CRF loss instead of the usual categorical cross entropy might provide advantages for generating coherent text along several tokens. This kind of loss is used in Google's text summarization model.

See for instance this issue in Chainer: chainer/chainer#1020

In the current version when building docker image with

make build-image GPU=1

the GPU flag is not being passed on to the appropriate make commands in the makefile.

Seem like vanilla Dropout is not so effective for Convolutional Layers. A better method might be to use SpatialDropout (https://faroit.github.io/keras-docs/1.2.0/layers/core/#spatialdropout2d) or no Dropout at all, just leave it for LSTM and Dense layers.

Unfortunately this layer is not yet implemented in Keras.