Recreation and extension of Elman's landmark 1990 results with Simple Recurrent Networks
We train a simple recurrent network on next word prediction and analyze the representations learned by the network. We observe that the network learns syntactic and semantic categories. The corpus currently consists of sentences generated by the simple grammar in data/grammar.py
, as in Elman's paper. We will also train the model on naturalistic data and compare the results (in preparation).
The model itself is trained in the notebook elman-syntax.ipynb
, which is a self-contained walkthrough of the key ideas.
- fix inconsistencies in the training loop
- generate realistic sentences with the model