pascanur / groundhog

Library for implementing RNNs with Theano

GroundHog

This library is built on top of Theano (http://deeplearning.net/software/theano/) and can be used with Jobman (http://deeplearning.net/software/jobman/). It is meant to provide a flexible yet efficient way of implementing complex recurrent models. Currently it supports variations of recurrent neural networks (such as DT-RNN, DOT-RNN, RNN Encoder-Decoder) and stacked version of them.

Most of the library documentation is still work in progress, but check the files containing Tut (in scripts) for a quick tutorial on how to use the library.

The library was developed at University of Montreal, in Lisa lab (http://lisa.iro.umontreal.ca/index\_en.html). Among the main contributors are Razvan Pascanu, KyungHyun Cho and Caglar Gulcehre.

The library is under the 3-clause BSD license, so it may be used for commercial purposes.

groundhog's Issues

Input data for NMT

How to give data for training and testing? And what is the format in which we give data for training

Type error in jobman

I am a researcher in the field of language modeling. I am trying to train an RNN-based language model. I take a train and valid sets (namely, ones included into Mikolov's RNNLM distribution). Then I: 1) Run generate.py and get .npz files.
2) In LM_dataset.py I comment out line 108:
self.penn_nchars = penn_data["n_chars"]
3) and set state['chunks'] = 'words'
4) To prevent TypeError I change all entries with float32 to float64 (e.g. in lines 105, 198).
Then, I run the DT_RNN_Tut.py
and get
data length is 81350
data length is 7090
data length is 8498
/usr/local/lib/python2.7/dist-packages/theano/scan_module/scan_perform_ext.py:85: RuntimeWarning: numpy.ndarray size changed, may indicate binary incompatibility
from scan_perform.scan_perform import *
/usr/local/lib/python2.7/dist-packages/theano/sandbox/rng_mrg.py:768: UserWarning: MRG_RandomStreams Can't determine #streams from size (Elemwise{Cast{int32}}.0), guessing 60*256
nstreams = self.n_streams(size)
Constructing grad function
Compiling grad function
took 42.7748370171
Validation computed every 1000
.. iter 0 cost 9.369 grad_norm 2.19e+00 traincost 1.35e+01 trainppl 1.17e+04 step time 0.343 sec whole time 45.245 sec lr 1.00e+00
Sample: Error in running algo (lr issue)
Took 0.108454414209 min
Traceback (most recent call last):
File "/opt/pycharm/pycharm-community-3.1.1/helpers/pydev/pydevd.py", line 1534, in
debugger.run(setup['file'], None, None)
File "/opt/pycharm/pycharm-community-3.1.1/helpers/pydev/pydevd.py", line 1145, in run
pydev_imports.execfile(file, globals, locals) #execute the script
File "scripts/DT_RNN_Tut.py", line 416, in
jobman(state, None)
File "scripts/DT_RNN_Tut.py", line 293, in jobman
main.main()
File "/home/mkudinov/workspace/GroundHog-master/groundhog/mainLoop.py", line 305, in main
[fn() for fn in self.hooks]
File "scripts/DT_RNN_Tut.py", line 260, in hook_fn
print dictionary[si],
IndexError: index out of bounds

index is set to the number exceeding the vocabulary size. At the same time the data is read properly so the error is made somewhere during the iteration.

Index out of bounds in hook_fn

I am a researcher in the field of language modeling. I am trying to train an RNN-based language model. I take a train and valid sets (namely, ones included into Mikolov's RNNLM distribution). Then I: 1) Run generate.py and get .npz files.
2) In LM_dataset.py I comment out line 108:
self.penn_nchars = penn_data["n_chars"]
3) and set state['chunks'] = 'words'
4) To prevent TypeError I change all entries with float32 to float64 (e.g. in lines 105, 198).
Then, I run the DT_RNN_Tut.py
and get
data length is 81350
data length is 7090
data length is 8498
/usr/local/lib/python2.7/dist-packages/theano/scan_module/scan_perform_ext.py:85: RuntimeWarning: numpy.ndarray size changed, may indicate binary incompatibility
from scan_perform.scan_perform import *
/usr/local/lib/python2.7/dist-packages/theano/sandbox/rng_mrg.py:768: UserWarning: MRG_RandomStreams Can't determine #streams from size (Elemwise{Cast{int32}}.0), guessing 60*256
nstreams = self.n_streams(size)
Constructing grad function
Compiling grad function
took 42.7748370171
Validation computed every 1000
.. iter 0 cost 9.369 grad_norm 2.19e+00 traincost 1.35e+01 trainppl 1.17e+04 step time 0.343 sec whole time 45.245 sec lr 1.00e+00
Sample: Error in running algo (lr issue)
Took 0.108454414209 min
Traceback (most recent call last):
File "/opt/pycharm/pycharm-community-3.1.1/helpers/pydev/pydevd.py", line 1534, in
debugger.run(setup['file'], None, None)
File "/opt/pycharm/pycharm-community-3.1.1/helpers/pydev/pydevd.py", line 1145, in run
pydev_imports.execfile(file, globals, locals) #execute the script
File "scripts/DT_RNN_Tut.py", line 416, in
jobman(state, None)
File "scripts/DT_RNN_Tut.py", line 293, in jobman
main.main()
File "/home/mkudinov/workspace/GroundHog-master/groundhog/mainLoop.py", line 305, in main
[fn() for fn in self.hooks]
File "scripts/DT_RNN_Tut.py", line 260, in hook_fn
print dictionary[si],
IndexError: index out of bounds

index is set to the number exceeding the vocabulary size. At the same time the data is read properly so the error is made somewhere during the iteration.

Shallow RNN in GroundHog

I want to train a simple shallow RNN with 1 hidden layer (like in Mikolov's work).
In this case I don't need an embedding layer and I would like to turn it off.
What is the right way to do it? I tried to do this:
rec_layer = rec(x, n_steps=x.shape[0],
init_state=h0*reset,
no_noise_bias=state['no_noise_bias'],
truncate_gradient=state['truncate_gradient'],
batch_size=1)
but it, obviously, didn't worrk.
I'm completely new at Teano so I didn't succeed in guessing it just looking through the code.

P.S. Obviously, it is not a bug and should not be here but I don't have another way of communication with you.

Input data for RNN language model

Sorry, but I have a new question.

I'm confused with theano programming style. I can't figure out what data goes to the first layer of the net. I see that a matrix containing dictionary indeces is read at each iteration inside SGD_momentum.call(self) but I cannot find the moment when this matrix is separated into one-hot vectors and sent to the first layer. After compilation catching this moment is completely impossible.

I'm, afraid that the first layer computes the dot product between indeces and weight matrix. How can I assure that it is not this way? How and where the input vector is transformed into the valid first layer input?

Beam search k-steps limiting outputs

I have a trained model and tested on 1000 lines en-fr corpus, the beam search sample just fed me back less than 20 translations. In the line 60 of sample.py: https://github.com/lisa-groundhog/GroundHog/blob/master/experiments/nmt/sample.py#L60

for k in range(3 * len(seq)):

Is there a specific reason we need to use this number of steps? Since I think it's not large enough for my model to generate results end with enc_dec.state['null_sym_target']. And in my understanding the early truncation of beam search process does not make sense? Why do we stop it? Or just because my trained model is bad (so can't get correct output).

No such file or directory : pentree_char_and_word.npz

I would like to use your good GroundHog to implement a sentences
segmentation task.

In order to understanf how GroundHog library works, I tried to run DT_RNN_Tut.py

But it says:
time python DT_RNN_Tut.py
Traceback (most recent call last):
File "DT_RNN_Tut.py", line 431, in
jobman(state, None)
File "DT_RNN_Tut.py", line 114, in jobman
train_data, valid_data, test_data = get_text_data(state)
File "DT_RNN_Tut.py", line 71,pentree_char_and_word.npz in get_text_data
can_fit=True)
File "/home/ubuntu/ggc/prove/DRNN/GroundHog-master/groundhog/datasets/LM_dataset.py",
line 97, in init
self.load_files()
File "/home/ubuntu/ggc/prove/DRNN/GroundHog-master/groundhog/datasets/LM_dataset.py",
line 105, in load_files
penn_data = numpy.load(self.path, mmap_mode=mmap_mode)
File "/usr/local/lib/python2.7/dist-packages/numpy/lib/npyio.py",
line 370, in load
fid = open(file, "rb")
IOError: [Errno 2] No such file or directory:
'/data/lisa/data/PennTreebankCorpus/pentree_char_and_word.npz'

What is pentree_char_and_word.npz?
And how to make DT_RNN_Tut.py working?

Looking forward to receive your kind helpfull hints.
Kind regards.
Marco