pascanur / groundhog Goto Github PK
View Code? Open in Web Editor NEWLibrary for implementing RNNs with Theano
License: BSD 3-Clause "New" or "Revised" License
Library for implementing RNNs with Theano
License: BSD 3-Clause "New" or "Revised" License
I want to train a simple shallow RNN with 1 hidden layer (like in Mikolov's work).
In this case I don't need an embedding layer and I would like to turn it off.
What is the right way to do it? I tried to do this:
rec_layer = rec(x, n_steps=x.shape[0],
init_state=h0*reset,
no_noise_bias=state['no_noise_bias'],
truncate_gradient=state['truncate_gradient'],
batch_size=1)
but it, obviously, didn't worrk.
I'm completely new at Teano so I didn't succeed in guessing it just looking through the code.
P.S. Obviously, it is not a bug and should not be here but I don't have another way of communication with you.
I have a trained model and tested on 1000 lines en-fr corpus, the beam search sample just fed me back less than 20 translations. In the line 60 of sample.py
: https://github.com/lisa-groundhog/GroundHog/blob/master/experiments/nmt/sample.py#L60
for k in range(3 * len(seq)):
Is there a specific reason we need to use this number of steps? Since I think it's not large enough for my model to generate results end with enc_dec.state['null_sym_target']
. And in my understanding the early truncation of beam search process does not make sense? Why do we stop it? Or just because my trained model is bad (so can't get correct output).
How to give data for training and testing? And what is the format in which we give data for training
I would like to use your good GroundHog to implement a sentences
segmentation task.
In order to understanf how GroundHog library works, I tried to run DT_RNN_Tut.py
But it says:
time python DT_RNN_Tut.py
Traceback (most recent call last):
File "DT_RNN_Tut.py", line 431, in
jobman(state, None)
File "DT_RNN_Tut.py", line 114, in jobman
train_data, valid_data, test_data = get_text_data(state)
File "DT_RNN_Tut.py", line 71,pentree_char_and_word.npz in get_text_data
can_fit=True)
File "/home/ubuntu/ggc/prove/DRNN/GroundHog-master/groundhog/datasets/LM_dataset.py",
line 97, in init
self.load_files()
File "/home/ubuntu/ggc/prove/DRNN/GroundHog-master/groundhog/datasets/LM_dataset.py",
line 105, in load_files
penn_data = numpy.load(self.path, mmap_mode=mmap_mode)
File "/usr/local/lib/python2.7/dist-packages/numpy/lib/npyio.py",
line 370, in load
fid = open(file, "rb")
IOError: [Errno 2] No such file or directory:
'/data/lisa/data/PennTreebankCorpus/pentree_char_and_word.npz'
What is pentree_char_and_word.npz?
And how to make DT_RNN_Tut.py working?
Looking forward to receive your kind helpfull hints.
Kind regards.
Marco
I am a researcher in the field of language modeling. I am trying to train an RNN-based language model. I take a train and valid sets (namely, ones included into Mikolov's RNNLM distribution). Then I: 1) Run generate.py and get .npz files.
2) In LM_dataset.py I comment out line 108:
self.penn_nchars = penn_data["n_chars"]
3) and set state['chunks'] = 'words'
4) To prevent TypeError I change all entries with float32 to float64 (e.g. in lines 105, 198).
Then, I run the DT_RNN_Tut.py
and get
data length is 81350
data length is 7090
data length is 8498
/usr/local/lib/python2.7/dist-packages/theano/scan_module/scan_perform_ext.py:85: RuntimeWarning: numpy.ndarray size changed, may indicate binary incompatibility
from scan_perform.scan_perform import *
/usr/local/lib/python2.7/dist-packages/theano/sandbox/rng_mrg.py:768: UserWarning: MRG_RandomStreams Can't determine #streams from size (Elemwise{Cast{int32}}.0), guessing 60*256
nstreams = self.n_streams(size)
Constructing grad function
Compiling grad function
took 42.7748370171
Validation computed every 1000
.. iter 0 cost 9.369 grad_norm 2.19e+00 traincost 1.35e+01 trainppl 1.17e+04 step time 0.343 sec whole time 45.245 sec lr 1.00e+00
Sample: Error in running algo (lr issue)
Took 0.108454414209 min
Traceback (most recent call last):
File "/opt/pycharm/pycharm-community-3.1.1/helpers/pydev/pydevd.py", line 1534, in
debugger.run(setup['file'], None, None)
File "/opt/pycharm/pycharm-community-3.1.1/helpers/pydev/pydevd.py", line 1145, in run
pydev_imports.execfile(file, globals, locals) #execute the script
File "scripts/DT_RNN_Tut.py", line 416, in
jobman(state, None)
File "scripts/DT_RNN_Tut.py", line 293, in jobman
main.main()
File "/home/mkudinov/workspace/GroundHog-master/groundhog/mainLoop.py", line 305, in main
[fn() for fn in self.hooks]
File "scripts/DT_RNN_Tut.py", line 260, in hook_fn
print dictionary[si],
IndexError: index out of bounds
index is set to the number exceeding the vocabulary size. At the same time the data is read properly so the error is made somewhere during the iteration.
Sorry, but I have a new question.
I'm confused with theano programming style. I can't figure out what data goes to the first layer of the net. I see that a matrix containing dictionary indeces is read at each iteration inside SGD_momentum.call(self) but I cannot find the moment when this matrix is separated into one-hot vectors and sent to the first layer. After compilation catching this moment is completely impossible.
I'm, afraid that the first layer computes the dot product between indeces and weight matrix. How can I assure that it is not this way? How and where the input vector is transformed into the valid first layer input?
I am a researcher in the field of language modeling. I am trying to train an RNN-based language model. I take a train and valid sets (namely, ones included into Mikolov's RNNLM distribution). Then I: 1) Run generate.py and get .npz files.
2) In LM_dataset.py I comment out line 108:
self.penn_nchars = penn_data["n_chars"]
3) and set state['chunks'] = 'words'
4) To prevent TypeError I change all entries with float32 to float64 (e.g. in lines 105, 198).
Then, I run the DT_RNN_Tut.py
and get
data length is 81350
data length is 7090
data length is 8498
/usr/local/lib/python2.7/dist-packages/theano/scan_module/scan_perform_ext.py:85: RuntimeWarning: numpy.ndarray size changed, may indicate binary incompatibility
from scan_perform.scan_perform import *
/usr/local/lib/python2.7/dist-packages/theano/sandbox/rng_mrg.py:768: UserWarning: MRG_RandomStreams Can't determine #streams from size (Elemwise{Cast{int32}}.0), guessing 60*256
nstreams = self.n_streams(size)
Constructing grad function
Compiling grad function
took 42.7748370171
Validation computed every 1000
.. iter 0 cost 9.369 grad_norm 2.19e+00 traincost 1.35e+01 trainppl 1.17e+04 step time 0.343 sec whole time 45.245 sec lr 1.00e+00
Sample: Error in running algo (lr issue)
Took 0.108454414209 min
Traceback (most recent call last):
File "/opt/pycharm/pycharm-community-3.1.1/helpers/pydev/pydevd.py", line 1534, in
debugger.run(setup['file'], None, None)
File "/opt/pycharm/pycharm-community-3.1.1/helpers/pydev/pydevd.py", line 1145, in run
pydev_imports.execfile(file, globals, locals) #execute the script
File "scripts/DT_RNN_Tut.py", line 416, in
jobman(state, None)
File "scripts/DT_RNN_Tut.py", line 293, in jobman
main.main()
File "/home/mkudinov/workspace/GroundHog-master/groundhog/mainLoop.py", line 305, in main
[fn() for fn in self.hooks]
File "scripts/DT_RNN_Tut.py", line 260, in hook_fn
print dictionary[si],
IndexError: index out of bounds
index is set to the number exceeding the vocabulary size. At the same time the data is read properly so the error is made somewhere during the iteration.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.