xinyadu / nqg Goto Github PK

In ParaSentDecoder, h_dim is not in self.args, thus it is not saved in the checkpoint - because only self.args is saved. Due to this, the shape of gradContextInput (https://github.com/xinyadu/nqg/blob/master/paragraph/onmt/modules/ParaSentDecoder.lua#L327) is incorrect and creates bad argument error on updating gradients using gradContextInput:add(gradInput[self.args.inputIndex.context]) (https://github.com/xinyadu/nqg/blob/master/paragraph/onmt/modules/ParaSentDecoder.lua#L354)

First let me compliment for the excellent code that you have written. It works, and easy to follow as well.

I had couple questions:

1. How did you prepare 70K question and answers?
2. In case of paragraph model, why we have to provide the sentence comprehension as well in training data. If needed, what is the basis of picking sentence for each paragraph comprehension for training data set?
3. Can I use the sentence model and increase the limit "src_seq_length=100" in "config-preprocess" file to accommodate multiple sentences instead of using paragraph model?

Appreciate your inputs.

Hi,
I was just wondering if you included the bash file in your repo, couldn't find it...
Thx!

Hi , thank you for your nice contribution!
I encounter problem in evaluation:
scores:

Traceback (most recent call last):
File "eval.py", line 101, in
eval(args.out_file, args.src_file, args.tgt_file)
File "eval.py", line 91, in eval
return QGEval.evaluate()
File "/home/zhaozhe/Desktop/opennmt/OpenNMT-py-master/qgevalcap/eval.py", line 24, in evaluate
(Meteor(),"METEOR"),
File "/home/zhaozhe/Desktop/opennmt/OpenNMT-py-master/qgevalcap/meteor/meteor.py", line 29, in init
stderr=subprocess.PIPE)
File "/usr/lib/python2.7/subprocess.py", line 711, in init
errread, errwrite)
File "/usr/lib/python2.7/subprocess.py", line 1343, in _execute_child
raise child_exception
OSError: [Errno 2] No such file or directory

I can not figure out why meteor can not be calculated

Hello. Thanks for sharing processed data. I just wonder how can I get the question_id for each sentence example from src-train.txt/src-dev.txt/src-test.txt? I would like to extract related answers from examples.

When i run "th translate.lua -model model/840B.300d.rnn.para_epoch15_26.70.t7 -config config-trans" command, I get this error, I don;t know how to solve it
===================Error============================================
THCudaCheck FAIL file=/home/beta/torch/extra/cutorch/lib/THC/THCGeneral.c line=70 error=38 : no CUDA-capable device is detected
/home/beta/torch/install/bin/luajit: ./onmt/utils/Cuda.lua:31: /home/beta/torch/install/share/lua/5.1/trepl/init.lua:389: cuda runtime error (38) : no CUDA-capable device is detected at /home/beta/torch/extra/cutorch/lib/THC/THCGeneral.c:70
stack traceback:
[C]: in function 'error'
./onmt/utils/Cuda.lua:31: in function 'init'
./onmt/translate/Translator.lua:31: in function 'init'
translate.lua:86: in function 'main'
translate.lua:239: in main chunk
[C]: in function 'dofile'
...beta/torch/install/lib/luarocks/rocks/trepl/scm-1/bin/th:150: in main chunk
[C]: at 0x55e7dcaee570

Hi, first of all, thank you for open sourcing the code base. In the README you mention that Paragraph-level model and sentence-level model will be made available soon.

Can you give a rough timeframe of when these models will be made available?

Is there the setting, which can define the list of unsplittable words.
E.g. I have a text about iPhone X. There is an info about Face ID, True Tone flash and so on.
When I gives this text to the model (trained on squad dataset or ms marco dataset ), it asks questions like this:
SENT 1: Face ID is a new authentification method....
PRED 1: What is Face (but we understand that question should be about Face ID)
SENT 2: It also has a quad-LED True Tone flash
PRED 2: What is Tone (instead of True Tone)

And the same situtions with other technology names.
I tried to and -phrase_table setting to translate config with file contains
Face ID ||| FaceID
and some other variation, but got no effect.

Thanks!

I have this issue while running th preprocess.lua -config config-preprocess

/usr/bin/luajit: /usr/share/lua/5.1/trepl/init.lua:389: /usr/share/lua/5.1/trepl/init.lua:389: /usr/share/lua/5.1/trepl/init.lua
:389: /usr/share/lua/5.1/trepl/init.lua:389: module 'nngraph' not found:
no field package.preload['nngraph']
no file './nngraph.lua'
no file '/usr/share/luajit-2.1.0-beta3/nngraph.lua'
no file '/usr/local/share/lua/5.1/nngraph.lua'
no file '/usr/local/share/lua/5.1/nngraph/init.lua'
no file '/usr/share/lua/5.1/nngraph.lua'
no file '/usr/share/lua/5.1/nngraph/init.lua'
no file './nngraph.so'
no file '/usr/local/lib/lua/5.1/nngraph.so'
no file '/usr/lib/x86_64-linux-gnu/lua/5.1/nngraph.so'
no file '/usr/local/lib/lua/5.1/loadall.so'
stack traceback:
[C]: in function 'error'
/usr/share/lua/5.1/trepl/init.lua:389: in function 'require'
preprocess.lua:1: in main chunk
[C]: in function 'dofile'
/usr/lib/torch-trepl/th:149: in main chunk
[C]: at 0x55bd871591d0

How do i fix it ?

Hi !
I found the research work that you did really interesting.
I think if there is a visual diagram of how your model is working, it would make it much easier to understand the implementation.

Thanks!

Can someone explain me which parameter should i change to run my code on multiple gpus?

mldl@mldlUB1604:~/ub16_prj/nqg/paragraph$ bash preprocess_embedding.sh
Traceback (most recent call last):
File "preprocess_embedding.py", line 76, in
main()
File "preprocess_embedding.py", line 52, in main
with open(args.dict, "r") as input_file:
IOError: [Errno 2] No such file or directory: 'data/qg.src.dict'

Traceback (most recent call last):
File "preprocess_embedding.py", line 76, in
main()
File "preprocess_embedding.py", line 52, in main
with open(args.dict, "r") as input_file:
IOError: [Errno 2] No such file or directory: 'data/qg.tgt.dict'
Traceback (most recent call last):
File "preprocess_embedding.py", line 76, in
main()
File "preprocess_embedding.py", line 52, in main
with open(args.dict, "r") as input_file:
IOError: [Errno 2] No such file or directory: 'data/qg.par.dict'

Can't seem to get passed this, any ideas?

/home/ubuntu/torch/install/bin/luajit: ./onmt/utils/Cuda.lua:31: ./onmt/utils/Cuda.lua:17: attempt to call field 'getKernelPeerToPeerAccess' (a nil value)
stack traceback:
[C]: in function 'error'
./onmt/utils/Cuda.lua:31: in function 'init'
train.lua:427: in function 'main'
train.lua:543: in main chunk
[C]: in function 'dofile'
...untu/torch/install/lib/luarocks/rocks/trepl/scm-1/bin/th:131: in main chunk
[C]: at 0x00406670

Using public AWS AMI Deep learning Torch7 and Caffe v3 (ami-3f77b354)

Thanks!

I don't have GPU machine. When I try to run the testing using the existing trained model. It given me error "cutorch" not installed. And for cutorch I need CUDA and for CUDA I need gpu machine. Is there a way around??

Is it possible to return answers as well?

I am trying to run this code from github on google colab.
To execute the code, i have to run the command "th preprocess.lua -config config-preprocess". Whenever I use this command on colab it says,

File "", line 1
th preprocess.lua -config config-preprocess
^
SyntaxError: invalid syntax

I tried installing Lua 5.2 on colab using the instructions in the link http://torch.ch/docs/getting-started.html .I ran the following commands
!git clone https://github.com/torch/distro.git ~/torch --recursive
cd ~/torch
!./clean.sh
!TORCH_LUA_VERSION=LUA52 ./install.sh
!luarocks install image
When I run the command !luarocks install image. It gives the error
/bin/bash: luarocks: command not found

This is the error I get when I try to train the model! Please help!

To generate new questions this line is used where config-trans specifies the input text:

$> th translate.lua -model model/<model file name> -config config-trans

1) What is the requirement of the input text? (stop words or any other requirements)

For sentence level model.
2) What qualifies as a good sentence for generating a question?

For paragraph level model.
3) On what basis should I split the paragraph into a sentence.

For sentence level model, I tried to generate questions by splitting input text into sentences based on . and gave the text file path in src field in config-trans.

For paragraph level model, I did the . based splitting for src file and for par file, I repeated the paragraph for each sentence in src.

Input text:
One of the most basic techniques of molecular biology to study protein function is molecular cloning. In this technique, DNA coding for a protein of interest is cloned using polymerase chain reaction (PCR), and/or restriction enzymes into a plasmid ( expression vector). A vector has 3 distinctive features: an origin of replication, a multiple cloning site (MCS), and a selective marker usually antibiotic resistance. Located upstream of the multiple cloning site are the promoter regions and the transcription start site which regulate the expression of cloned gene.

src
One of the most basic techniques of molecular biology to study protein function is molecular cloning.
In this technique, DNA coding for a protein of interest is cloned using polymerase chain reaction (PCR), and/or restriction enzymes into a plasmid ( expression vector).
A vector has 3 distinctive features: an origin of replication, a multiple cloning site (MCS), and a selective marker usually antibiotic resistance.
Located upstream of the multiple cloning site are the promoter regions and the transcription start site which regulate the expression of cloned gene.

Where am I doing it wrong?

Note: From the paper ;
DirectIn is an intuitive yet meaningful baseline in which the longest sub-sentence of the sentence is directly taken as the predicted question. To split the sentence into sub-sentences, we use a set of splitters, i.e. , {“?”, “!”, “,”, “.”, “;”}.

After doing:

python preprocess_embedding.py --embedding data/qg-train.t7 --dict data/qg.src.dict --output data/qg.src.840B.300d.npy

error:
Traceback (most recent call last):
File "preprocess_embedding.py", line 76, in
main()
File "preprocess_embedding.py", line 49, in main
word2embedding[line[0]] = np.asarray(map(float, line[1 : ]))
ValueError: invalid literal for float(): 0

when i ran the
python3 preprocess_embedding.py --embedding ./archive/embeddings/glov
e.840B.300d.txt --dict ./data/qg.src.dict --output ./data/qg.src.840b.300d.npy

after some time this process is being killed nothing is being saved, and my gcloud instance is not responding to any commands.

I could run the model on a single cpu by changing gpuid=0 in the config file. But I want to know if there is any few modifications to do in order to run the model on multiple cpus for faster training. Thank you!
PS: If you have a trained model for paragraph please do share it do that I don't run the training at all :) 💯

I am running the script for generating (.t7) files and i am facing the below issue:

$$  python preprocess_embedding.py --embedding /media/xxxx/NewVolume/glove.840B.300d.txt --dict ./data/qg.{src,tgt}.dict --output ./data/qg.{src,tgt}.840B.300d.npy
usage: preprocess_embedding.py [-h] --embedding EMBEDDING --dict DICT --output
                               OUTPUT [--seed SEED]

preprocess_embedding.py: error: unrecognized arguments: ./data/qg.tgt.dict ./data/qg.tgt.840B.300d.npy
t

I met the following error when I tried to process the paragraph level data. I have no idea about how to use Lua or correct. Could anyone give some advices?

preprocess.lua -config config-preprocess

Building source vocabulary...
Created dictionary of size 45004 (pruned from 71333)

Building target vocabulary...
Created dictionary of size 28003 (pruned from 35366)

Building prgrph vocabulary...
Created dictionary of size 45004 (pruned from 87180)

Preparing training data...
... shuffling sentences
... sorting sentences by size
/usr/bin/lua: preprocess.lua:162: bad argument #1 to 'pairs' (table expected, got userdata)
stack traceback:
[C]: in function 'pairs'
preprocess.lua:162: in function 'vecToTensor'
preprocess.lua:270: in function 'makeData'
preprocess.lua:321: in function 'main'
preprocess.lua:354: in main chunk
[C]: ?

tjebbe@tjebbe-VirtualBox:~/nqg/sentence$ th train.lua -config config-trainLoading data from 'data/qg-train.t7'...

• vocabulary size: source = 45004; target = 28003
• additional features: source = 0; target = 0
• maximum sequence length: source = 100; target = 50
• number of training sentences: 69979
• maximum batch size: 64
  Building model...
• using input feeding
  Initializing parameters...
• number of parameters: 26319403
  Preparing memory optimization...
• sharing 71% of output/gradInput tensors memory between clones
  Start training...

Epoch 1 ; Iteration 50/1140 ; Learning rate 1.0000 ; Source tokens/s 171 ; Perplexity 72871.79
Epoch 1 ; Iteration 100/1140 ; Learning rate 1.0000 ; Source tokens/s 175 ; Perplexity 27607.56
Epoch 1 ; Iteration 150/1140 ; Learning rate 1.0000 ; Source tokens/s 187 ; Perplexity 12133.82
Epoch 1 ; Iteration 200/1140 ; Learning rate 1.0000 ; Source tokens/s 181 ; Perplexity 6763.18
Epoch 1 ; Iteration 250/1140 ; Learning rate 1.0000 ; Source tokens/s 182 ; Perplexity 4378.42
Epoch 1 ; Iteration 300/1140 ; Learning rate 1.0000 ; Source tokens/s 183 ; Perplexity 3127.49
Epoch 1 ; Iteration 350/1140 ; Learning rate 1.0000 ; Source tokens/s 182 ; Perplexity 2385.68
Epoch 1 ; Iteration 400/1140 ; Learning rate 1.0000 ; Source tokens/s 181 ; Perplexity 1878.88
Epoch 1 ; Iteration 450/1140 ; Learning rate 1.0000 ; Source tokens/s 181 ; Perplexity 1532.97
Epoch 1 ; Iteration 500/1140 ; Learning rate 1.0000 ; Source tokens/s 180 ; Perplexity 1294.61
Epoch 1 ; Iteration 550/1140 ; Learning rate 1.0000 ; Source tokens/s 177 ; Perplexity 1122.80
Epoch 1 ; Iteration 600/1140 ; Learning rate 1.0000 ; Source tokens/s 175 ; Perplexity 984.16
Epoch 1 ; Iteration 650/1140 ; Learning rate 1.0000 ; Source tokens/s 173 ; Perplexity 874.13
Epoch 1 ; Iteration 700/1140 ; Learning rate 1.0000 ; Source tokens/s 173 ; Perplexity 789.38
Epoch 1 ; Iteration 750/1140 ; Learning rate 1.0000 ; Source tokens/s 174 ; Perplexity 718.86
Epoch 1 ; Iteration 800/1140 ; Learning rate 1.0000 ; Source tokens/s 174 ; Perplexity 661.67
Epoch 1 ; Iteration 850/1140 ; Learning rate 1.0000 ; Source tokens/s 175 ; Perplexity 612.37
Epoch 1 ; Iteration 900/1140 ; Learning rate 1.0000 ; Source tokens/s 175 ; Perplexity 572.65
Epoch 1 ; Iteration 950/1140 ; Learning rate 1.0000 ; Source tokens/s 177 ; Perplexity 536.55
Epoch 1 ; Iteration 1000/1140 ; Learning rate 1.0000 ; Source tokens/s 176 ; Perplexity 504.60
Epoch 1 ; Iteration 1050/1140 ; Learning rate 1.0000 ; Source tokens/s 177 ; Perplexity 476.61
Epoch 1 ; Iteration 1100/1140 ; Learning rate 1.0000 ; Source tokens/s 178 ; Perplexity 449.97
Validation perplexity: 119.12233583123
Saving checkpoint to 'model/840B.300d.600rnn_epoch1_119.12.t7'...
/home/tjebbe/torch/install/bin/luajit: cannot open <model/840B.300d.600rnn_epoch1_119.12.t7> in mode w at /home/tjebbe/torch/pkg/torch/lib/TH/THDiskFile.c:673
stack traceback:
[C]: at 0x7f248d4ac210
[C]: in function 'DiskFile'
/home/tjebbe/torch/install/share/lua/5.1/torch/File.lua:385: in function 'save'
./onmt/train/Checkpoint.lua:28: in function 'save'
./onmt/train/Checkpoint.lua:63: in function 'saveEpoch'
train.lua:415: in function 'trainModel'
train.lua:540: in function 'main'
train.lua:543: in main chunk
[C]: in function 'dofile'
...ebbe/torch/install/lib/luarocks/rocks/trepl/scm-1/bin/th:150: in main chunk

I did set it to just 1 epoch for testing (especially for errors after training so i dont waste my time as i dont have GPU). The embeddings file i used is 6B.100d instead of 840B.300d but that shouldn't be a problem. It is the sentence part and not the paragraph (haven't tested that one).

When running the command:
python preprocess_embedding.py --embedding <path to embedding txt file> --dict ./data/qg.{src,tgt}.dict --output ./data/qg.{src,tgt}.840B.300d.npy

I get the error "could not convert string to float" as in preprocess_embedding.py we convert the content of the embedding txt file (.txt) to float (line 50: word2embedding[line[0]] = np.asarray(map(float, line[1 : ])))

Am I missing something? The embedded file to give is a .txt file isn't it?

Hope anyone can help me with that. Thank you in advance.

Many thanks for your kind code sharing!

Could you provide the code to preprocess the data?
Or Could you give us your configurations to use CoreNLP?

Thanks again!

First replace in preprocess_embedding.sh with real path.Where is the embeddig txt ,I mean there are lots of txt in this program ,I don't know what should I put in there.

When I run the command
python preprocess_embedding.py --embedding glove.840B.300d.txt --dict ./data/qg.tgt.dict --output ./data/qg.tgt.840B.300d.npy

Error:
Unknown word:
Traceback (most recent call last):
File "preprocess_embedding.py", line 76, in
main()
File "preprocess_embedding.py", line 64, in main
embedding[i] = word2embedding[w]
TypeError: float() argument must be a string or a number, not 'map'

Any ideas?

hey everybody,
while trying to run the preprocessing part at the project,
at the comment: th convert.lua
there is an error : segmentation fault(core dumped).
from the line:
"array = npy4th.loadnpy("qg.src.840B.300d.npy")"
the reading of the file is fail.
we didn't solve it yet, may one of you can help us,
thanks.

Hello!
I had finished the above commands before the last evaluation.

I have two question:
1.I don't know how to key the last command
"./eval.py --out_file "
the output file must be created first?
and what the output file type should be created?
Can u give me a example?

I have train 15 epochs.t7 in Paragraph and Sentence
the command "th translate.lua -model model/ -config config-trans"
means i must to execute translate.lua in 30 times?

I am a rookie in the kind of model
I will be very grateful If you can answer me !

Hi! Thank you for opensourcing your qg work.

I tried running with default parameters and because I got errors when running qgevalcap, I evaluated with bleus with coco-caption :
https://github.com/XgDuan/coco-caption/
And got 34.79/19.04/12.29/8.44 for BLEU1~4, which is a bit behind scores reported in ACL paper.
I am not sure did I accidentally do something wrong since I am not familiar with lua codes, or is it just a different implementation/parameter with bleu scores (if there is)?

Sorry for very poor English. :/
Thank you very much!

Traceback (most recent call last):
  File "./eval.py", line 101, in <module>
    eval(args.out_file, args.src_file, args.tgt_file)
  File "./eval.py", line 72, in eval
    pair['prediction'] = output[idx]
IndexError: list index out of range

How do i fix this when iam running eval.py with an output file

Using a Jupiter notebook. running !th does work.

cd sentence

!th preprocess.lua -config config-preprocess

Building source vocabulary...
/Users/kevinice/torch/install/bin/lua: ./onmt/utils/FileReader.lua:4: ../data/processed/src-train.txt: No such file or directory
stack traceback:
[C]: in function 'assert'
./onmt/utils/FileReader.lua:4: in function '__init'
/Users/kevinice/torch/install/share/lua/5.2/torch/init.lua:91: in function 'new'
preprocess.lua:50: in function 'makeVocabulary'
preprocess.lua:124: in function 'initVocabulary'
preprocess.lua:276: in function 'main'
preprocess.lua:311: in main chunk
[C]: in function 'dofile'
...nice/torch/install/lib/luarocks/rocks/trepl/scm-1/bin/th:150: in main chunk
[C]: in ?

Can I get an ETA on the sentence level training dataset?

I am really happy seeing question generation work like this. However, i am not familiar with torch, is there any resource written in tensrflow that can reproduce your work?

We want to know which path did you replace at in preprocess_embedding.sh with real path?
thanks

In the paper, you've mentioned that the number of training data is 70484. But when I run the preprocessing, the number of pairs decreases because some of them have length more than 100. So the results that you reported on the paper are from the whole dataset or when you limit the source length to 100? Thanks

Hello!
When i run this code received followed error:
"th convert.lua"
/ home/my/torch/install/bin/lua: /home/my/torch/install/share/lua/5.2/trepl/init.lua:389: module 'npy4th' not found:No LuaRocks module found for npy4th
no field package.preload['npy4th']
no file '/home/my/.luarocks/share/lua/5.2/npy4th.lua'
no file '/home/my/.luarocks/share/lua/5.2/npy4th/init.lua'
no file '/home/my/torch/install/share/lua/5.2/npy4th.lua'
no file '/home/my/torch/install/share/lua/5.2/npy4th/init.lua'
no file '/home/my/code/torch/install/share/lua/5.2/npy4th.lua'
no file '/home/my/code/torch/install/share/lua/5.2/npy4th/init.lua'
no file '/home/my/code/torch/install/lib/lua/5.2/npy4th.lua'
no file '/home/my/code/torch/install/lib/lua/5.2/npy4th/init.lua'
no file './npy4th.lua'
no file '/home/my/.luarocks/lib/lua/5.2/npy4th.so'
no file '/home/my/torch/install/lib/lua/5.2/npy4th.so'
no file '/home/my/torch/install/lib/npy4th.so'
no file '/home/my/code/torch/install/lib/npy4th.so'
no file '/home/my/code/.luarocks/lib/lua/5.2/npy4th.so'
no file '/home/my/code/torch/install/lib/lua/5.2/npy4th.so'
no file '/home/my/code/torch/install/lib/lua/5.2/loadall.so'
no file './npy4th.so'
stack traceback:
[C]: in function 'error'
/home/my/torch/install/share/lua/5.2/trepl/init.lua:389: in function 'require'
convert.lua:1: in main chunk
[C]: in function 'dofile'
...aran/torch/install/lib/luarocks/rocks/trepl/scm-1/bin/th:150: in main chunk
[C]: in ?

Also i try to setup npy4th by this commands:
"git clone https://github.com/htwaijry/npy4th.git
cd npy4th
luarocks make"

but receive these errors:

cmake -E make_directory build;
cd build;
cmake .. -DCMAKE_BUILD_TYPE=Release -DCMAKE_PREFIX_PATH="/home/my/torch/install/bin/.." -DCMAKE_INSTALL_PREFIX="/home/my/torch/install/lib/luarocks/rocks/npy4th/1.3-3";
make

"-- Found Torch7 in /home/my/torch/install
-- Configuring done
-- Generating done
-- Build files have been written to: /home/my/npy4th/build
[ 50%] Built target cnpy
[ 75%] Building CXX object CMakeFiles/npy4th.dir/npy4th.cpp.o
/home/my/npy4th/npy4th.cpp: In function ‘void load_array_to_lua(lua_State*, cnpy::NpyArray&)’:
/home/my/npy4th/npy4th.cpp:37:33: warning: ‘const char* luaT_checktypename2id(lua_State*, const char*)’ is deprecated [-Wdeprecated-declarations]
luaT_pushudata(L, tensor, luaT_checktypename2id(L, "torch.FloatTensor"));
^
In file included from /home/my/npy4th/npy4th.cpp:12:0:
/home/my/torch/install/include/luaT.h:132:38: note: declared here
LUAT_API LUAT_DEPRECATED const char* luaT_checktypename2id(lua_State L, const
^
/home/my/npy4th/npy4th.cpp:37:33: warning: ‘const char luaT_checktypename2id(lua_State*, const char*)’ is deprecated [-Wdeprecated-declarations]
luaT_pushudata(L, tensor, luaT_checktypename2id(L, "torch.FloatTensor"));
^
In file included from /home/my/npy4th/npy4th.cpp:12:0:
/home/my/torch/install/include/luaT.h:132:38: note: declared here
LUAT_API LUAT_DEPRECATED const char* luaT_checktypename2id(lua_State L, const
^
/home/my/npy4th/npy4th.cpp:37:77: warning: ‘const char luaT_checktypename2id(lua_State*, const char*)’ is deprecated [-Wdeprecated-declarations]
luaT_pushudata(L, tensor, luaT_checktypename2id(L, "torch.FloatTensor"));
^
In file included from /home/my/npy4th/npy4th.cpp:12:0:
/home/my/torch/install/include/luaT.h:132:38: note: declared here
LUAT_API LUAT_DEPRECATED const char* luaT_checktypename2id(lua_State L, const
^
/home/my/npy4th/npy4th.cpp:43:33: warning: ‘const char luaT_checktypename2id(lua_State*, const char*)’ is deprecated [-Wdeprecated-declarations]
luaT_pushudata(L, tensor, luaT_checktypename2id(L, "torch.DoubleTensor"))
^
In file included from /home/my/npy4th/npy4th.cpp:12:0:
/home/my/torch/install/include/luaT.h:132:38: note: declared here
LUAT_API LUAT_DEPRECATED const char* luaT_checktypename2id(lua_State L, const
^
/home/my/npy4th/npy4th.cpp:43:33: warning: ‘const char luaT_checktypename2id(lua_State*, const char*)’ is deprecated [-Wdeprecated-declarations]
luaT_pushudata(L, tensor, luaT_checktypename2id(L, "torch.DoubleTensor"))
^
In file included from /home/my/npy4th/npy4th.cpp:12:0:
/home/my/torch/install/include/luaT.h:132:38: note: declared here
LUAT_API LUAT_DEPRECATED const char* luaT_checktypename2id(lua_State L, const
^
/home/my/npy4th/npy4th.cpp:43:78: warning: ‘const char luaT_checktypename2id(lua_State*, const char*)’ is deprecated [-Wdeprecated-declarations]
luaT_pushudata(L, tensor, luaT_checktypename2id(L, "torch.DoubleTensor"));
^
In file included from /home/my/npy4th/npy4th.cpp:12:0:
/home/my/torch/install/include/luaT.h:132:38: note: declared here
LUAT_API LUAT_DEPRECATED const char* luaT_checktypename2id(lua_State L, const
^
/home/my/npy4th/npy4th.cpp:50:33: warning: ‘const char luaT_checktypename2id(lua_State*, const char*)’ is deprecated [-Wdeprecated-declarations]
luaT_pushudata(L, tensor, luaT_checktypename2id(L, "torch.ByteTensor"));
^
In file included from /home/my/npy4th/npy4th.cpp:12:0:
/home/my/torch/install/include/luaT.h:132:38: note: declared here
LUAT_API LUAT_DEPRECATED const char* luaT_checktypename2id(lua_State L, const
^
/home/my/npy4th/npy4th.cpp:50:33: warning: ‘const char luaT_checktypename2id(lua_State*, const char*)’ is deprecated [-Wdeprecated-declarations]
luaT_pushudata(L, tensor, luaT_checktypename2id(L, "torch.ByteTensor"));
^
In file included from /home/my/npy4th/npy4th.cpp:12:0:
/home/my/torch/install/include/luaT.h:132:38: note: declared here
LUAT_API LUAT_DEPRECATED const char* luaT_checktypename2id(lua_State L, const
^
/home/my/npy4th/npy4th.cpp:50:76: warning: ‘const char luaT_checktypename2id(lua_State*, const char*)’ is deprecated [-Wdeprecated-declarations]
luaT_pushudata(L, tensor, luaT_checktypename2id(L, "torch.ByteTensor"));
^
In file included from /home/my/npy4th/npy4th.cpp:12:0:
/home/my/torch/install/include/luaT.h:132:38: note: declared here
LUAT_API LUAT_DEPRECATED const char* luaT_checktypename2id(lua_State L, const
^
/home/my/npy4th/npy4th.cpp:56:33: warning: ‘const char luaT_checktypename2id(lua_State*, const char*)’ is deprecated [-Wdeprecated-declarations]
luaT_pushudata(L, tensor, luaT_checktypename2id(L, "torch.ShortTensor"));
^
In file included from /home/my/npy4th/npy4th.cpp:12:0:
/home/my/torch/install/include/luaT.h:132:38: note: declared here
LUAT_API LUAT_DEPRECATED const char* luaT_checktypename2id(lua_State L, const
^
/home/my/npy4th/npy4th.cpp:56:33: warning: ‘const char luaT_checktypename2id(lua_State*, const char*)’ is deprecated [-Wdeprecated-declarations]
luaT_pushudata(L, tensor, luaT_checktypename2id(L, "torch.ShortTensor"));
^
In file included from /home/my/npy4th/npy4th.cpp:12:0:
/home/my/torch/install/include/luaT.h:132:38: note: declared here
LUAT_API LUAT_DEPRECATED const char* luaT_checktypename2id(lua_State L, const
^
/home/my/npy4th/npy4th.cpp:56:77: warning: ‘const char luaT_checktypename2id(lua_State*, const char*)’ is deprecated [-Wdeprecated-declarations]
luaT_pushudata(L, tensor, luaT_checktypename2id(L, "torch.ShortTensor"));
^
In file included from /home/my/npy4th/npy4th.cpp:12:0:
/home/my/torch/install/include/luaT.h:132:38: note: declared here
LUAT_API LUAT_DEPRECATED const char* luaT_checktypename2id(lua_State L, const
^
/home/my/npy4th/npy4th.cpp:62:33: warning: ‘const char luaT_checktypename2id(lua_State*, const char*)’ is deprecated [-Wdeprecated-declarations]
luaT_pushudata(L, tensor, luaT_checktypename2id(L, "torch.IntTensor"));
^
In file included from /home/my/npy4th/npy4th.cpp:12:0:
/home/my/torch/install/include/luaT.h:132:38: note: declared here
LUAT_API LUAT_DEPRECATED const char* luaT_checktypename2id(lua_State L, const
^
/home/my/npy4th/npy4th.cpp:62:33: warning: ‘const char luaT_checktypename2id(lua_State*, const char*)’ is deprecated [-Wdeprecated-declarations]
luaT_pushudata(L, tensor, luaT_checktypename2id(L, "torch.IntTensor"));
^
In file included from /home/my/npy4th/npy4th.cpp:12:0:
/home/my/torch/install/include/luaT.h:132:38: note: declared here
LUAT_API LUAT_DEPRECATED const char* luaT_checktypename2id(lua_State L, const
^
/home/my/npy4th/npy4th.cpp:62:75: warning: ‘const char luaT_checktypename2id(lua_State*, const char*)’ is deprecated [-Wdeprecated-declarations]
luaT_pushudata(L, tensor, luaT_checktypename2id(L, "torch.IntTensor"));
^
In file included from /home/my/npy4th/npy4th.cpp:12:0:
/home/my/torch/install/include/luaT.h:132:38: note: declared here
LUAT_API LUAT_DEPRECATED const char* luaT_checktypename2id(lua_State L, const
^
/home/my/npy4th/npy4th.cpp:68:33: warning: ‘const char luaT_checktypename2id(lua_State*, const char*)’ is deprecated [-Wdeprecated-declarations]
luaT_pushudata(L, tensor, luaT_checktypename2id(L, "torch.LongTensor"));
^
In file included from /home/my/npy4th/npy4th.cpp:12:0:
/home/my/torch/install/include/luaT.h:132:38: note: declared here
LUAT_API LUAT_DEPRECATED const char* luaT_checktypename2id(lua_State L, const
^
/home/my/npy4th/npy4th.cpp:68:33: warning: ‘const char luaT_checktypename2id(lua_State*, const char*)’ is deprecated [-Wdeprecated-declarations]
luaT_pushudata(L, tensor, luaT_checktypename2id(L, "torch.LongTensor"));
^
In file included from /home/my/npy4th/npy4th.cpp:12:0:
/home/my/torch/install/include/luaT.h:132:38: note: declared here
LUAT_API LUAT_DEPRECATED const char* luaT_checktypename2id(lua_State L, const
^
/home/my/npy4th/npy4th.cpp:68:76: warning: ‘const char luaT_checktypename2id(lua_State*, const char*)’ is deprecated [-Wdeprecated-declarations]
luaT_pushudata(L, tensor, luaT_checktypename2id(L, "torch.LongTensor"));
^
In file included from /home/my/npy4th/npy4th.cpp:12:0:
/home/my/torch/install/include/luaT.h:132:38: note: declared here
LUAT_API LUAT_DEPRECATED const char* luaT_checktypename2id(lua_State L, const
^
/home/my/npy4th/npy4th.cpp: At global scope:
/home/my/npy4th/npy4th.cpp:213:1: error: elements of array ‘const luaL_reg npyth []’ have incomplete type
};
^
/home/my/npy4th/npy4th.cpp:213:1: error: storage size of ‘npyth’ isn’t known
/home/my/npy4th/npy4th.cpp: In function ‘int luaopen_libnpy4th(lua_State)’:
/home/my/npy4th/npy4th.cpp:216:40: error: ‘luaL_openlib’ was not declared in this scope
luaL_openlib(L, "libnpy4th", npyth, 0);
^
CMakeFiles/npy4th.dir/build.make:62: recipe for target 'CMakeFiles/npy4th.dir/npy4th.cpp.o' failed
make[2]: *** [CMakeFiles/npy4th.dir/npy4th.cpp.o] Error 1
CMakeFiles/Makefile2:67: recipe for target 'CMakeFiles/npy4th.dir/all' failed
make[1]: *** [CMakeFiles/npy4th.dir/all] Error 2
Makefile:127: recipe for target 'all' failed
make: *** [all] Error 2

Error: Build error: Failed building."

How can I fix it and run this model?
Please help me.
Thanks very much.

Is perplexity the only metric which we can use to evaluate the training model or do we have any other metrics which we can include or try ?

@xinyadu

This error was occurring while running the "./preprocess_embeddings.sh" after which the rest of the code ran and out put was generated , but however due to the below error i was not able to successfully run the "th convert.lua" because of the below error did not generate a p[articular file which was required by the "th convert.lua" command

Can someone help me in this and i am running this in python2 virtual environment only

Traceback (most recent call last):
File "preprocess_embedding.py", line 76, in
main()
File "preprocess_embedding.py", line 49, in main
word2embedding[line[0]] = np.asarray(map(float, line[1 : ]))
ValueError: invalid literal for float(): 1

@Binathi