qaq-v / mmt Goto Github PK

after overcame many difficulties,eventually I managed to run train_mm.py
then I find there are too many bugs in translate_mm.py,can you just give me a runnable translate_mm.py file ,thank you!!!

I have followed the instuctions detailed at run.sh

But at this step,

python train_mm.py -data data/jpen -path_to_train_img_feats imgfeat/jpen_train_resnet50_cnn_features.hdf5 -path_to_valid_img_feats imgfeat/jpen_valid_resnet50_cnn_features.hdf5 -gpuid 0 -epochs 200 -layers 6 -rnn_size 300 -word_vec_size 512 -encoder_type transformer -decoder_type transformer -position_encoding -max_generator_batches 2 -dropout 0.1 -accum_count 1 -use_both -optim adam -adam_beta2 0.998 -decay_method noam -warmup_steps 8000 -learning_rate 2 -max_grad_norm 0 -param_init 0 -label_smoothing 0.1 --multimodal_model_type graphtransformer -batch_size 10 -save_model model/ADAM-jpen -pre_word_vecs_enc data/embeddings.enc.pt -pre_word_vecs_dec data/embeddings.dec.pt

I got the following error:

Traceback (most recent call last):
File "train_mm.py", line 447, in
main()
File "train_mm.py", line 444, in main
model_opt)
File "train_mm.py", line 246, in train_model
train_stats = trainer.train(train_iter, epoch, report_func)
File "/home/MMT-mainTF/onmt/TrainerMultimodal.py", line 122, in train
report_stats, normalization)
File "/home/MMT-mainTF/onmt/TrainerMultimodal.py", line 279, in _gradient_accumulation
self.model(src, tgt, src_lengths, img_feats, dec_state)
File "/home/vir-env/mnmtpy36/lib/python3.6/site-packages/torch/nn/modules/module.py", line 547, in call
result = self.forward(*input, **kwargs)
File "/home/MMT-mainTF/onmt/Models.py", line 1519, in forward
input_mm, context = self.encoder.forward_mm(src, img_proj, lengths)
ValueError: too many values to unpack (expected 2)

Could you please help in resolving this issue.

How do we get files like test_2016_flickr.lc.norm.tok.de ? I don't think the download_data script gets this. I also looked at the multi30k dataset (https://github.com/multi30k/dataset) , but can't seem to find the above file (the closest is test_2017_flickr.de).

i got so many problems about the functions of torchtext when i ran this code, so what is the true version of torchtext and pytorch that can run correctly?

I have run your code, but I got very low bleu score. Could you please provide some advice?
I followed the following steps:
1. download dataset to the path of data/raw

2. bash  preprocess.sh

3. bash word_embedding.sh

4. bash image_feature.sh

5. bash train.sh

6. bash test.sh

7. bash bleu.sh

And then i will get the result like this:

Could you please tell me your environment and how to get the correct bleu score? @QAQ-v
Look forward to your early reply.

I have followed the instuctions detailed at run.sh

But at this step,

python train_mm.py -data ${DATA_PATH}/demo -save_model ${MODEL_PATH}/ADAM
-path_to_train_img_feats ${DATA_PATH}/flickr30k_train_resnet50_cnn_features.hdf5
-path_to_valid_img_feats data/flickr30K/features/flickr30k_valid_resnet50_cnn_features.hdf5
-gpuid 0 -epochs 200 -layers 6 -rnn_size 300 -word_vec_size 512
-encoder_type transformer -decoder_type transformer -position_encoding
-max_generator_batches 2 -dropout 0.1
-batch_size 128 -accum_count 1 -use_both
-optim adam -adam_beta2 0.998 -decay_method noam -warmup_steps 8000 -learning_rate 2
-max_grad_norm 0 -param_init 0
-label_smoothing 0.1 --multimodal_model_type graphtransformer
-pre_word_vecs_enc ${DATA_PATH}/embeddings.enc.pt
-pre_word_vecs_dec ${DATA_PATH}/embeddings.dec.pt

I got the following error:

/home/young/anaconda3/envs/MMT_baseline/lib/python3.6/site-packages/torch/nn/modules/module.py:727: UserWarning: Implicit dimension choice for softmax has been deprecated. Change the call to include dim=X as an argument.
result = self.forward(*input, **kwargs)
Traceback (most recent call last):
File "train_mm.py", line 473, in
main()
File "train_mm.py", line 470, in main
model_opt)
File "train_mm.py", line 246, in train_model
train_stats = trainer.train(train_iter, epoch, report_func)
File "/home/young/Downloads/MMT-main/onmt/TrainerMultimodal.py", line 122, in train
report_stats, normalization)
File "/home/young/Downloads/MMT-main/onmt/TrainerMultimodal.py", line 279, in _gradient_accumulation
self.model(src, tgt, src_lengths, img_feats, dec_state)
File "/home/young/anaconda3/envs/MMT_baseline/lib/python3.6/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
result = self.forward(*input, **kwargs)
File "/home/young/Downloads/MMT-main/onmt/Models.py", line 1528, in forward
context_lengths=lengths)
File "/home/young/anaconda3/envs/MMT_baseline/lib/python3.6/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
result = self.forward(*input, **kwargs)
File "/home/young/Downloads/MMT-main/onmt/modules/Transformer.py", line 357, in forward
src_pad_mask, tgt_pad_mask)
File "/home/young/anaconda3/envs/MMT_baseline/lib/python3.6/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
result = self.forward(*input, **kwargs)
File "/home/young/Downloads/MMT-main/onmt/modules/Transformer.py", line 230, in forward
.expand_as(tgt_pad_mask), 0) # (batch, tgtlen, tgtlen)
RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!

Could you please help in resolving this issue.

I am getting this error while testing the pre trained model.

File "C:\Users\mayank.umrao\Documents\multimodal\MMT-main1\MMT-main\translate_mm.py", line 172, in
main()
File "C:\Users\mayank.umrao\Documents\multimodal\MMT-main1\MMT-main\translate_mm.py", line 136, in main
batch_data = translator.translate_batch(batch, data, sent_idx)
File "C:\Users\mayank.umrao\Documents\multimodal\MMT-main1\MMT-main\onmt\translate\TranslatorMultimodal.py", line 142, in translate_batch
img_proj = self.model.encoder_images(img_feats )
File "C:\Users\mayank.umrao\Miniconda3\envs\multi_final\lib\site-packages\torch\nn\modules\module.py", line 1130, in _call_impl
return forward_call(*input, **kwargs)
File "C:\Users\mayank.umrao\Documents\multimodal\MMT-main1\MMT-main\onmt\Models.py", line 720, in forward
out = self.layers(input)
File "C:\Users\mayank.umrao\Miniconda3\envs\multi_final\lib\site-packages\torch\nn\modules\module.py", line 1130, in _call_impl
return forward_call(*input, **kwargs)
File "C:\Users\mayank.umrao\Miniconda3\envs\multi_final\lib\site-packages\torch\nn\modules\container.py", line 139, in forward
input = module(input)
File "C:\Users\mayank.umrao\Miniconda3\envs\multi_final\lib\site-packages\torch\nn\modules\module.py", line 1130, in _call_impl
return forward_call(*input, **kwargs)
File "C:\Users\mayank.umrao\Documents\multimodal\MMT-main1\MMT-main\onmt\Models.py", line 682, in forward
return input.view(*self.shape)
RuntimeError: shape '[-1, 49, 2048]' is invalid for input of size 25088

I am unable to resolve this issue??