ResourceExhaustedError (see above for traceback): OOM when allocating tensor with shape[50,68,394054] about nmt-keras

[hiteshvaidya]

So, I googled a bit and found the solution. Tensorflow, keras by default takes all the available GPUs. If we want to avoid that then we can tweak the settings. Just for your information, following are the links to how that can be done,
keras-team/keras#6031
https://www.tensorflow.org/programmers_guide/using_gpu

Also, we can run the program as,
python <program_name> --num_gpus=<no._of_GPUs_u_want_2_allocate>
This will take the first 2 GPUs for execution as tensorflow considers GPUs in numeric order.

[hiteshvaidya]

Thank you so much for your help. I made following changes as per your suggestion and I am able to train.
MAX_INPUT_TEXT_LEN = 100
MAX_OUTPUT_TEXT_LEN = 100
INPUT_VOCABULARY_SIZE = 50000
OUTPUT_VOCABULARY_SIZE = 50000
BATCH_SIZE = 50

However, I didn't find any option for setting the parameters like gpuid or multiple gpuids etc. Please let me know if there is any way to put a control on the use of GPUs. The above experiment that I told was running on 3 GPUs but still showed 10 hours for completing just one epoch. I don't think it should take that much time because in the past I had run few experiments on OpenNMT backend using the same corpus and it didn't take so much time.

Thank you.

[lvapeab]

It seems you have a way too large vocabulary (394054 words!).

You must use a shortlist of the vocabulary (e.g. keep the top-K frequent words, mapping the rest to an unknown token). You can easily set this with the INPUT_VOCABULARY_SIZE and OUTPUT_VOCABULARY_SIZE options.

Moreover, the options MAX_INPUT_TEXT_LEN = 11351 MAX_OUTPUT_TEXT_LEN = 7898 indicate the max number of words allowed in each sentence. You probably want to reduce this number (to e.g. 80 words (this depends on your corpora).

Finally, I recommend you to have a look to sub-word preprocessing methods, such as byte-pair-encoding, which are effective when dealing with large vocabularies.

[lvapeab]

This is probably related to your Keras'/Tensorflow configuration.
I've never tried multi-gpu configuration, but on a single GPU, it should work.

[lvapeab]

Thanks for the info ^^