Training Start!
0%| | 0/43573 [00:00<?, ?it/s]
Traceback (most recent call last):
File "model/train.py", line 91, in <module>
solver.train()
File "/projects/da33/tao/test-project/neural_chat/model/utils/time_track.py", line 18, in timed
result = method(*args, **kwargs)
File "/projects/da33/tao/test-project/neural_chat/model/solver.py", line 905, in train
mode='train', kl_mult=kl_mult)
File "/projects/da33/tao/test-project/neural_chat/model/solver.py", line 1079, in _process_batch
extra_context_inputs=extra_context_inputs)
File "/projects/da33/tao/test-project/neural_chat/env/lib64/python3.6/site-packages/torch/nn/modules/module.py", line 491, in __call__
result = self.forward(*input, **kwargs)
File "/projects/da33/tao/test-project/neural_chat/model/models.py", line 361, in forward
sentence_length)
File "/projects/da33/tao/test-project/neural_chat/env/lib64/python3.6/site-packages/torch/nn/modules/module.py", line 491, in __call__
result = self.forward(*input, **kwargs)
File "/projects/da33/tao/test-project/neural_chat/model/layers/encoder.py", line 127, in forward
outputs, hidden = self.rnn(rnn_input, hidden)
File "/projects/da33/tao/test-project/neural_chat/env/lib64/python3.6/site-packages/torch/nn/modules/module.py", line 491, in __call__
result = self.forward(*input, **kwargs)
File "/projects/da33/tao/test-project/neural_chat/env/lib64/python3.6/site-packages/torch/nn/modules/rnn.py", line 192, in forward
output, hidden = func(input, self.all_weights, hx, batch_sizes)
File "/projects/da33/tao/test-project/neural_chat/env/lib64/python3.6/site-packages/torch/nn/_functions/rnn.py", line 323, in forward
return func(input, *fargs, **fkwargs)
File "/projects/da33/tao/test-project/neural_chat/env/lib64/python3.6/site-packages/torch/nn/_functions/rnn.py", line 275, in forward
train, dropout_seed, dropout_state)
File "/projects/da33/tao/test-project/neural_chat/env/lib64/python3.6/site-packages/torch/backends/cudnn/rnn.py", line 44, in init_dropout_state
if dropout_p != 0 else None
RuntimeError: CUDNN_STATUS_EXECUTION_FAILED
This problem spends me four days. Do you have any idea how to solve it?