Traceback (most recent call last):
File "C:\Users\47263\Anaconda3\lib\site-packages\tensorflow\python\client\session.py", line 1322, in _do_call
return fn(*args)
File "C:\Users\47263\Anaconda3\lib\site-packages\tensorflow\python\client\session.py", line 1307, in _run_fn
options, feed_dict, fetch_list, target_list, run_metadata)
File "C:\Users\47263\Anaconda3\lib\site-packages\tensorflow\python\client\session.py", line 1409, in _call_tf_sessionrun
run_metadata)
tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor with shape[32,4096] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
[[Node: bidirectional_rnn/fw/fw/while/fw/multi_rnn_cell/cell_1/basic_lstm_cell/MatMul = MatMul[T=DT_FLOAT, transpose_a=false, transpose_b=false, _device="/job:localhost/replica:0/task:0/device:GPU:0"](bidirectional_rnn/fw/fw/while/fw/multi_rnn_cell/cell_1/basic_lstm_cell/concat, bidirectional_rnn/fw/fw/while/fw/multi_rnn_cell/cell_1/basic_lstm_cell/MatMul/Enter)]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
[[Node: decoder/while/BasicDecoderStep/decoder/attention_wrapper/assert_equal/All/_129 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_862_d..._equal/All", tensor_type=DT_BOOL, _device="/job:localhost/replica:0/task:0/device:CPU:0"](^_cloopdecoder/while/BasicDecoderStep/decoder/attention_wrapper/assert_equal/Assert/Assert/data_0/_5)]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "D:/CODE/project/tensorflow/seq2seq-couplet/couplet.py", line 15, in
m.train(5000000)
File "D:\CODE\project\tensorflow\seq2seq-couplet\model.py", line 149, in train
self.train_target_seq_len: target_seq_len})
File "C:\Users\47263\Anaconda3\lib\site-packages\tensorflow\python\client\session.py", line 900, in run
run_metadata_ptr)
File "C:\Users\47263\Anaconda3\lib\site-packages\tensorflow\python\client\session.py", line 1135, in _run
feed_dict_tensor, options, run_metadata)
File "C:\Users\47263\Anaconda3\lib\site-packages\tensorflow\python\client\session.py", line 1316, in _do_run
run_metadata)
File "C:\Users\47263\Anaconda3\lib\site-packages\tensorflow\python\client\session.py", line 1335, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor with shape[32,4096] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
[[Node: bidirectional_rnn/fw/fw/while/fw/multi_rnn_cell/cell_1/basic_lstm_cell/MatMul = MatMul[T=DT_FLOAT, transpose_a=false, transpose_b=false, _device="/job:localhost/replica:0/task:0/device:GPU:0"](bidirectional_rnn/fw/fw/while/fw/multi_rnn_cell/cell_1/basic_lstm_cell/concat, bidirectional_rnn/fw/fw/while/fw/multi_rnn_cell/cell_1/basic_lstm_cell/MatMul/Enter)]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
[[Node: decoder/while/BasicDecoderStep/decoder/attention_wrapper/assert_equal/All/_129 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_862_d..._equal/All", tensor_type=DT_BOOL, _device="/job:localhost/replica:0/task:0/device:CPU:0"](^_cloopdecoder/while/BasicDecoderStep/decoder/attention_wrapper/assert_equal/Assert/Assert/data_0/_5)]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
Caused by op 'bidirectional_rnn/fw/fw/while/fw/multi_rnn_cell/cell_1/basic_lstm_cell/MatMul', defined at:
File "D:/CODE/project/tensorflow/seq2seq-couplet/couplet.py", line 13, in
restore_model=False)
File "D:\CODE\project\tensorflow\seq2seq-couplet\model.py", line 45, in init
self._init_train()
File "D:\CODE\project\tensorflow\seq2seq-couplet\model.py", line 72, in _init_train
self.num_units, self.layers, self.dropout)
File "D:\CODE\project\tensorflow\seq2seq-couplet\seq2seq.py", line 129, in seq2seq
num_units, layers, input_keep_prob)
File "D:\CODE\project\tensorflow\seq2seq-couplet\seq2seq.py", line 26, in bi_encoder
time_major = False)
File "C:\Users\47263\Anaconda3\lib\site-packages\tensorflow\python\ops\rnn.py", line 403, in bidirectional_dynamic_rnn
time_major=time_major, scope=fw_scope)
File "C:\Users\47263\Anaconda3\lib\site-packages\tensorflow\python\ops\rnn.py", line 618, in dynamic_rnn
dtype=dtype)
File "C:\Users\47263\Anaconda3\lib\site-packages\tensorflow\python\ops\rnn.py", line 815, in _dynamic_rnn_loop
swap_memory=swap_memory)
File "C:\Users\47263\Anaconda3\lib\site-packages\tensorflow\python\ops\control_flow_ops.py", line 3209, in while_loop
result = loop_context.BuildLoop(cond, body, loop_vars, shape_invariants)
File "C:\Users\47263\Anaconda3\lib\site-packages\tensorflow\python\ops\control_flow_ops.py", line 2941, in BuildLoop
pred, body, original_loop_vars, loop_vars, shape_invariants)
File "C:\Users\47263\Anaconda3\lib\site-packages\tensorflow\python\ops\control_flow_ops.py", line 2878, in _BuildLoop
body_result = body(*packed_vars_for_body)
File "C:\Users\47263\Anaconda3\lib\site-packages\tensorflow\python\ops\control_flow_ops.py", line 3179, in
body = lambda i, lv: (i + 1, orig_body(*lv))
File "C:\Users\47263\Anaconda3\lib\site-packages\tensorflow\python\ops\rnn.py", line 784, in _time_step
skip_conditionals=True)
File "C:\Users\47263\Anaconda3\lib\site-packages\tensorflow\python\ops\rnn.py", line 239, in _rnn_step
new_output, new_state = call_cell()
File "C:\Users\47263\Anaconda3\lib\site-packages\tensorflow\python\ops\rnn.py", line 772, in
call_cell = lambda: cell(input_t, state)
File "C:\Users\47263\Anaconda3\lib\site-packages\tensorflow\python\ops\rnn_cell_impl.py", line 232, in call
return super(RNNCell, self).call(inputs, state)
File "C:\Users\47263\Anaconda3\lib\site-packages\tensorflow\python\layers\base.py", line 329, in call
outputs = super(Layer, self).call(inputs, *args, **kwargs)
File "C:\Users\47263\Anaconda3\lib\site-packages\tensorflow\python\keras\engine\base_layer.py", line 703, in call
outputs = self.call(inputs, *args, **kwargs)
File "C:\Users\47263\Anaconda3\lib\site-packages\tensorflow\python\ops\rnn_cell_impl.py", line 1325, in call
cur_inp, new_state = cell(cur_inp, cur_state)
File "C:\Users\47263\Anaconda3\lib\site-packages\tensorflow\python\ops\rnn_cell_impl.py", line 1126, in call
output, new_state = self._cell(inputs, state, scope=scope)
File "C:\Users\47263\Anaconda3\lib\site-packages\tensorflow\python\ops\rnn_cell_impl.py", line 339, in call
*args, **kwargs)
File "C:\Users\47263\Anaconda3\lib\site-packages\tensorflow\python\layers\base.py", line 329, in call
outputs = super(Layer, self).call(inputs, *args, **kwargs)
File "C:\Users\47263\Anaconda3\lib\site-packages\tensorflow\python\keras\engine\base_layer.py", line 703, in call
outputs = self.call(inputs, *args, **kwargs)
File "C:\Users\47263\Anaconda3\lib\site-packages\tensorflow\python\ops\rnn_cell_impl.py", line 638, in call
array_ops.concat([inputs, h], 1), self._kernel)
File "C:\Users\47263\Anaconda3\lib\site-packages\tensorflow\python\ops\math_ops.py", line 2014, in matmul
a, b, transpose_a=transpose_a, transpose_b=transpose_b, name=name)
File "C:\Users\47263\Anaconda3\lib\site-packages\tensorflow\python\ops\gen_math_ops.py", line 4567, in mat_mul
name=name)
File "C:\Users\47263\Anaconda3\lib\site-packages\tensorflow\python\framework\op_def_library.py", line 787, in _apply_op_helper
op_def=op_def)
File "C:\Users\47263\Anaconda3\lib\site-packages\tensorflow\python\framework\ops.py", line 3414, in create_op
op_def=op_def)
File "C:\Users\47263\Anaconda3\lib\site-packages\tensorflow\python\framework\ops.py", line 1740, in init
self._traceback = self._graph._extract_stack() # pylint: disable=protected-access
ResourceExhaustedError (see above for traceback): OOM when allocating tensor with shape[32,4096] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
[[Node: bidirectional_rnn/fw/fw/while/fw/multi_rnn_cell/cell_1/basic_lstm_cell/MatMul = MatMul[T=DT_FLOAT, transpose_a=false, transpose_b=false, _device="/job:localhost/replica:0/task:0/device:GPU:0"](bidirectional_rnn/fw/fw/while/fw/multi_rnn_cell/cell_1/basic_lstm_cell/concat, bidirectional_rnn/fw/fw/while/fw/multi_rnn_cell/cell_1/basic_lstm_cell/MatMul/Enter)]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
[[Node: decoder/while/BasicDecoderStep/decoder/attention_wrapper/assert_equal/All/_129 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_862_d..._equal/All", tensor_type=DT_BOOL, _device="/job:localhost/replica:0/task:0/device:CPU:0"](^_cloopdecoder/while/BasicDecoderStep/decoder/attention_wrapper/assert_equal/Assert/Assert/data_0/_5)]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.