2018-03-13 15:20:04.122000: I C:\tf_jenkins\workspace\rel-win\M\windows-gpu\PY\3
6\tensorflow\core\common_runtime\bfc_allocator.cc:680] Stats:
Limit: 1631711232
InUse: 1229069056
MaxInUse: 1631711232
NumAllocs: 51
MaxAllocSize: 422898176
2018-03-13 15:20:04.122000: W C:\tf_jenkins\workspace\rel-win\M\windows-gpu\PY\3
6\tensorflow\core\common_runtime\bfc_allocator.cc:279] _______________________
**************************************************************************x
2018-03-13 15:20:04.123000: W C:\tf_jenkins\workspace\rel-win\M\windows-gpu\PY\3
6\tensorflow\core\framework\op_kernel.cc:1202] OP_REQUIRES failed at assign_op.h
:111 : Resource exhausted: OOM when allocating tensor with shape[98304,1024] and
type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_b
fc
Traceback (most recent call last):
File "D:\Program Files (x86)\anaconda\lib\site-packages\tensorflow\python\clie
nt\session.py", line 1361, in _do_call
return fn(*args)
File "D:\Program Files (x86)\anaconda\lib\site-packages\tensorflow\python\clie
nt\session.py", line 1340, in _run_fn
target_list, status, run_metadata)
File "D:\Program Files (x86)\anaconda\lib\site-packages\tensorflow\python\fram
ework\errors_impl.py", line 516, in exit
c_api.TF_GetCode(self.status.status))
tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocat
ing tensor with shape[98304,1024] and type float on /job:localhost/replica:0/tas
k:0/device:GPU:0 by allocator GPU_0_bfc
[[Node: save/Assign_7 = Assign[T=DT_FLOAT, _class=["loc:@Variable_2"],
use_locking=true, validate_shape=true, _device="/job:localhost/replica:0/task:0/
device:GPU:0"](Variable_2/Adam, save/RestoreV2/_23)]]
Hint: If you want to see a list of allocated tensors when OOM happens, add repor
t_tensor_allocations_upon_oom to RunOptions for current allocation info.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "run_model.py", line 117, in
saver.restore(sess, "./model/model.ckpt")
File "D:\Program Files (x86)\anaconda\lib\site-packages\tensorflow\python\trai
ning\saver.py", line 1755, in restore
{self.saver_def.filename_tensor_name: save_path})
File "D:\Program Files (x86)\anaconda\lib\site-packages\tensorflow\python\clie
nt\session.py", line 905, in run
run_metadata_ptr)
File "D:\Program Files (x86)\anaconda\lib\site-packages\tensorflow\python\clie
nt\session.py", line 1137, in _run
feed_dict_tensor, options, run_metadata)
File "D:\Program Files (x86)\anaconda\lib\site-packages\tensorflow\python\clie
nt\session.py", line 1355, in _do_run
options, run_metadata)
File "D:\Program Files (x86)\anaconda\lib\site-packages\tensorflow\python\clie
nt\session.py", line 1374, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocat
ing tensor with shape[98304,1024] and type float on /job:localhost/replica:0/tas
k:0/device:GPU:0 by allocator GPU_0_bfc
[[Node: save/Assign_7 = Assign[T=DT_FLOAT, _class=["loc:@Variable_2"],
use_locking=true, validate_shape=true, _device="/job:localhost/replica:0/task:0/
device:GPU:0"](Variable_2/Adam, save/RestoreV2/_23)]]
Hint: If you want to see a list of allocated tensors when OOM happens, add repor
t_tensor_allocations_upon_oom to RunOptions for current allocation info.
Caused by op 'save/Assign_7', defined at:
File "run_model.py", line 113, in
saver=tf.train.Saver()
File "D:\Program Files (x86)\anaconda\lib\site-packages\tensorflow\python\trai
ning\saver.py", line 1293, in init
self.build()
File "D:\Program Files (x86)\anaconda\lib\site-packages\tensorflow\python\trai
ning\saver.py", line 1302, in build
self._build(self._filename, build_save=True, build_restore=True)
File "D:\Program Files (x86)\anaconda\lib\site-packages\tensorflow\python\trai
ning\saver.py", line 1339, in _build
build_save=build_save, build_restore=build_restore)
File "D:\Program Files (x86)\anaconda\lib\site-packages\tensorflow\python\trai
ning\saver.py", line 796, in _build_internal
restore_sequentially, reshape)
File "D:\Program Files (x86)\anaconda\lib\site-packages\tensorflow\python\trai
ning\saver.py", line 471, in _AddRestoreOps
assign_ops.append(saveable.restore(saveable_tensors, shapes))
File "D:\Program Files (x86)\anaconda\lib\site-packages\tensorflow\python\trai
ning\saver.py", line 161, in restore
self.op.get_shape().is_fully_defined())
File "D:\Program Files (x86)\anaconda\lib\site-packages\tensorflow\python\ops
state_ops.py", line 280, in assign
validate_shape=validate_shape)
File "D:\Program Files (x86)\anaconda\lib\site-packages\tensorflow\python\ops
gen_state_ops.py", line 61, in assign
use_locking=use_locking, name=name)
File "D:\Program Files (x86)\anaconda\lib\site-packages\tensorflow\python\fram
ework\op_def_library.py", line 787, in _apply_op_helper
op_def=op_def)
File "D:\Program Files (x86)\anaconda\lib\site-packages\tensorflow\python\fram
ework\ops.py", line 3271, in create_op
op_def=op_def)
File "D:\Program Files (x86)\anaconda\lib\site-packages\tensorflow\python\fram
ework\ops.py", line 1650, in init
self._traceback = self._graph._extract_stack() # pylint: disable=protected-
access
ResourceExhaustedError (see above for traceback): OOM when allocating tensor wit
h shape[98304,1024] and type float on /job:localhost/replica:0/task:0/device:GPU
:0 by allocator GPU_0_bfc
[[Node: save/Assign_7 = Assign[T=DT_FLOAT, _class=["loc:@Variable_2"],
use_locking=true, validate_shape=true, _device="/job:localhost/replica:0/task:0/
device:GPU:0"](Variable_2/Adam, save/RestoreV2/_23)]]
Hint: If you want to see a list of allocated tensors when OOM happens, add repor
t_tensor_allocations_upon_oom to RunOptions for current allocation info.