failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED about keras-retinanet

[hgaiser]

Are you sure your COCO directory is set up correctly?

[wanghuok]

yes. before update keras. it works all right.

[complicatedlee]

I also meet this problem, did u solve it ? thx. @wanghuok

[wanghuok]

@complicatedlee It is my fault. My coco path is not correct. but when i correct coco path. i get this error. my gpu is GTX 1060 with 6GB vedio memory.

loading annotations into memory...
Done (t=12.17s)
creating index...
index created!
loading annotations into memory...
Done (t=0.38s)
creating index...
index created!
Epoch 1/20
2017-11-08 17:41:09.261753: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED
2017-11-08 17:41:09.261779: W tensorflow/stream_executor/stream.cc:1901] attempting to perform BLAS operation using StreamExecutor without BLAS support
Traceback (most recent call last):
File "examples/train_coco.py", line 106, in
keras.callbacks.ReduceLROnPlateau(monitor='loss', factor=0.1, patience=2, verbose=1, mode='auto', epsilon=0.0001, cooldown=0, min_lr=0),
File "/home/wh/.local/lib/python2.7/site-packages/keras/legacy/interfaces.py", line 87, in wrapper
return func(*args, **kwargs)
File "/home/wh/.local/lib/python2.7/site-packages/keras/engine/training.py", line 2077, in fit_generator
class_weight=class_weight)
File "/home/wh/.local/lib/python2.7/site-packages/keras/engine/training.py", line 1797, in train_on_batch
outputs = self.train_function(ins)
File "/home/wh/.local/lib/python2.7/site-packages/keras/backend/tensorflow_backend.py", line 2332, in call
**self.session_kwargs)
File "/home/wh/.local/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 889, in run
run_metadata_ptr)
File "/home/wh/.local/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1120, in _run
feed_dict_tensor, options, run_metadata)
File "/home/wh/.local/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1317, in _do_run
options, run_metadata)
File "/home/wh/.local/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1336, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InternalError: Blas SGEMM launch failed : m=22500, n=256, k=64
[[Node: res2a_branch1/convolution = Conv2D[T=DT_FLOAT, data_format="NHWC", padding="VALID", strides=[1, 1, 1, 1], use_cudnn_on_gpu=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](pool1/MaxPool, res2a_branch1/kernel/read)]]
[[Node: loss/add/_2591 = _Recvclient_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_12603_loss/add", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]]

Caused by op u'res2a_branch1/convolution', defined at:
File "examples/train_coco.py", line 60, in
model = create_model(weights=args.weights)
File "examples/train_coco.py", line 38, in create_model
return ResNet50RetinaNet(image, num_classes=90, weights=weights)
File "/home/wh/gitmodel/keras-retinanet/keras_retinanet/models/resnet.py", line 36, in ResNet50RetinaNet
resnet = keras_resnet.models.ResNet50(image, include_top=False, freeze_bn=True)
File "/home/wh/.local/lib/python2.7/site-packages/keras_resnet/models/_2d.py", line 181, in ResNet50
return ResNet(inputs, blocks, block=keras_resnet.blocks.bottleneck_2d, include_top=include_top, classes=classes, *args, **kwargs)
File "/home/wh/.local/lib/python2.7/site-packages/keras_resnet/models/_2d.py", line 70, in ResNet
x = block(features, stage_id, block_id, numerical_name=(blocks[stage_id] > 6), freeze_bn=freeze_bn)(x)
File "/home/wh/.local/lib/python2.7/site-packages/keras_resnet/blocks/_2d.py", line 139, in f
shortcut = keras.layers.Conv2D(filters * 4, (1, 1), strides=stride, name="res{}{}_branch1".format(stage_char, block_char), **parameters)(x)
File "/home/wh/.local/lib/python2.7/site-packages/keras/engine/topology.py", line 603, in call
output = self.call(inputs, **kwargs)
File "/home/wh/.local/lib/python2.7/site-packages/keras/layers/convolutional.py", line 164, in call
dilation_rate=self.dilation_rate)
File "/home/wh/.local/lib/python2.7/site-packages/keras/backend/tensorflow_backend.py", line 3185, in conv2d
data_format=tf_data_format)
File "/home/wh/.local/lib/python2.7/site-packages/tensorflow/python/ops/nn_ops.py", line 751, in convolution
return op(input, filter)
File "/home/wh/.local/lib/python2.7/site-packages/tensorflow/python/ops/nn_ops.py", line 835, in call
return self.conv_op(inp, filter)
File "/home/wh/.local/lib/python2.7/site-packages/tensorflow/python/ops/nn_ops.py", line 499, in call
return self.call(inp, filter)
File "/home/wh/.local/lib/python2.7/site-packages/tensorflow/python/ops/nn_ops.py", line 187, in call
name=self.name)
File "/home/wh/.local/lib/python2.7/site-packages/tensorflow/python/ops/gen_nn_ops.py", line 631, in conv2d
data_format=data_format, name=name)
File "/home/wh/.local/lib/python2.7/site-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper
op_def=op_def)
File "/home/wh/.local/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 2956, in create_op
op_def=op_def)
File "/home/wh/.local/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 1470, in init
self._traceback = self._graph._extract_stack() # pylint: disable=protected-access

InternalError (see above for traceback): Blas SGEMM launch failed : m=22500, n=256, k=64
[[Node: res2a_branch1/convolution = Conv2D[T=DT_FLOAT, data_format="NHWC", padding="VALID", strides=[1, 1, 1, 1], use_cudnn_on_gpu=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](pool1/MaxPool, res2a_branch1/kernel/read)]]
[[Node: loss/add/_2591 = _Recvclient_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_12603_loss/add", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]]

[wanghuok]

@complicatedlee
The structure of your COCO dataset should be:

/path/to/COCO/
/path/to/COCO/annotations/
/path/to/COCO/images/train2017/
/path/to/COCO/images/val2017/

[complicatedlee]

maybe you should set your batchsize smaller

[complicatedlee]

@wanghuok

[complicatedlee]

I also find my problem, I just set the wrong path... thanks for your help~ @wanghuok

[wanghuok]

@complicatedlee You are welcome. My batch_size is already set to 1. Is it due to my less GPU memory?

[complicatedlee]

If you use anaconda, maybe you could update anaconda using the following command
conda update --all
then try again. I'm not sure this is useful for your problem... @wanghuok

[hgaiser]

I'm afraid 6gb might not be enough. You can try to reduce the size of the images further by modifying the image_min_side and image_max_side values in the generator.

[wanghuok]

I don't know what the problem is. but it works after i switched to python3.