Hello,

I am trying to train SSDlite mobilenet v2 following your tutorial but I am stuck withi this error :

INFO:tensorflow:Starting Queues.
2018-06-05 13:29:49.582518: W tensorflow/core/framework/op_kernel.cc:1318] OP_REQUIRES failed at queue_ops.cc:105 : Invalid argument: Shape mismatch in tuple component 16. Expected [1,?,?,3], got [1,381,638,4]
INFO:tensorflow:Error reported to Coordinator: <class 'tensorflow.python.framework.errors_impl.InvalidArgumentError'>, Shape mismatch in tuple component 16. Expected [1,?,?,3], got [1,381,638,4]
	 [[Node: batch/padding_fifo_queue_enqueue = QueueEnqueueV2[Tcomponents=[DT_STRING, DT_INT32, DT_FLOAT, DT_INT32, DT_FLOAT, ..., DT_INT32, DT_INT32, DT_INT32, DT_STRING, DT_INT32], timeout_ms=-1, _device="/job:localhost/replica:0/task:0/device:CPU:0"](batch/padding_fifo_queue, IteratorGetNext, Shape_3, IteratorGetNext:1, Shape_5, Merge_1, Shape_4, Merge_2, Shape_10, IteratorGetNext:4, Shape_7, IteratorGetNext:5, Shape_8, IteratorGetNext:6, Shape_6, IteratorGetNext:7, Shape_9, ExpandDims_1, Shape, IteratorGetNext:9, Shape_3, IteratorGetNext:10, Shape_3, IteratorGetNext:11, Shape_3)]]
2018-06-05 13:29:49.704473: W tensorflow/core/framework/op_kernel.cc:1318] OP_REQUIRES failed at queue_ops.cc:105 : Invalid argument: Shape mismatch in tuple component 16. Expected [1,?,?,3], got [1,495,450,4]
2018-06-05 13:29:49.707130: W tensorflow/core/framework/op_kernel.cc:1318] OP_REQUIRES failed at queue_ops.cc:105 : Invalid argument: Shape mismatch in tuple component 16. Expected [1,?,?,3], got [1,426,640,4]
2018-06-05 13:29:49.709755: W tensorflow/core/framework/op_kernel.cc:1318] OP_REQUIRES failed at queue_ops.cc:105 : Invalid argument: Shape mismatch in tuple component 16. Expected [1,?,?,3], got [1,145,179,4]
2018-06-05 13:29:49.741023: W tensorflow/core/framework/op_kernel.cc:1318] OP_REQUIRES failed at queue_ops.cc:105 : Invalid argument: Shape mismatch in tuple component 16. Expected [1,?,?,3], got [1,400,431,4]
2018-06-05 13:29:49.743633: W tensorflow/core/framework/op_kernel.cc:1318] OP_REQUIRES failed at queue_ops.cc:105 : Invalid argument: Shape mismatch in tuple component 16. Expected [1,?,?,3], got [1,150,229,4]

Did someone get this error and solve it ? I check if I forgot some parameters but it does not seems the case. My images are all in .jpg formats.

EDIT : I tried to use frcnn resnet50 but I have some errors too regarding the softmax function.

EDIT 2 : It seems the problem comes from the function we modified in tf.example_decoder.py, even with 3 channels it doesn't work

background = cv2.imread('path/to/background')[:, :, ::-1]
i know the basic way to train object detection api using test and train folder but from where this background path comes into picture can you please explain more briefly?

Please switch to tf.train.MonitoredTrainingSession
2018-06-13 17:12:00.050617: I tensorflow/core/platform/cpu_feature_guard.cc:137] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2
INFO:tensorflow:Restoring parameters from faster_rcnn_resnet101_kitti_2018_01_28_edit/model.ckpt
INFO:tensorflow:Starting Session.
INFO:tensorflow:Saving checkpoint to path training_dir/Training8_FRCNN/model.ckpt
INFO:tensorflow:Starting Queues.
INFO:tensorflow:global_step/sec: 0
INFO:tensorflow:Error reported to Coordinator: <class 'tensorflow.python.framework.errors_impl.InvalidArgumentError'>, Input to reshape is a tensor with 14336000 values, but the requested shape has 3584000
[[Node: Reshape_9 = Reshape[T=DT_UINT8, Tshape=DT_INT32, _device="/job:localhost/replica:0/task:0/device:CPU:0"](DecodeRaw, Reshape_9/shape)]]
INFO:tensorflow:Caught OutOfRangeError. Stopping Training.
INFO:tensorflow:Finished training! Saving model to disk.
Traceback (most recent call last):
File "train.py", line 202, in
tf.app.run()
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/platform/app.py", line 124, in run
_sys.exit(main(argv))
File "train.py", line 198, in main
worker_job_name, is_chief, FLAGS.train_dir)
File "/media/ssd/JupyterHubNotebooks/calebbowyer/models/object_detection/trainer.py", line 296, in train
saver=saver)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/contrib/slim/python/slim/learning.py", line 782, in train
ignore_live_threads=ignore_live_threads)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/training/supervisor.py", line 826, in stop
ignore_live_threads=ignore_live_threads)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/training/coordinator.py", line 387, in join
six.reraise(*self._exc_info_to_raise)
File "/usr/local/lib/python3.5/dist-packages/six.py", line 693, in reraise
raise value
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/training/queue_runner_impl.py", line 250, in _run
enqueue_callable()
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1251, in _single_operation_run
self._session, None, {}, [], target_list, status, None)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/errors_impl.py", line 473, in exit
c_api.TF_GetCode(self.status.status))
tensorflow.python.framework.errors_impl.InvalidArgumentError: Input to reshape is a tensor with 14336000 values, but the requested shape has 3584000
[[Node: Reshape_9 = Reshape[T=DT_UINT8, Tshape=DT_INT32, _device="/job:localhost/replica:0/task:0/device:CPU:0"](DecodeRaw, Reshape_9/shape)]]

I have this new error that leads me to think I am close to training with 1 depth channel.
I have modified the convolutional weight in the first layer as follow:
var_to_edit_names = ['FirstStageFeatureExtractor/resnet_v1_101/conv1/weights']

[7, 7, 3, 64] --> [7,7,1,64] And that seemed to work, but for some reason it's saying my matrix has 4 times the number of elements than it actually does. (3584000 * 4) = 14336000.

I would really appreciate some help or insight with this one!

I have followed the procedure you gave and I successfully convert my ten channel images to tfrecored file. But I faced the following problem while I am trying to train it. FYI, I am using SSD_inception_V2., CUDA 8,tensorflow-gpu 1.3.0.
I tensorflow/core/common_runtime/simple_placer.cc:669] Ignoring device specification /GPU:0 for node 'prefetch_queue_Dequeue' because the input edge from 'prefetch_queue' is a reference connection and already has a device field set to /CPU:0
INFO:tensorflow:Starting Session.
INFO:tensorflow:Starting Queues.
INFO:tensorflow:global_step/sec: 0
INFO:tensorflow:Error reported to Coordinator: <class 'tensorflow.python.framework.errors_impl.InvalidArgumentError'>, Input to reshape is a tensor with 24576000 values, but the requested shape has 3072000
[[Node: Reshape_9 = Reshape[T=DT_UINT8, Tshape=DT_INT32, _device="/job:localhost/replica:0/task:0/cpu:0"](DecodeRaw, Cast)]]

Caused by op u'Reshape_9', defined at:
File "object_detection/train.py", line 201, in
tf.app.run()
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/platform/app.py", line 44, in run
_sys.exit(main(_sys.argv[:1] + flags_passthrough))
File "object_detection/train.py", line 195, in main
worker_job_name, is_chief, FLAGS.train_dir)
File "/mnt/disk1/fish/phase2/models/object_detection/trainer.py", line 185, in train
data_augmentation_options)
File "/mnt/disk1/fish/phase2/models/object_detection/trainer.py", line 59, in _create_input_queue
tensor_dict = create_tensor_dict_fn()
File "/mnt/disk1/fish/phase2/models/object_detection/builders/input_reader_builder.py", line 63, in build
return tf_example_decoder.TfExampleDecoder().decode(string_tensor)
File "/mnt/disk1/fish/phase2/models/object_detection/data_decoders/tf_example_decoder.py", line 139, in decode
tensors = decoder.decode(serialized_example, items=keys)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/slim/python/slim/data/tfexample_decoder.py", line 418, in decode
outputs.append(handler.tensors_to_item(keys_to_tensors))
File "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/slim/python/slim/data/tfexample_decoder.py", line 97, in tensors_to_item
return self._func(keys_to_tensors)
File "/mnt/disk1/fish/phase2/models/object_detection/data_decoders/tf_example_decoder.py", line 99, in _read_image
image = tf.reshape(tf.decode_raw(image_encoded, tf.uint8), to_shape)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/gen_array_ops.py", line 2630, in reshape
name=name)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/op_def_library.py", line 763, in apply_op
op_def=op_def)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 2395, in create_op
original_op=self._default_original_op, op_def=op_def)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 1264, in init
self._traceback = _extract_stack()

InvalidArgumentError (see above for traceback): Input to reshape is a tensor with 24576000 values, but the requested shape has 3072000
[[Node: Reshape_9 = Reshape[T=DT_UINT8, Tshape=DT_INT32, _device="/job:localhost/replica:0/task:0/cpu:0"](DecodeRaw, Cast)]]

Hi,

I am trying to get this running for a 4-channel image but keep getting the same error when trying to train:

INFO:tensorflow:Scale of 0 disables regularizer.
INFO:tensorflow:Scale of 0 disables regularizer.
WARNING:tensorflow:From build/bdist.linux-x86_64/egg/object_detection/trainer.py:210: create_global_step (from tensorflow.contrib.framework.python.ops.variables) is deprecated and will be removed in a future version.
Instructions for updating:
Please switch to tf.train.create_global_step
INFO:tensorflow:Scale of 0 disables regularizer.
INFO:tensorflow:Scale of 0 disables regularizer.
INFO:tensorflow:Scale of 0 disables regularizer.
INFO:tensorflow:depth of additional conv before box predictor: 0
INFO:tensorflow:Scale of 0 disables regularizer.
WARNING:tensorflow:From build/bdist.linux-x86_64/egg/object_detection/meta_architectures/faster_rcnn_meta_arch.py:1670: get_or_create_global_step (from tensorflow.contrib.framework.python.ops.variables) is deprecated and will be removed in a future version.
Instructions for updating:
Please switch to tf.train.get_or_create_global_step
INFO:tensorflow:Summary name /clone_loss is illegal; using clone_loss instead.
/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/gradients_impl.py:96: UserWarning: Converting sparse IndexedSlices to a dense Tensor of unknown shape. This may consume a large amount of memory.
  "Converting sparse IndexedSlices to a dense Tensor of unknown shape. "
2018-01-19 18:10:08.846043: I tensorflow/core/platform/cpu_feature_guard.cc:137] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 FMA
INFO:tensorflow:Restoring parameters from /home/david/data/datasets/multi_channel/tilbury/tensorflow/train/model.ckpt-0
INFO:tensorflow:Starting Session.
INFO:tensorflow:Saving checkpoint to path /home/david/data/datasets/multi_channel/tilbury/tensorflow/train/model.ckpt
INFO:tensorflow:Starting Queues.
INFO:tensorflow:global_step/sec: 0
INFO:tensorflow:Error reported to Coordinator: <class 'tensorflow.python.framework.errors_impl.InvalidArgumentError'>, assertion failed: [Unable to decode bytes as JPEG, PNG, GIF, or BMP]
	 [[Node: case/If_0/decode_image/cond_jpeg/cond_png/cond_gif/Assert_1/Assert = Assert[T=[DT_STRING], summarize=3, _device="/job:localhost/replica:0/task:0/device:CPU:0"](case/If_0/decode_image/cond_jpeg/cond_png/cond_gif/is_bmp, case/If_0/decode_image/cond_jpeg/cond_png/cond_gif/Assert_1/Assert/data_0)]]
INFO:tensorflow:Caught OutOfRangeError. Stopping Training.
INFO:tensorflow:Finished training! Saving model to disk.
Traceback (most recent call last):
  File "object_detection/train.py", line 198, in <module>
    tf.app.run()
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/platform/app.py", line 48, in run
    _sys.exit(main(_sys.argv[:1] + flags_passthrough))
  File "object_detection/train.py", line 194, in main
    worker_job_name, is_chief, FLAGS.train_dir)
  File "build/bdist.linux-x86_64/egg/object_detection/trainer.py", line 332, in train
    
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/slim/python/slim/learning.py", line 775, in train
    sv.stop(threads, close_summary_writer=True)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/supervisor.py", line 792, in stop
    stop_grace_period_secs=self._stop_grace_secs)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/coordinator.py", line 389, in join
    six.reraise(*self._exc_info_to_raise)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/queue_runner_impl.py", line 238, in _run
    enqueue_callable()
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 1231, in _single_operation_run
    target_list_as_strings, status, None)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/errors_impl.py", line 473, in __exit__
    c_api.TF_GetCode(self.status.status))
tensorflow.python.framework.errors_impl.InvalidArgumentError: assertion failed: [Unable to decode bytes as JPEG, PNG, GIF, or BMP]
	 [[Node: case/If_0/decode_image/cond_jpeg/cond_png/cond_gif/Assert_1/Assert = Assert[T=[DT_STRING], summarize=3, _device="/job:localhost/replica:0/task:0/device:CPU:0"](case/If_0/decode_image/cond_jpeg/cond_png/cond_gif/is_bmp, case/If_0/decode_image/cond_jpeg/cond_png/cond_gif/Assert_1/Assert/data_0)]]

I am thinking it is probably something to do with my tf records but cant seem to work it out.

Any ideas?

Please, could gave us also a tutorial for evaluation. Thanks

Thank you very much for the tutorial. Since you wrote it, they have changed the API a lot, in particular, for example, in data_decoders/tf_example_decoder.py they added the parameter num_additional_channels and then in addition to the "old" image containing 3 channels there is now image_additional_channels .

I cannot understand what it is for and how to use it, but at least according to the name it seems it should store any additional channels (beyond 3) and hopefully use them. Have you seen this new feature? Do you know whether it should take care of multi-channel images?

Hi,
I need help with the transfer learning section. Can you give me some more details about how to access the correct tensor and modify it?

When executing the train.py file i run into this error. I thought i had solved this before, but it's coming back. Not sure if it has to do with how we are encoding the tf records, but this is what mine looks like before i show the error message. def create_tf_example(group,path,files):

with tf.gfile.GFile(os.path.join(path, '{}'.format(str(group.filename)+'.jpg')), 'rb') as fid:
    encoded_jpg = fid.read()
encoded_jpg = fid.read()
encoded_jpg_io = io.BytesIO(encoded_jpg)
image = Image.open(encoded_jpg_io)
w_jpeg, h_jpeg = image.size
#print(w_jpeg, h_jpeg)

#define raw data height and width
h_raw = 512
w_raw = 7000

filename = (str(group.filename)).encode('utf8')
image_format = b'.jpg'
xmins = []
xmaxs = []
ymins = []
ymaxs = []
classes_text = []
classes = []

#Conversion factor from jpeg coordinates to raw data coordinates
jpeg_to_raw_w = w_raw / w_jpeg
jpeg_to_raw_h = h_raw / h_jpeg

for index, row in group.object.iterrows():
    xmins.append((row['xmin'] * jpeg_to_raw_w) / w_raw)
    xmaxs.append((row['xmax'] * jpeg_to_raw_w) / w_raw)
    ymins.append((row['ymin'] * jpeg_to_raw_h) / h_raw)
    ymaxs.append((row['ymax'] * jpeg_to_raw_h) / h_raw)
    classes_text.append(str(row['class']).encode('utf8')) 
    classes.append(class_text_to_int(str(row['class'])))

# Read in data as multi-dimensional arrays
#print(group.filename)
#print(group.filename2)
idx = files.index('/media/ssd/JupyterHubNotebooks/calebbowyer/models/object_detection/Object-Detection/images/test4/' + group.filename + '.npy')
plot_data = np.load(files[idx])
plot_data = plot_data.reshape(512,7000,1)

# Encode your matrix input as string
encoded_inputs = plot_data.tostring()

print(encoded_inputs)

print("encoded_inputs is: ", encoded_inputs)

tf_example = tf.train.Example(features=tf.train.Features(feature={
    'image/height': dataset_util.int64_feature(h_raw),
    'image/width': dataset_util.int64_feature(w_raw),
    'image/filename': dataset_util.bytes_feature(filename),
    'image/source_id': dataset_util.bytes_feature(filename),
    'image/encoded': dataset_util.bytes_feature(encoded_inputs), #Stores the raw data!
    'image/format': dataset_util.bytes_feature(image_format),
    'image/object/bbox/xmin': dataset_util.float_list_feature(xmins),
    'image/object/bbox/xmax': dataset_util.float_list_feature(xmaxs),
    'image/object/bbox/ymin': dataset_util.float_list_feature(ymins),
    'image/object/bbox/ymax': dataset_util.float_list_feature(ymaxs),
    'image/object/class/text': dataset_util.bytes_list_feature(classes_text),
    'image/object/class/label': dataset_util.int64_list_feature(classes),
    'image/channels': dataset_util.int64_feature(1)
}))   
return tf_example

Now this the recurring error message no matter how i change the above TF example.

nstructions for updating:
Please switch to tf.train.get_or_create_global_step
INFO:tensorflow:Summary name /clone_loss is illegal; using clone_loss instead.
/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/gradients_impl.py:97: UserWarning: Converting sparse IndexedSlices to a dense Tensor of unknown shape. This may consume a large amount of memory.
"Converting sparse IndexedSlices to a dense Tensor of unknown shape. " This line bothers me.
WARNING:tensorflow:From /usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/clip_ops.py:110: calling reduce_sum (from tensorflow.python.ops.math_ops) with keep_dims is deprecated and will be removed in a future version.
Instructions for updating:
keep_dims is deprecated, use keepdims instead
WARNING:tensorflow:From /usr/local/lib/python3.5/dist-packages/tensorflow/contrib/slim/python/slim/learning.py:736: Supervisor.init (from tensorflow.python.training.supervisor) is deprecated and will be removed in a future version.
Instructions for updating:
Please switch to tf.train.MonitoredTrainingSession
2018-06-07 13:05:51.876589: I tensorflow/core/platform/cpu_feature_guard.cc:137] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2
INFO:tensorflow:Restoring parameters from faster_rcnn_resnet101_kitti_2018_01_28/model.ckpt
INFO:tensorflow:Starting Session.
INFO:tensorflow:Saving checkpoint to path Training9_FRCNN/model.ckpt
INFO:tensorflow:Starting Queues.
2018-06-07 13:06:22.581377: W tensorflow/core/framework/op_kernel.cc:1198] Invalid argument: assertion failed: [Unable to decode bytes as JPEG, PNG, GIF, or BMP]
[[Node: case/cond/decode_image/cond_jpeg/cond_png/cond_gif/Assert_1/Assert = Assert[T=[DT_STRING],summarize=3](case/cond/decode_image/cond_jpeg/cond_png/cond_gif/is_bmp, case/cond/decode_image/cond_jpeg/cond_png/cond_gif/Assert_1/Assert/data_0)]]
INFO:tensorflow:Error reported to Coordinator: <class 'tensorflow.python.framework.errors_impl.InvalidArgumentError'>, assertion failed: [Unable to decode bytes as JPEG, PNG, GIF, or BMP]
[[Node: case/cond/decode_image/cond_jpeg/cond_png/cond_gif/Assert_1/Assert = Assert[T=[DT_STRING],summarize=3](case/cond/decode_image/cond_jpeg/cond_png/cond_gif/is_bmp, case/cond/decode_image/cond_jpeg/cond_png/cond_gif/Assert_1/Assert/data_0)]]
[[Node: IteratorGetNext = IteratorGetNextoutput_shapes=[[], [?], [?,4], [?], [?], [?], [?], [?], [?,?,3], [], [], []], output_types=[DT_STRING, DT_FLOAT, DT_FLOAT, DT_INT64, DT_INT64, DT_INT64, DT_BOOL, DT_FLOAT, DT_UINT8, DT_STRING, DT_INT32, DT_STRING], _device="/job:localhost/replica:0/task:0/device:CPU:0"]]
2018-06-07 13:06:24.613328: W tensorflow/core/framework/op_kernel.cc:1198] Invalid argument: assertion failed: [Unable to decode bytes as JPEG, PNG, GIF, or BMP]
[[Node: case/cond/decode_image/cond_jpeg/cond_png/cond_gif/Assert_1/Assert = Assert[T=[DT_STRING],summarize=3](case/cond/decode_image/cond_jpeg/cond_png/cond_gif/is_bmp, case/cond/decode_image/cond_jpeg/cond_png/cond_gif/Assert_1/Assert/data_0)]]
2018-06-07 13:06:24.613480: W tensorflow/core/framework/op_kernel.cc:1198] Invalid argument: assertion failed: [Unable to decode bytes as JPEG, PNG, GIF, or BMP]
[[Node: case/cond/decode_image/cond_jpeg/cond_png/cond_gif/Assert_1/Assert = Assert[T=[DT_STRING],summarize=3](case/cond/decode_image/cond_jpeg/cond_png/cond_gif/is_bmp, case/cond/decode_image/cond_jpeg/cond_png/cond_gif/Assert_1/Assert/data_0)]]
2018-06-07 13:06:24.613542: W tensorflow/core/framework/op_kernel.cc:1198] Invalid argument: assertion failed: [Unable to decode bytes as JPEG, PNG, GIF, or BMP]
[[Node: case/cond/decode_image/cond_jpeg/cond_png/cond_gif/Assert_1/Assert = Assert[T=[DT_STRING],summarize=3](case/cond/decode_image/cond_jpeg/cond_png/cond_gif/is_bmp, case/cond/decode_image/cond_jpeg/cond_png/cond_gif/Assert_1/Assert/data_0)]]
2018-06-07 13:06:24.614043: W tensorflow/core/framework/op_kernel.cc:1198] Invalid argument: assertion failed: [Unable to decode bytes as JPEG, PNG, GIF, or BMP]
[[Node: case/cond/decode_image/cond_jpeg/cond_png/cond_gif/Assert_1/Assert = Assert[T=[DT_STRING],summarize=3](case/cond/decode_image/cond_jpeg/cond_png/cond_gif/is_bmp, case/cond/decode_image/cond_jpeg/cond_png/cond_gif/Assert_1/Assert/data_0)]]
2018-06-07 13:06:24.614851: W tensorflow/core/framework/op_kernel.cc:1198] Invalid argument: assertion failed: [Unable to decode bytes as JPEG, PNG, GIF, or BMP]
[[Node: case/cond/decode_image/cond_jpeg/cond_png/cond_gif/Assert_1/Assert = Assert[T=[DT_STRING],summarize=3](case/cond/decode_image/cond_jpeg/cond_png/cond_gif/is_bmp, case/cond/decode_image/cond_jpeg/cond_png/cond_gif/Assert_1/Assert/data_0)]]
2018-06-07 13:06:29.286529: W tensorflow/core/framework/op_kernel.cc:1198] Invalid argument: assertion failed: [Unable to decode bytes as JPEG, PNG, GIF, or BMP]
[[Node: case/cond/decode_image/cond_jpeg/cond_png/cond_gif/Assert_1/Assert = Assert[T=[DT_STRING],summarize=3](case/cond/decode_image/cond_jpeg/cond_png/cond_gif/is_bmp, case/cond/decode_image/cond_jpeg/cond_png/cond_gif/Assert_1/Assert/data_0)]]
INFO:tensorflow:global_step/sec: 0
2018-06-07 13:06:29.959738: W tensorflow/core/framework/op_kernel.cc:1198] Invalid argument: assertion failed: [Unable to decode bytes as JPEG, PNG, GIF, or BMP]
[[Node: case/cond/decode_image/cond_jpeg/cond_png/cond_gif/Assert_1/Assert = Assert[T=[DT_STRING],summarize=3](case/cond/decode_image/cond_jpeg/cond_png/cond_gif/is_bmp, case/cond/decode_image/cond_jpeg/cond_png/cond_gif/Assert_1/Assert/data_0)]]
INFO:tensorflow:Caught OutOfRangeError. Stopping Training.
INFO:tensorflow:Finished training! Saving model to disk.
Traceback (most recent call last):
File "train.py", line 171, in
tf.app.run()
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/platform/app.py", line 124, in run
_sys.exit(main(argv))
File "train.py", line 167, in main
worker_job_name, is_chief, FLAGS.train_dir)
File "/media/ssd/darpa_data/models/research/object_detection/trainer.py", line 360, in train
saver=saver)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/contrib/slim/python/slim/learning.py", line 782, in train
ignore_live_threads=ignore_live_threads)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/training/supervisor.py", line 826, in stop
ignore_live_threads=ignore_live_threads)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/training/coordinator.py", line 387, in join
six.reraise(*self._exc_info_to_raise)
File "/usr/local/lib/python3.5/dist-packages/six.py", line 693, in reraise
raise value
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/training/queue_runner_impl.py", line 250, in _run
enqueue_callable()
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1251, in _single_operation_run
self._session, None, {}, [], target_list, status, None)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/errors_impl.py", line 473, in exit
c_api.TF_GetCode(self.status.status))
tensorflow.python.framework.errors_impl.InvalidArgumentError: assertion failed: [Unable to decode bytes as JPEG, PNG, GIF, or BMP]
[[Node: case/cond/decode_image/cond_jpeg/cond_png/cond_gif/Assert_1/Assert = Assert[T=[DT_STRING],summarize=3](case/cond/decode_image/cond_jpeg/cond_png/cond_gif/is_bmp, case/cond/decode_image/cond_jpeg/cond_png/cond_gif/Assert_1/Assert/data_0)]]
[[Node: IteratorGetNext = IteratorGetNextoutput_shapes=[[], [?], [?,4], [?], [?], [?], [?], [?], [?,?,3], [], [], []], output_types=[DT_STRING, DT_FLOAT, DT_FLOAT, DT_INT64, DT_INT64, DT_INT64, DT_BOOL, DT_FLOAT, DT_UINT8, DT_STRING, DT_INT32, DT_STRING], _device="/job:localhost/replica:0/task:0/device:CPU:0"]]

I have scan your tutorial.Great work!But I have some confusion about your procedure.In my understand, you add num_channel to faser_rcnn_meta_arch.py.No other place should I change or add?

Hi,
first of all, thanks for your work and for sharing it.
I am trying to modify the detection API for 4 channels objects, RGB+Depth.
I followed and modified the files as you suggested, but when I try to start training, I get the following error:

Traceback (most recent call last):
File "/home/xxx/.local/lib/python3.5/site-packages/tensorflow/python/framework/common_shapes.py", line 654, in _call_cpp_shape_fn_impl
input_tensors_as_shapes, status)
File "/home/xxx/anaconda3/lib/python3.5/contextlib.py", line 66, in exit
next(self.gen)
File "/home/xxx/.local/lib/python3.5/site-packages/tensorflow/python/framework/errors_impl.py", line 466, in raise_exception_on_not_ok_status
pywrap_tensorflow.TF_GetCode(status))
tensorflow.python.framework.errors_impl.InvalidArgumentError: Dimensions must be equal, but are 3 and 4 for 'Preprocessor/sub' (op: 'Sub') with input shapes: [1,?,?,3], [1,1,4].

I am working on trying to resolve it and I was wondering if you encountered anything similar and you alredy know what must be modified.
In addition, I would like to know which tensorflow version you used (my guess is not the latest) because, for example, the modification
optional uint32 num_input_channels = 28 [default=3];
in object_detection/protos/faster_rcnn.proto has to be modify to
optional uint32 num_input_channels = 30 [default=3];
since the following
optional HardExampleMiner hard_example_miner = 28;
is present.
So, I would like to understand if my error is due to TF version (new TF requires additional changes) or something else.

BTW, thanks again!