feed_dict={K.learning_phase(): 1} while executing train_mitoses.py about deep-histopath

[sukruburakcetin]

Hi @dusenberrymw, I'd like to say big thank you for your kind, quick response and your amazing effort.

Do you mind if I ask what is your current set up's version numbers?, I looked around but I couldn't find any info about it.

[sukruburakcetin]

Hi again @dusenberrymw, I'ld like to mention that I tried your work on Linux platform(ubuntu 16.04LTS), I reduced training batch size due to limited GPU ram availability and it is working flawlessly.

[dusenberrymw]

Hi @Narthyard, that's great that you were able to successfully run it on Linux. Although I didn't post an update here yesterday, we continued to look into it, and @gweidner was able to reproduce the exact issue on a Windows setup. Thus, it appears to be a core TensorFlow issue. We'll create a minimal example that causes the same issue, and then look into options.

For your particular workflow, is your Linux setup sufficient for you to be able to move forward for now, or is this issue on Windows still a blocker?

Thanks again for trying out the code!

[sukruburakcetin]

Hi @dusenberrymw, first of all, I'd like to say thank you for taking the time to work with me. You really went out of your way to ensure my case is solved the way I like.

So far so good, I'll carry on with Linux platform for now. I'll limit the number of patches to examine the traning model efficiently and quickly. The current set up produces 782,490 patches, and my GPU and its ram capability are way too low. I tried to set them all but it takes ages to feed the model, so I interrupted it after one and a half day.

Now, I will just use limited number of patches to be ensure that system is producing the desired outputs. Then, I may try it on super computer to examine the environment in a different way, so I also log the traning time when the system setup is premium.

I'm looking forward to try your freshly added and updated "predict_mitosis" code. I'll report for the upcoming errors if any. You are doing amazing job, thanks again for both of you. @gweidner

[sukruburakcetin]

@dusenberrymw

and these are other errors that I've got during compiling:

Traceback (most recent call last):
  File "C:\Users\Burak\Anaconda3\lib\site-packages\tensorflow\python\client\session.py", line 1327, in _do_call
    return fn(*args)
  File "C:\Users\Burak\Anaconda3\lib\site-packages\tensorflow\python\client\session.py", line 1306, in _run_fn
    status, run_metadata)
  File "C:\Users\Burak\Anaconda3\lib\contextlib.py", line 66, in __exit__
    next(self.gen)
  File "C:\Users\Burak\Anaconda3\lib\site-packages\tensorflow\python\framework\errors_impl.py", line 466, in raise_exception_on_not_ok_status
    pywrap_tensorflow.TF_GetCode(status))
tensorflow.python.framework.errors_impl.InvalidArgumentError: slice index -1 of dimension 0 out of bounds.
         [[Node: strided_slice = StridedSlice[Index=DT_INT32, T=DT_STRING, begin_mask=0, ellipsis_mask=0, end_mask=0, new_axis_mask=0, shrink_axis_mask=1](StringSplit:1, strided_slice/stack, strided_slice/stack_1, strided_slice/stack_2)]]
         [[Node: data/IteratorGetNext = IteratorGetNext[output_shapes=[[?,64,64,3], [?], [?]], output_types=[DT_FLOAT, DT_FLOAT, DT_STRING], _device="/job:localhost/replica:0/task:0/cpu:0"](data/Iterator)]]

[sukruburakcetin]

Now, I wanted to try something different:

I changed my Anaconda version from 3.5 to 3.6 and changed cudnn version from v5.1 to v6.0.

(tensorflow) C:\Users\Burak\Desktop\downloads2\tensorflow-status-checker>python tensorflow_self_check.py
TensorFlow successfully installed.
The installed version of TensorFlow includes GPU support.

After that, I rerun the code itself with the same exact configuration, but still getting same error.

(tensorflow) E:\deep-histopath>python train_mitoses.py
Using TensorFlow backend.
experiment directory: experiments\mitoses\171003_172944_vgg_mokoko
2017-10-03 17:29:45.435787: I C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\platform\cpu_feature_guard.cc:137] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX AVX2
2017-10-03 17:29:45.737953: I C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\common_runtime\gpu\gpu_device.cc:965] Found device 0 with properties:
name: GeForce GT 710 major: 3 minor: 5 memoryClockRate(GHz): 0.954
pciBusID: 0000:01:00.0
totalMemory: 1.00GiB freeMemory: 838.58MiB
2017-10-03 17:29:45.738055: I C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\common_runtime\gpu\gpu_device.cc:1055] Creating TensorFlow device (/device:GPU:0) -> (device: 0, name: GeForce GT 710, pci bus id: 0000:01:00.0, compute capability: 3.5)
2017-10-03 17:30:28.563156: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-03 17:30:28.563205: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-03 17:30:28.563313: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-03 17:30:28.575001: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
         [[Node: strided_slice = StridedSlice[Index=DT_INT32, T=DT_STRING, begin_mask=0, ellipsis_mask=0, end_mask=0, new_axis_mask=0, shrink_axis_mask=1](StringSplit:1, strided_slice/stack, strided_slice/stack_1, strided_slice/stack_2)]]
2017-10-03 17:30:28.575028: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-03 17:30:28.585909: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-03 17:30:28.589385: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-03 17:30:28.597035: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-03 17:30:28.600166: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-03 17:30:28.606988: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
Traceback (most recent call last):
  File "C:\Users\Burak\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\client\session.py", line 1323, in _do_call
    return fn(*args)
  File "C:\Users\Burak\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\client\session.py", line 1302, in _run_fn
    status, run_metadata)
  File "C:\Users\Burak\Anaconda3\envs\tensorflow\lib\contextlib.py", line 88, in __exit__
    next(self.gen)
  File "C:\Users\Burak\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\framework\errors_impl.py", line 467, in raise_exception_on_not_ok_status
    c_api.TF_GetCode(status.status))
tensorflow.python.framework.errors_impl.InvalidArgumentError: slice index -1 of dimension 0 out of bounds.
         [[Node: strided_slice = StridedSlice[Index=DT_INT32, T=DT_STRING, begin_mask=0, ellipsis_mask=0, end_mask=0, new_axis_mask=0, shrink_axis_mask=1](StringSplit:1, strided_slice/stack, strided_slice/stack_1, strided_slice/stack_2)]]
         [[Node: data/IteratorGetNext = IteratorGetNext[output_shapes=[[?,64,64,3], [?], [?]], output_types=[DT_FLOAT, DT_FLOAT, DT_STRING], _device="/job:localhost/replica:0/task:0/cpu:0"](data/Iterator)]]
         [[Node: data/boolean_mask_2/Shape/_287 = _HostRecv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_668_data/boolean_mask_2/Shape", tensor_type=DT_INT32, _device="/job:localhost/replica:0/task:0/device:GPU:0"]()]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "train_mitoses.py", line 752, in <module>
    args.log_interval, args.threads, args.checkpoint, args.resume)
  File "train_mitoses.py", line 600, in train
    feed_dict={K.learning_phase(): 1})
  File "C:\Users\Burak\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\client\session.py", line 889, in run
    run_metadata_ptr)
  File "C:\Users\Burak\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\client\session.py", line 1120, in _run
    feed_dict_tensor, options, run_metadata)
  File "C:\Users\Burak\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\client\session.py", line 1317, in _do_run
    options, run_metadata)
  File "C:\Users\Burak\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\client\session.py", line 1336, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: slice index -1 of dimension 0 out of bounds.
         [[Node: strided_slice = StridedSlice[Index=DT_INT32, T=DT_STRING, begin_mask=0, ellipsis_mask=0, end_mask=0, new_axis_mask=0, shrink_axis_mask=1](StringSplit:1, strided_slice/stack, strided_slice/stack_1, strided_slice/stack_2)]]
         [[Node: data/IteratorGetNext = IteratorGetNext[output_shapes=[[?,64,64,3], [?], [?]], output_types=[DT_FLOAT, DT_FLOAT, DT_STRING], _device="/job:localhost/replica:0/task:0/cpu:0"](data/Iterator)]]
         [[Node: data/boolean_mask_2/Shape/_287 = _HostRecv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_668_data/boolean_mask_2/Shape", tensor_type=DT_INT32, _device="/job:localhost/replica:0/task:0/device:GPU:0"]()]]
2017-10-03 17:30:28.626792: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-03 17:30:28.626985: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-03 17:30:28.632310: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-03 17:30:28.639850: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-03 17:30:28.654266: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-03 17:30:28.656794: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-03 17:30:28.662539: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-03 17:30:28.671291: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-03 17:30:28.681184: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-03 17:30:28.684647: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-03 17:30:28.692484: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-03 17:30:28.710251: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-03 17:30:28.710270: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-03 17:30:28.710258: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-03 17:30:28.722299: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-03 17:30:28.732081: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.

[dusenberrymw]

Hi @Narthyard, thanks for reaching out, and thanks for trying out the code! Interesting issue -- I have not run into this issue on my current setup, so this will be a good exercise to find out what is going on in the case of different setups.

Looking at the error message, it looks like it is an issue either with the Dataset API, or with the tf.boolean_mask calls I'm using on the image tensors coming out of the Dataset object.

tensorflow.python.framework.errors_impl.InvalidArgumentError: slice index -1 of dimension 0 out of bounds.
         [[Node: strided_slice = StridedSlice[Index=DT_INT32, T=DT_STRING, begin_mask=0, ellipsis_mask=0, end_mask=0, new_axis_mask=0, shrink_axis_mask=1](StringSplit:1, strided_slice/stack, strided_slice/stack_1, strided_slice/stack_2)]]
         [[Node: data/IteratorGetNext = IteratorGetNext[output_shapes=[[?,64,64,3], [?], [?]], output_types=[DT_FLOAT, DT_FLOAT, DT_STRING], _device="/job:localhost/replica:0/task:0/cpu:0"](data/Iterator)]]
         [[Node: data/boolean_mask_2/Shape/_287 = _HostRecv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_668_data/boolean_mask_2/Shape", tensor_type=DT_INT32, _device="/job:localhost/replica:0/task:0/device:GPU:0"]()]]
2017-10-03 17:30:28.626792: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-03 17:30:28.626985: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-03 17:30:28.632310: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-03 17:30:28.639850: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-03 17:30:28.654266: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-03 17:30:28.656794: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-03 17:30:28.662539: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-03 17:30:28.671291: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-03 17:30:28.681184: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-03 17:30:28.684647: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-03 17:30:28.692484: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-03 17:30:28.710251: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-03 17:30:28.710270: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-03 17:30:28.710258: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-03 17:30:28.722299: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-03 17:30:28.732081: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.

I'll investigate this a bit further with the TensorFlow debugger and see if I can determine the exact op and get back to you.

[sukruburakcetin]

@dusenberrymw By the way, I didn't download the testing_image_data due to fact that I just wanted to train the model without validation. Is the error itself correlate with lacking test image data? I'm not sure about it, that may be a silly question.

[dusenberrymw]

@Narthyard I debugged this a bit, and it was indeed an issue with the output of the Dataset API. I believe it was due to a file pattern used to match train/val images that had paths specific to *nix operating systems, and thus was failing on Windows. I pushed an update to make this file patten generic -- can you grab the latest code and try it again?

As for the testing images, your original intuition was correct -- for training the model, the testing images are not necessary. For clarity, the preprocess script will still split the training images into a training and validation set, but those still come from the original training folder, and not the testing folder.

Here is my current setup:

• OS: macOS & Linux
• Python 3.6 (official release, and miniconda release)
• Keras bleeding edge (not strictly necessary to be bleeding edge, but I make updates sometimes)
• Tensorflow bleeding edge (not necessary, latest 1.3 release should be fine -- that being said, the Dataset API is still new (and it just moved out of staging in tf.contrib to it's new home in tf.data), so I might end up needing to bump that requirement up)
• everything else is the latest version installed with pip

Please let me know if the latest change fixes the issue!

[sukruburakcetin]

@dusenberrymw I grabbed your current update by hoping to solve the problem(your solution was indeed sounded plausible) , but unfortunately still facing with the same problem.

C:\Users\Burak\Anaconda3\envs\tensorflow\python.exe E:/dusenberrymw/deep-histopath/train_mitoses.py --exp_name MitosisTraining --exp_name_suffix Version_SupportWindowsPaths
Using TensorFlow backend.
experiment directory: experiments\mitoses\MitosisTraining_Version_SupportWindowsPaths
2017-10-05 11:30:13.586755: I C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\platform\cpu_feature_guard.cc:137] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX AVX2
2017-10-05 11:30:13.825582: I C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\common_runtime\gpu\gpu_device.cc:965] Found device 0 with properties: 
name: GeForce GT 710 major: 3 minor: 5 memoryClockRate(GHz): 0.954
pciBusID: 0000:01:00.0
totalMemory: 1.00GiB freeMemory: 837.65MiB
2017-10-05 11:30:13.825833: I C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\common_runtime\gpu\gpu_device.cc:1055] Creating TensorFlow device (/device:GPU:0) -> (device: 0, name: GeForce GT 710, pci bus id: 0000:01:00.0, compute capability: 3.5)
2017-10-05 11:30:38.293369: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-05 11:30:38.293387: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-05 11:30:38.293413: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-05 11:30:38.296365: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-05 11:30:38.296572: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-05 11:30:38.299372: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-05 11:30:38.299383: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
	 [[Node: strided_slice = StridedSlice[Index=DT_INT32, T=DT_STRING, begin_mask=0, ellipsis_mask=0, end_mask=0, new_axis_mask=0, shrink_axis_mask=1](StringSplit:1, strided_slice/stack, strided_slice/stack_1, strided_slice/stack_2)]]
2017-10-05 11:30:38.299481: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-05 11:30:38.302215: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-05 11:30:38.304406: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-05 11:30:38.331863: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-05 11:30:38.341073: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-05 11:30:38.341304: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-05 11:30:38.343026: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-05 11:30:38.343388: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-05 11:30:38.346331: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-05 11:30:38.346564: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-05 11:30:38.348455: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-05 11:30:38.348663: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\frameworkTraceback (most recent call last):
\op_kernel.cc:  File "C:\Users\Burak\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\client\session.py", line 1323, in _do_call
1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
    return fn(*args)
  File "C:\Users\Burak\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\client\session.py", line 1302, in _run_fn
    status, run_metadata)
  File "C:\Users\Burak\Anaconda3\envs\tensorflow\lib\contextlib.py", line 88, in __exit__
    next(self.gen)
  File "C:\Users\Burak\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\framework\errors_impl.py", line 467, in raise_exception_on_not_ok_status
    c_api.TF_GetCode(status.status))
tensorflow.python.framework.errors_impl.InvalidArgumentError: slice index -1 of dimension 0 out of bounds.
	 [[Node: strided_slice = StridedSlice[Index=DT_INT32, T=DT_STRING, begin_mask=0, ellipsis_mask=0, end_mask=0, new_axis_mask=0, shrink_axis_mask=1](StringSplit:1, strided_slice/stack, strided_slice/stack_1, strided_slice/stack_2)]]
	 [[Node: data/IteratorGetNext = IteratorGetNext[output_shapes=[[?,64,64,3], [?], [?]], output_types=[DT_FLOAT, DT_FLOAT, DT_STRING], _device="/job:localhost/replica:0/task:0/cpu:0"](data/Iterator)]]
	 [[Node: data/IteratorGetNext/_195 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_243_data/IteratorGetNext", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:GPU:0"]()]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "E:/dusenberrymw/deep-histopath/train_mitoses.py", line 751, in <module>
    args.log_interval, args.threads, args.checkpoint, args.resume)
  File "E:/dusenberrymw/deep-histopath/train_mitoses.py", line 600, in train
    feed_dict={K.learning_phase(): 1})
  File "C:\Users\Burak\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\client\session.py", line 889, in run
    run_metadata_ptr)
  File "C:\Users\Burak\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\client\session.py", line 1120, in _run
    feed_dict_tensor, options, run_metadata)
  File "C:\Users\Burak\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\client\session.py", line 1317, in _do_run
    options, run_metadata)
  File "C:\Users\Burak\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\client\session.py", line 1336, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: slice index -1 of dimension 0 out of bounds.
	 [[Node: strided_slice = StridedSlice[Index=DT_INT32, T=DT_STRING, begin_mask=0, ellipsis_mask=0, end_mask=0, new_axis_mask=0, shrink_axis_mask=1](StringSplit:1, strided_slice/stack, strided_slice/stack_1, strided_slice/stack_2)]]
	 [[Node: data/IteratorGetNext = IteratorGetNext[output_shapes=[[?,64,64,3], [?], [?]], output_types=[DT_FLOAT, DT_FLOAT, DT_STRING], _device="/job:localhost/replica:0/task:0/cpu:0"](data/Iterator)]]
	 [[Node: data/IteratorGetNext/_195 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_243_data/IteratorGetNext", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:GPU:0"]()]]
2017-10-05 11:30:38.360602: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-05 11:30:38.360683: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-05 11:30:38.360864: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-05 11:30:38.360961: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-05 11:30:38.363820: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-05 11:30:38.375785: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-05 11:30:38.375858: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-05 11:30:38.376007: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-05 11:30:38.376372: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-05 11:30:38.378195: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-05 11:30:38.380096: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-05 11:30:38.380135: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-05 11:30:38.380207: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-05 11:30:38.381332: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-05 11:30:38.382377: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-05 11:30:38.384195: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-05 11:30:38.385019: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-05 11:30:38.384275: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-05 11:30:38.385913: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-05 11:30:38.386316: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-05 11:30:38.387240: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-05 11:30:38.388187: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-05 11:30:38.388614: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-05 11:30:38.390043: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-05 11:30:38.398258: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-05 11:30:38.398334: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-05 11:30:38.398637: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-05 11:30:38.398770: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-05 11:30:38.400192: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-05 11:30:38.400990: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-05 11:30:38.414696: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-05 11:30:38.414768: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-05 11:30:38.415164: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-05 11:30:38.427581: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-05 11:30:38.427654: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-05 11:30:38.427609: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-05 11:30:38.427978: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-05 11:30:38.430165: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-05 11:30:38.430358: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-05 11:30:38.430729: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-05 11:30:38.432163: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-05 11:30:38.443497: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-05 11:30:38.443526: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-05 11:30:38.443728: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-05 11:30:38.455431: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-05 11:30:38.462065: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-05 11:30:38.471308: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-05 11:30:38.471325: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-05 11:30:38.480262: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-05 11:30:38.480291: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-05 11:30:38.480494: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-05 11:30:38.482919: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.

Process finished with exit code 1

[dusenberrymw]

@Narthyard Your welcome, and thank you for taking the time to try out the codebase and report issues! I'm going to close this issue now and keep track of the dependency bug as a separate task. Thanks for reporting it and trying out the code!