rioffe / carnd-capstone-solution

Solution to Term 3, Project 3 of Udacity Nanodegree: Capstone Project

CarND-Capstone-Solution

Solution to Term 3, Project 3 of Udacity Nanodegree: Capstone Project

Team Silicon Forest Cruiser

Members

• Marcus Sinhalage ([email protected])
• Yury Kirpichev ([email protected])
• Duksan Ryu ([email protected])
• Steven Thomas ([email protected])
• Robert Ioffe ([email protected])

Team Lead

Robert Ioffe ([email protected])

Develop

Please see Contributing for detailed information.

This is the project repo for the final project of the Udacity Self-Driving Car Nanodegree: Programming a Real Self-Driving Car. For more information about the project, see the project introduction here.

Controllers

We use PID controller with low pass filtering over current linear velocity and stop accelearation for acceleration/breaking and a Udacity provided YawController for steering. We experimented with the PID controller with low pass filtering for steering as well, but found YawController to perform better.

Traffic Light Detection and Recognition Models

We developed several traffic light detection and recognition models. Detailed information about our models. We use three different models simultaneously to determine the color of the traffic lights (we use separate models for the simulation environment and separate models for the real environment - one of our models was trained on both real and simulated images and is uses for detection in both simulated and real environments).

Final Project Video

Please use one of the two installation options, either native or docker installation.

Native Installation

• Be sure that your workstation is running Ubuntu 16.04 Xenial Xerus or Ubuntu 14.04 Trusty Tahir. Ubuntu downloads can be found here.

• If using a Virtual Machine to install Ubuntu, use the following configuration as minimum:

  • 2 CPU
  • 2 GB system memory
  • 25 GB of free hard drive space

  The Udacity provided virtual machine has ROS and Dataspeed DBW already installed, so you can skip the next two steps if you are using this.

• Follow these instructions to install ROS

  • ROS Kinetic if you have Ubuntu 16.04.
  • ROS Indigo if you have Ubuntu 14.04.

• Dataspeed DBW

  • Use this option to install the SDK on a workstation that already has ROS installed: One Line SDK Install (binary)

• Download the Udacity Simulator.

Docker Installation

Install Docker

Build the docker container

docker build . -t capstone

Run the docker file

docker run -p 4567:4567 -v $PWD:/capstone -v /tmp/log:/root/.ros/ --rm -it capstone

Port Forwarding

To set up port forwarding, please refer to the instructions from term 2

Usage

1. Clone the project repository

git clone https://github.com/udacity/CarND-Capstone.git

2. Install python dependencies

cd CarND-Capstone
pip install -r requirements.txt

3. Make and run styx

./model_extraction.sh
cd ros
catkin_make
source devel/setup.sh
roslaunch launch/styx.launch

4. Run the simulator

Real world testing

1. Download training bag that was recorded on the Udacity self-driving car.
2. Unzip the file

unzip traffic_light_bag_file.zip

3. Play the bag file

rosbag play -l traffic_light_bag_file/traffic_light_training.bag

4. Launch your project in site mode

cd CarND-Capstone/ros
roslaunch launch/site.launch

5. Confirm that traffic light detection works on real life images

carnd-capstone-solution's Issues

When running tensorflow under ROS, I get CUBLAS_STATUS_NOT_INITIALIZED

So far wasn't able to resolve the issue.

[INFO] [1522158065.805556]: image_cb is called
2018-03-27 06:41:07.228565: E tensorflow/stream_executor/cuda/cuda_blas.cc:444] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED
2018-03-27 06:41:07.228591: W tensorflow/stream_executor/stream.cc:1901] attempting to perform BLAS operation using StreamExecutor without BLAS support
[ERROR] [1522158067.230025]: bad callback: <bound method TLDetector.image_cb of <__main__.TLDetector object at 0x7f9838e5cdd0>>
Traceback (most recent call last):
  File "/opt/ros/kinetic/lib/python2.7/dist-packages/rospy/topics.py", line 750, in _invoke_callback
    cb(msg)
  File "/home/rioffe/self_driving_car_class/CarND-Capstone-Solution/ros/src/tl_detector/tl_detector.py", line 99, in image_cb
    closest_tl_xy, light_wp, state = self.process_traffic_lights()
  File "/home/rioffe/self_driving_car_class/CarND-Capstone-Solution/ros/src/tl_detector/tl_detector.py", line 193, in process_traffic_lights
    state = self.get_light_state(self.lights[np.argmin(ds)])
  File "/home/rioffe/self_driving_car_class/CarND-Capstone-Solution/ros/src/tl_detector/tl_detector.py", line 159, in get_light_state
    return self.light_classifier.get_classification(image)
  File "/home/rioffe/self_driving_car_class/CarND-Capstone-Solution/ros/src/tl_detector/light_classification/tl_classifier.py", line 30, in get_classification
    output_dict = self.detector.run_inference_for_single_image(image)
  File "/home/rioffe/self_driving_car_class/CarND-Capstone-Solution/ros/src/tl_detector/light_classification/tl_detector.py", line 46, in run_inference_for_single_image
    feed_dict={self.image_tensor: np.expand_dims(image, 0)})
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 895, in run
    run_metadata_ptr)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 1128, in _run
    feed_dict_tensor, options, run_metadata)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 1344, in _do_run
    options, run_metadata)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 1363, in _do_call
    raise type(e)(node_def, op, message)
InternalError: Blas SGEMM launch failed : m=65536, n=64, k=24
	 [[Node: FeatureExtractor/InceptionV2/InceptionV2/Conv2d_1a_7x7/separable_conv2d = Conv2D[T=DT_FLOAT, data_format="NHWC", dilations=[1, 1, 1, 1], padding="VALID", strides=[1, 1, 1, 1], use_cudnn_on_gpu=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](FeatureExtractor/InceptionV2/InceptionV2/Conv2d_1a_7x7/separable_conv2d/depthwise, FeatureExtractor/InceptionV2/Conv2d_1a_7x7/pointwise_weights)]]
	 [[Node: Postprocessor/BatchMultiClassNonMaxSuppression/map/while/LoopCond/_155 = _HostRecv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_2072_Postprocessor/BatchMultiClassNonMaxSuppression/map/while/LoopCond", tensor_type=DT_BOOL, _device="/job:localhost/replica:0/task:0/device:CPU:0"](^_cloopPostprocessor/BatchMultiClassNonMaxSuppression/map/while/strided_slice/stack_1/_11)]]

Caused by op u'FeatureExtractor/InceptionV2/InceptionV2/Conv2d_1a_7x7/separable_conv2d', defined at:
  File "/home/rioffe/self_driving_car_class/CarND-Capstone-Solution/ros/src/tl_detector/tl_detector.py", line 200, in <module>
    TLDetector()
  File "/home/rioffe/self_driving_car_class/CarND-Capstone-Solution/ros/src/tl_detector/tl_detector.py", line 54, in __init__
    self.light_classifier = TLClassifier()
  File "/home/rioffe/self_driving_car_class/CarND-Capstone-Solution/ros/src/tl_detector/light_classification/tl_classifier.py", line 11, in __init__
    self.detector = TLDetector(model_path)
  File "/home/rioffe/self_driving_car_class/CarND-Capstone-Solution/ros/src/tl_detector/light_classification/tl_detector.py", line 22, in __init__
    tf.import_graph_def(od_graph_def, name='')
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/util/deprecation.py", line 316, in new_func
    return func(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/importer.py", line 554, in import_graph_def
    op_def=op_def)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 3160, in create_op
    op_def=op_def)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 1625, in __init__
    self._traceback = self._graph._extract_stack()  # pylint: disable=protected-access

InternalError (see above for traceback): Blas SGEMM launch failed : m=65536, n=64, k=24
	 [[Node: FeatureExtractor/InceptionV2/InceptionV2/Conv2d_1a_7x7/separable_conv2d = Conv2D[T=DT_FLOAT, data_format="NHWC", dilations=[1, 1, 1, 1], padding="VALID", strides=[1, 1, 1, 1], use_cudnn_on_gpu=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](FeatureExtractor/InceptionV2/InceptionV2/Conv2d_1a_7x7/separable_conv2d/depthwise, FeatureExtractor/InceptionV2/Conv2d_1a_7x7/pointwise_weights)]]
	 [[Node: Postprocessor/BatchMultiClassNonMaxSuppression/map/while/LoopCond/_155 = _HostRecv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_2072_Postprocessor/BatchMultiClassNonMaxSuppression/map/while/LoopCond", tensor_type=DT_BOOL, _device="/job:localhost/replica:0/task:0/device:CPU:0"](^_cloopPostprocessor/BatchMultiClassNonMaxSuppression/map/while/strided_slice/stack_1/_11)]]


[INFO] [1522158067.232028]: image_cb is called
2018-03-27 06:41:07.233066: I tensorflow/stream_executor/stream.cc:1851] stream 0x180d8f80 did not wait for stream: 0xada3ef0
2018-03-27 06:41:07.233089: I tensorflow/stream_executor/stream.cc:4637] stream 0x180d8f80 did not memcpy host-to-device; source: 0x7f9805a00000
2018-03-27 06:41:07.233127: F tensorflow/core/common_runtime/gpu/gpu_util.cc:343] CPU->GPU Memcpy failed
[tl_detector-8] process has died [pid 28553, exit code -6, cmd /home/rioffe/self_driving_car_class/CarND-Capstone-Solution/ros/src/tl_detector/tl_detector.py __name:=tl_detector __log:=/home/rioffe/.ros/log/7314028c-31c4-11e8-962e-1c1b0d675a18/tl_detector-8.log].
log file: /home/rioffe/.ros/log/7314028c-31c4-11e8-962e-1c1b0d675a18/tl_detector-8*.log

Failed to use frozen_sim_inception model

2018-03-27 13:35:16.955887: E tensorflow/core/common_runtime/executor.cc:651] Executor failed to create kernel. Invalid argument: NodeDef mentions attr 'index_type' not in Op<name=Fill; signature=dims:int32, value:T -> output:T; attr=T:type>; NodeDef: MultipleGridAnchorGenerator/Meshgrid/ExpandedShape/ones = Fill[T=DT_INT32, index_type=DT_INT32, _device="/job:localhost/replica:0/task:0/device:GPU:0"](MultipleGridAnchorGenerator/Meshgrid/ExpandedShape/Reshape, Postprocessor/BatchMultiClassNonMaxSuppression/map/TensorArrayUnstack_1/range/delta). (Check whether your GraphDef-interpreting binary is up to date with your GraphDef-generating binary.).
	 [[Node: MultipleGridAnchorGenerator/Meshgrid/ExpandedShape/ones = Fill[T=DT_INT32, index_type=DT_INT32, _device="/job:localhost/replica:0/task:0/device:GPU:0"](MultipleGridAnchorGenerator/Meshgrid/ExpandedShape/Reshape, Postprocessor/BatchMultiClassNonMaxSuppression/map/TensorArrayUnstack_1/range/delta)]]
2018-03-27 13:35:17.001860: E tensorflow/core/common_runtime/executor.cc:651] Executor failed to create kernel. Invalid argument: NodeDef mentions attr 'index_type' not in Op<name=Fill; signature=dims:int32, value:T -> output:T; attr=T:type>; NodeDef: MultipleGridAnchorGenerator/Meshgrid/ExpandedShape/ones = Fill[T=DT_INT32, index_type=DT_INT32, _device="/job:localhost/replica:0/task:0/device:GPU:0"](MultipleGridAnchorGenerator/Meshgrid/ExpandedShape/Reshape, Postprocessor/BatchMultiClassNonMaxSuppression/map/TensorArrayUnstack_1/range/delta). (Check whether your GraphDef-interpreting binary is up to date with your GraphDef-generating binary.).
	 [[Node: MultipleGridAnchorGenerator/Meshgrid/ExpandedShape/ones = Fill[T=DT_INT32, index_type=DT_INT32, _device="/job:localhost/replica:0/task:0/device:GPU:0"](MultipleGridAnchorGenerator/Meshgrid/ExpandedShape/Reshape, Postprocessor/BatchMultiClassNonMaxSuppression/map/TensorArrayUnstack_1/range/delta)]]
2018-03-27 13:35:17.051425: E tensorflow/core/common_runtime/executor.cc:651] Executor failed to create kernel. Invalid argument: NodeDef mentions attr 'index_type' not in Op<name=Fill; signature=dims:int32, value:T -> output:T; attr=T:type>; NodeDef: MultipleGridAnchorGenerator/Meshgrid/ExpandedShape/ones = Fill[T=DT_INT32, index_type=DT_INT32, _device="/job:localhost/replica:0/task:0/device:GPU:0"](MultipleGridAnchorGenerator/Meshgrid/ExpandedShape/Reshape, Postprocessor/BatchMultiClassNonMaxSuppression/map/TensorArrayUnstack_1/range/delta). (Check whether your GraphDef-interpreting binary is up to date with your GraphDef-generating binary.).
	 [[Node: MultipleGridAnchorGenerator/Meshgrid/ExpandedShape/ones = Fill[T=DT_INT32, index_type=DT_INT32, _device="/job:localhost/replica:0/task:0/device:GPU:0"](MultipleGridAnchorGenerator/Meshgrid/ExpandedShape/Reshape, Postprocessor/BatchMultiClassNonMaxSuppression/map/TensorArrayUnstack_1/range/delta)]]
[ERROR] [1522182917.055964]: bad callback: <bound method TLDetector.image_cb of <__main__.TLDetector object at 0x7f2320ffecd0>>
Traceback (most recent call last):
  File "/opt/ros/kinetic/lib/python2.7/dist-packages/rospy/topics.py", line 750, in _invoke_callback
    cb(msg)
  File "/home/rioffe/self_driving_car_class/CarND-Capstone-Solution/ros/src/tl_detector/tl_detector.py", line 98, in image_cb
    closest_tl_xy, light_wp, state = self.process_traffic_lights()
  File "/home/rioffe/self_driving_car_class/CarND-Capstone-Solution/ros/src/tl_detector/tl_detector.py", line 192, in process_traffic_lights
    state = self.get_light_state(self.lights[np.argmin(ds)])
  File "/home/rioffe/self_driving_car_class/CarND-Capstone-Solution/ros/src/tl_detector/tl_detector.py", line 158, in get_light_state
    return self.light_classifier.get_classification(image)
  File "/home/rioffe/self_driving_car_class/CarND-Capstone-Solution/ros/src/tl_detector/light_classification/tl_classifier.py", line 33, in get_classification
    output_dict = self.detector.run_inference_for_single_image(image)
  File "/home/rioffe/self_driving_car_class/CarND-Capstone-Solution/ros/src/tl_detector/light_classification/tl_detector.py", line 44, in run_inference_for_single_image
    feed_dict={self.image_tensor: np.expand_dims(image, 0)})
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 895, in run
    run_metadata_ptr)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 1128, in _run
    feed_dict_tensor, options, run_metadata)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 1344, in _do_run
    options, run_metadata)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 1363, in _do_call
    raise type(e)(node_def, op, message)
InvalidArgumentError: NodeDef mentions attr 'index_type' not in Op<name=Fill; signature=dims:int32, value:T -> output:T; attr=T:type>; NodeDef: MultipleGridAnchorGenerator/Meshgrid/ExpandedShape/ones = Fill[T=DT_INT32, index_type=DT_INT32, _device="/job:localhost/replica:0/task:0/device:GPU:0"](MultipleGridAnchorGenerator/Meshgrid/ExpandedShape/Reshape, Postprocessor/BatchMultiClassNonMaxSuppression/map/TensorArrayUnstack_1/range/delta). (Check whether your GraphDef-interpreting binary is up to date with your GraphDef-generating binary.).
	 [[Node: MultipleGridAnchorGenerator/Meshgrid/ExpandedShape/ones = Fill[T=DT_INT32, index_type=DT_INT32, _device="/job:localhost/replica:0/task:0/device:GPU:0"](MultipleGridAnchorGenerator/Meshgrid/ExpandedShape/Reshape, Postprocessor/BatchMultiClassNonMaxSuppression/map/TensorArrayUnstack_1/range/delta)]]

Caused by op u'MultipleGridAnchorGenerator/Meshgrid/ExpandedShape/ones', defined at:
  File "/home/rioffe/self_driving_car_class/CarND-Capstone-Solution/ros/src/tl_detector/tl_detector.py", line 207, in <module>
    TLDetector()
  File "/home/rioffe/self_driving_car_class/CarND-Capstone-Solution/ros/src/tl_detector/tl_detector.py", line 54, in __init__
    self.light_classifier = TLClassifier()
  File "/home/rioffe/self_driving_car_class/CarND-Capstone-Solution/ros/src/tl_detector/light_classification/tl_classifier.py", line 14, in __init__
    self.detector = TLDetector(model_path)
  File "/home/rioffe/self_driving_car_class/CarND-Capstone-Solution/ros/src/tl_detector/light_classification/tl_detector.py", line 18, in __init__
    tf.import_graph_def(od_graph_def, name='')
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/util/deprecation.py", line 316, in new_func
    return func(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/importer.py", line 554, in import_graph_def
    op_def=op_def)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 3160, in create_op
    op_def=op_def)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 1625, in __init__
    self._traceback = self._graph._extract_stack()  # pylint: disable=protected-access

InvalidArgumentError (see above for traceback): NodeDef mentions attr 'index_type' not in Op<name=Fill; signature=dims:int32, value:T -> output:T; attr=T:type>; NodeDef: MultipleGridAnchorGenerator/Meshgrid/ExpandedShape/ones = Fill[T=DT_INT32, index_type=DT_INT32, _device="/job:localhost/replica:0/task:0/device:GPU:0"](MultipleGridAnchorGenerator/Meshgrid/ExpandedShape/Reshape, Postprocessor/BatchMultiClassNonMaxSuppression/map/TensorArrayUnstack_1/range/delta). (Check whether your GraphDef-interpreting binary is up to date with your GraphDef-generating binary.).
	 [[Node: MultipleGridAnchorGenerator/Meshgrid/ExpandedShape/ones = Fill[T=DT_INT32, index_type=DT_INT32, _device="/job:localhost/replica:0/task:0/device:GPU:0"](MultipleGridAnchorGenerator/Meshgrid/ExpandedShape/Reshape, Postprocessor/BatchMultiClassNonMaxSuppression/map/TensorArrayUnstack_1/range/delta)]]

Cannot figure out how to create one tensorflow session and keep it open

Current code will keep creating tensorflow devices due to the following line:

with tf.Session() as sess:

in light_classification/tl_detector.py

I tried to create
self.sess = tf.Session(graph=self.detection_graph)

and then in run_inference_for_single_image

    with self.detection_graph.as_default():
        with self.sess.as_default() as sess:

But I get the following error:

[ERROR] [1522017712.254920]: bad callback: <bound method TLDetector.image_cb of <main.TLDetector object at 0x7f3c039355d0>>
Traceback (most recent call last):
File "/opt/ros/kinetic/lib/python2.7/dist-packages/rospy/topics.py", line 750, in _invoke_callback
cb(msg)
File "/home/rioffe/self_driving_car_class/CarND-Capstone-Solution/ros/src/tl_detector/tl_detector.py", line 99, in image_cb
closest_tl_xy, light_wp, state = self.process_traffic_lights()
File "/home/rioffe/self_driving_car_class/CarND-Capstone-Solution/ros/src/tl_detector/tl_detector.py", line 193, in process_traffic_lights
state = self.get_light_state(self.lights[np.argmin(ds)])
File "/home/rioffe/self_driving_car_class/CarND-Capstone-Solution/ros/src/tl_detector/tl_detector.py", line 159, in get_light_state
return self.light_classifier.get_classification(image)
File "/home/rioffe/self_driving_car_class/CarND-Capstone-Solution/ros/src/tl_detector/light_classification/tl_classifier.py", line 30, in get_classification
output_dict = self.detector.run_inference_for_single_image(image)
File "/home/rioffe/self_driving_car_class/CarND-Capstone-Solution/ros/src/tl_detector/light_classification/tl_detector.py", line 43, in run_inference_for_single_image
feed_dict={image_tensor: np.expand_dims(image, 0)})
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 905, in run
run_metadata_ptr)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 1137, in _run
feed_dict_tensor, options, run_metadata)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 1355, in _do_run
options, run_metadata)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 1374, in _do_call
raise type(e)(node_def, op, message)
InternalError: Blas SGEMM launch failed : m=65536, n=64, k=24
[[Node: FeatureExtractor/InceptionV2/InceptionV2/Conv2d_1a_7x7/separable_conv2d = Conv2D[T=DT_FLOAT, data_format="NHWC", dilations=[1, 1, 1, 1], padding="VALID", strides=[1, 1, 1, 1], use_cudnn_on_gpu=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](FeatureExtractor/InceptionV2/InceptionV2/Conv2d_1a_7x7/separable_conv2d/depthwise, FeatureExtractor/InceptionV2/Conv2d_1a_7x7/pointwise_weights)]]
[[Node: Postprocessor/BatchMultiClassNonMaxSuppression/map/while/MultiClassNonMaxSuppression/strided_slice_1/_75 = _Recvclient_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_1158_...ed_slice_1", tensor_type=DT_INT32, _device="/job:localhost/replica:0/task:0/device:CPU:0"]]

Caused by op u'FeatureExtractor/InceptionV2/InceptionV2/Conv2d_1a_7x7/separable_conv2d', defined at:
File "/home/rioffe/self_driving_car_class/CarND-Capstone-Solution/ros/src/tl_detector/tl_detector.py", line 200, in
TLDetector()
File "/home/rioffe/self_driving_car_class/CarND-Capstone-Solution/ros/src/tl_detector/tl_detector.py", line 54, in init
self.light_classifier = TLClassifier()
File "/home/rioffe/self_driving_car_class/CarND-Capstone-Solution/ros/src/tl_detector/light_classification/tl_classifier.py", line 11, in init
self.detector = TLDetector(model_path)
File "/home/rioffe/self_driving_car_class/CarND-Capstone-Solution/ros/src/tl_detector/light_classification/tl_detector.py", line 18, in init
tf.import_graph_def(od_graph_def, name='')
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/util/deprecation.py", line 432, in new_func
return func(*args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/importer.py", line 553, in import_graph_def
op_def=op_def)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 3271, in create_op
op_def=op_def)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 1650, in init
self._traceback = self._graph._extract_stack() # pylint: disable=protected-access

InternalError (see above for traceback): Blas SGEMM launch failed : m=65536, n=64, k=24
[[Node: FeatureExtractor/InceptionV2/InceptionV2/Conv2d_1a_7x7/separable_conv2d = Conv2D[T=DT_FLOAT, data_format="NHWC", dilations=[1, 1, 1, 1], padding="VALID", strides=[1, 1, 1, 1], use_cudnn_on_gpu=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](FeatureExtractor/InceptionV2/InceptionV2/Conv2d_1a_7x7/separable_conv2d/depthwise, FeatureExtractor/InceptionV2/Conv2d_1a_7x7/pointwise_weights)]]
[[Node: Postprocessor/BatchMultiClassNonMaxSuppression/map/while/MultiClassNonMaxSuppression/strided_slice_1/_75 = _Recvclient_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_1158_...ed_slice_1", tensor_type=DT_INT32, _device="/job:localhost/replica:0/task:0/device:CPU:0"]]

Car stops at three phantom lights before the end of the track

The car stops at three phantom lights before the very end of the track:
There is this comment in tl_detector.py, line 172:

This assumes ego always travels around loop in start direction. Should be fixed to use Yuri's calculation from waypoint_updater.py.

Steve, could you fix this?