jessicayung / self-driving-car-nd

Udacity's Self-Driving Car Nanodegree project files and notes.

self-driving-car-nd

Udacity's Self-Driving Car Nanodegree project files and notes.

This repository contains project files and lecture notes for Udacity's Self-Driving Car Engineer Nanodegree program which I started working on on 27 October, 2016.

The Self-Driving Car Engineer is an online certification intended to prepare students to become self-driving car engineers. The program was developed by Udacity in partnership with Mercedes-Benz, NVIDIA, Otto, DiDi, BMW, McLaren and NextEv.

See also: My notes for Udacity's Machine Learning Nanodegree.

Program Outline:

Term 1: Deep Learning and Computer Vision

1. Deep Learning

• deep-learning-notes-and-labs: Notes on Deep Learning, Tensorflow and Keras
• Project 2: Traffic Sign Classifier
• Project 3: Behavioural Cloning
  • Train a car to drive in a 3D simulator using a deep neural network.
  • Input data comprises steering angles and camera images captured by driving with a keyboard / mouse / joystick in the simulator.

2. Computer Vision

• computer-vision-notes-and-labs: Notes on Computer Vision
• Project 1: Finding Lane Lines (Intro to Computer Vision)
• Project 4: Advanced Lane Lines
• Project 5: Vehicle Detection

Term 2: Sensor Fusion, Localisation and Control

1. Sensor Fusion

• Combining lidar and radar data to track objects in the environment using Kalman filters.
• Project 1: Extended Kalman Filters
• Project 2: Unscented Kalman Filters

2. Localisation

• Locate a car relative to the world (Align a car and sensors to the map).
• Use particle filters to localise the vehicle.
• Project 3: Kidnapped Vehicle (Particle Filters)

3. Control

• Fundamental concepts of robotic control.
• Build algorithms to steer car and wheels so as to meet an objective.
• Project 4: PID Controller
• Project 5: Model Predictive Control

Term 3: Path Planning, Controlling a Self-Driving Car

1. Path Planning

• Finding a sequence of steps in a maze (navigating cities, parking lots)
• Project 1: Path Planning (Driving a car down a highway with other cars in a simulator)

2. Advanced Deep Learning: Semantic Segmentation

• Fully Convolutional Networks
• Inference Performance (Optimising NNs in TensorFlow for Inference Speed)
• Project 2: Semantic Segmentation (Identifying free space on the road in a video clip)

3. Functional Safety

4. System Integration

• Put your code in a self-driving car

self-driving-car-nd's Issues

Fit Generator Issue

//==================================================================
Hello Mam,
I am planning to do a project on behavioral cloning, and I have been following your open source tutorial since then. I am a newbie to deep learning. I've been struck in this step for about a week. I am really sorry if this is a dumb question. Than you in advance for your great help.
//==================================================================

batch_size = 32
nb_epoch = 20

checkpointer = ModelCheckpoint(filepath="./tmp/comma-4c.{epoch:02d}-{val_loss:.2f}.hdf5", verbose=1, save_best_only=False)

model.fit_generator(train_generator,
samples_per_epoch=len(train_samples),
validation_data=validation_generator,
nb_val_samples=len(validation_samples), nb_epoch=nb_epoch,
callbacks=[checkpointer])
//==================================================================
//==================================================================
//==================================================================

C:\Users\krish\Anaconda3\lib\site-packages\ipykernel_launcher.py:10: UserWarning: The semantics of the Keras 2 argument steps_per_epoch is not the same as the Keras 1 argument samples_per_epoch. steps_per_epoch is the number of batches to draw from the generator at each epoch. Basically steps_per_epoch = samples_per_epoch/batch_size. Similarly nb_val_samples->validation_steps and val_samples->steps arguments have changed. Update your method calls accordingly.

Remove the CWD from sys.path while we load stuff.

C:\Users\krish\Anaconda3\lib\site-packages\ipykernel_launcher.py:10: UserWarning: Update your fit_generator call to the Keras 2 API: fit_generator(<generator..., validation_data=<generator..., callbacks=[<keras.ca..., steps_per_epoch=614, epochs=20, validation_steps=33)

Remove the CWD from sys.path while we load stuff.

Epoch 1/20

Exception in thread Thread-6:
Traceback (most recent call last):
File "C:\Users\krish\Anaconda3\lib\threading.py", line 916, in _bootstrap_inner
self.run()
File "C:\Users\krish\Anaconda3\lib\threading.py", line 864, in run
self._target(*self._args, **self._kwargs)
File "C:\Users\krish\Anaconda3\lib\site-packages\keras\utils\data_utils.py", line 568, in data_generator_task
generator_output = next(self._generator)
File "", line 13, in generator
center_image = mpimg.imread(name)
File "C:\Users\krish\Anaconda3\lib\site-packages\matplotlib\image.py", line 1278, in imread
im = pilread(fname)
File "C:\Users\krish\Anaconda3\lib\site-packages\matplotlib\image.py", line 1256, in pilread
with Image.open(fname) as image:
File "C:\Users\krish\Anaconda3\lib\site-packages\PIL\Image.py", line 2477, in open
fp = builtins.open(filename, "rb")
OSError: [Errno 22] Invalid argument: './data/C:\Users\krish\Desktop\IMG\center_2017_10_17_19_10_50_094.jpg'

---

StopIteration Traceback (most recent call last)
in ()
8 validation_data=validation_generator,
9 nb_val_samples=len(validation_samples), nb_epoch=nb_epoch,
---> 10 callbacks=[checkpointer])

~\Anaconda3\lib\site-packages\keras\legacy\interfaces.py in wrapper(*args, **kwargs)
85 warnings.warn('Update your ' + object_name + 86 ' call to the Keras 2 API: ' + signature, stacklevel=2)
---> 87 return func(*args, **kwargs)
88 wrapper._original_function = func
89 return wrapper

~\Anaconda3\lib\site-packages\keras\models.py in fit_generator(self, generator, steps_per_epoch, epochs, verbose, callbacks, validation_data, validation_steps, class_weight, max_queue_size, workers, use_multiprocessing, initial_epoch)
1119 workers=workers,
1120 use_multiprocessing=use_multiprocessing,
-> 1121 initial_epoch=initial_epoch)
1122
1123 @interfaces.legacy_generator_methods_support

~\Anaconda3\lib\site-packages\keras\legacy\interfaces.py in wrapper(*args, **kwargs)
85 warnings.warn('Update your ' + object_name + 86 ' call to the Keras 2 API: ' + signature, stacklevel=2)
---> 87 return func(*args, **kwargs)
88 wrapper._original_function = func
89 return wrapper

~\Anaconda3\lib\site-packages\keras\engine\training.py in fit_generator(self, generator, steps_per_epoch, epochs, verbose, callbacks, validation_data, validation_steps, class_weight, max_queue_size, workers, use_multiprocessing, shuffle, initial_epoch)
2009 batch_index = 0
2010 while steps_done < steps_per_epoch:
-> 2011 generator_output = next(output_generator)
2012
2013 if not hasattr(generator_output, 'len'):

StopIteration:

t2 P5 MPC Project Error

Problem traced to mpc.Solve(state, coeffs) (i.e. in MPC.cpp);

Listening to port 4567
Connected!!! 42["telemetry",{"ptsx":[-32.16173,-43.49173,-61.09,-78.29172,-93.05002,-107.7717],"ptsy":[113.361,105.941,92.88499,78.73102,65.34102,50.57938],"psi_unity":4.12033,"psi":3.733651,"x":-40.62,"y":108.73,"steering_angle":0,"throttle":0,"speed":0}]
cppad-20150000.2 error from unknown source
Error detected by false result for
c_index < num_cap
at line 838 in the file
/usr/local/include/cppad/thread_alloc.hpp
Assertion failed: (false), function Default, file /usr/local/include/cppad/error_handler.hpp, line 210.
Abort trap: 6

the potential solution to the failing case for the yellow lane video in Project 1

Thank you for sharing the code and the detailed description of your code. In Shortcomings of your writeup, you said that there are some failing cases in the yellow lane video, I write my code based on yours and encounter the same issue, today I find the potential reason of the issue. The reason may be that there are some horizontal line segments after Hough Transform and I have verified that these horizontal line segments can't be filtered in function draw_lines and find_line_fit.

In function draw_lines, in the first for line in lines loop, I add following code to filter these line segments
if np.abs(slope) < 0.09:
continue

because tan(5) = 0.087, here is degree, not radian, so I use 0.09 to filter these line segments with degree less than 5, maybe you can have a try to see whether it works

p4: gathering pixels near peak for each sliding window -> cannot append np arrays

x = []
y = []
...
np.append(x, x_additional)
np.append(y, y_additional)
print("x: ", x)
print("y: ", y)

x and y are still empty whereas x_additional and y_additional are nonempty, e.g. x_additional: [1072 1072 1072 ..., 1140 1142 1143]

are all the exercises self-contained?

are all the files to run this locally in the repo? looking at the miniflow for example, it doesn't look like the raw data file has been included. is that right?

The last bug about challenge.mp4

Hi,friend!Thanks for your share!
Have u tried to fixed the bug about generating the challenge.mp4?
I tried,but failed!
Thanks again!