commaai / research

dataset and code for 2016 paper "Learning a Driving Simulator"

the people's comma

the paper

Learning a Driving Simulator

the comma.ai driving dataset

7 and a quarter hours of largely highway driving. Enough to train what we had in Bloomberg.

Examples

We present two Machine Learning Experiments to show possible ways to use this dataset:

Training a steering angle predictor

Training a generative image model

Downloading the dataset

./get_data.sh

or get it at archive.org comma dataset

45 GB compressed, 80 GB uncompressed

dog/2016-01-30--11-24-51 (7.7G)
dog/2016-01-30--13-46-00 (8.5G)
dog/2016-01-31--19-19-25 (3.0G)
dog/2016-02-02--10-16-58 (8.1G)
dog/2016-02-08--14-56-28 (3.9G)
dog/2016-02-11--21-32-47 (13G)
dog/2016-03-29--10-50-20 (12G)
emily/2016-04-21--14-48-08 (4.4G)
emily/2016-05-12--22-20-00 (7.5G)
frodo/2016-06-02--21-39-29 (6.5G)
frodo/2016-06-08--11-46-01 (2.7G)

Dataset referenced on this page is copyrighted by comma.ai and published under the Creative Commons Attribution-NonCommercial-ShareAlike 3.0 License. This means that you must attribute the work in the manner specified by the authors, you may not use this work for commercial purposes and if you alter, transform, or build upon this work, you may distribute the resulting work only under the same license.

Dataset structure

The dataset consists of 10 videos clips of variable size recorded at 20 Hz with a camera mounted on the windshield of an Acura ILX 2016. In parallel to the videos we also recorded some measurements such as car's speed, acceleration, steering angle, GPS coordinates, gyroscope angles. See the full log list here. These measurements are transformed into a uniform 100 Hz time base.

The dataset folder structure is the following:

+-- dataset
|   +-- camera
|   |   +-- 2016-04-21--14-48-08
|   |   ...
|   +-- log
|   |   +-- 2016-04-21--14-48-08
|   |   ...

All the files come in hdf5 format and are named with the time they were recorded. The camera dataset has shape number_frames x 3 x 160 x 320 and uint8 type. One of the log hdf5-datasets is called cam1_ptr and addresses the alignment between camera frames and the other measurements.

Requirements

anaconda
tensorflow-0.9
keras-1.0.6
cv2

Hiring

Want a job at comma.ai?

Show us amazing stuff on this dataset

Credits

Riccardo Biasini, George Hotz, Sam Khalandovsky, Eder Santana, and Niel van der Westhuizen

research's Issues

TypeError: moments() got an unexpected keyword argument 'shift'

Hi guys,

I'm having this issue when running the autoencoder model following the tutorial

python train_generative_model.py autoencoder --batch 64

Using TensorFlow backend.
/usr/local/lib/python2.7/dist-packages/keras/backend/tensorflow_backend.py:158: UserWarning: Could not automatically initialize variable, make sure you do it manually (e.g. via `tf.initialize_all_variables()`).
  warnings.warn('Could not automatically initialize variable, '
Traceback (most recent call last):
  File "train_generative_model.py", line 159, in <module>
    g_train, d_train, sampler, saver, loader, extras = get_model(sess=sess, name=args.name, batch_size=args.batch, gpu=args.gpu)
  File "/media/vivian/DATA/research/models/autoencoder.py", line 141, in get_model
    G = generator(batch_size, gf_dim, ch, rows, cols)
  File "/media/vivian/DATA/research/models/autoencoder.py", line 49, in generator
    model.add(BN(mode=2, axis=3, name="g_bn0", gamma_init=mean_normal, epsilon=1e-5))
  File "/usr/local/lib/python2.7/dist-packages/keras/models.py", line 312, in add
    output_tensor = layer(self.outputs[0])
  File "/usr/local/lib/python2.7/dist-packages/keras/engine/topology.py", line 514, in __call__
    self.add_inbound_node(inbound_layers, node_indices, tensor_indices)
  File "/usr/local/lib/python2.7/dist-packages/keras/engine/topology.py", line 572, in add_inbound_node
    Node.create_node(self, inbound_layers, node_indices, tensor_indices)
  File "/usr/local/lib/python2.7/dist-packages/keras/engine/topology.py", line 149, in create_node
    output_tensors = to_list(outbound_layer.call(input_tensors[0], mask=input_masks[0]))
  File "/usr/local/lib/python2.7/dist-packages/keras/layers/normalization.py", line 122, in call
    epsilon=self.epsilon)
  File "/usr/local/lib/python2.7/dist-packages/keras/backend/tensorflow_backend.py", line 670, in normalize_batch_in_training
    shift=None, name=None, keep_dims=False)
TypeError: moments() got an unexpected keyword argument 'shift'

Anyone has the idea how to solve this?

What is the mapping between the frame and the steering angle

The camera collected data at 20 hz while the steering angle was logged at 100hz, so how can I get the one to one mapping of the steering angle and the corresponding frame.
Thanks

How to make HDF5 file?

Hi!

I am student learning about this neural network training stuff and I want to use some my own videos as training data. But I don't know how to change a video into a hdf5 format file. Actually I have read the hdf5 documentation but still quite confused about how can I generate it.

How do you make those videos and logs into hdf5 files?

Thank you so much if you can help me

bad marshal data errorin the view steering model.py

This error (ValueError: bad marshal data (unknown type code)) result when trying to execute the view steering model.py
here is the result from the cmd prompt

Traceback (most recent call last):
File "C:\Users\lenovo\Anaconda3\lib\site-packages\keras\utils\generic_utils.py
", line 229, in func_load
raw_code = codecs.decode(code.encode('ascii'), 'base64')
UnicodeEncodeError: 'ascii' codec can't encode character '\xe0' in position 46:
ordinal not in range(128)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "view_steering_model.py", line 94, in
model = model_from_json(json.load(jfile))
File "C:\Users\lenovo\Anaconda3\lib\site-packages\keras\models.py", line 349,
in model_from_json
return layer_module.deserialize(config, custom_objects=custom_objects)
File "C:\Users\lenovo\Anaconda3\lib\site-packages\keras\layers_init_.py", l
ine 55, in deserialize
printable_module_name='layer')
File "C:\Users\lenovo\Anaconda3\lib\site-packages\keras\utils\generic_utils.py
", line 144, in deserialize_keras_object
list(custom_objects.items())))
File "C:\Users\lenovo\Anaconda3\lib\site-packages\keras\models.py", line 1349,
in from_config
layer = layer_module.deserialize(conf, custom_objects=custom_objects)
File "C:\Users\lenovo\Anaconda3\lib\site-packages\keras\layers_init_.py", l
ine 55, in deserialize
printable_module_name='layer')
File "C:\Users\lenovo\Anaconda3\lib\site-packages\keras\utils\generic_utils.py
", line 144, in deserialize_keras_object
list(custom_objects.items())))
File "C:\Users\lenovo\Anaconda3\lib\site-packages\keras\layers\core.py", line
711, in from_config
function = func_load(config['function'], globs=globs)
File "C:\Users\lenovo\Anaconda3\lib\site-packages\keras\utils\generic_utils.py
", line 234, in func_load
code = marshal.loads(raw_code)
ValueError: bad marshal data (unknown type code)

NaN error when training autoencoder

I had hoped I could solve this for myself, but I regrettably couldn't, so I'm hoping someone here knows how to fix this:

When training the autoencoder as prescribed by the DriveSimulator.md file,

I get a NaN error by tensorflow. This is a completely unpredictable error and it happens in different epochs everytime i try to train again.

Here is my output:

Epoch 1/200
   64/10000 [..............................] - ETA: 1903s - g_loss: 4.4450 - d_loss: 5.4199 - d_loss_fake: 4.4598 - d_loss_legit: 0.9601 - time: 10.4118I tensorflow/core/common_runtime/gpu/pool_$
llocator.cc:244] PoolAllocator: After 2061 get requests, put_count=2041 evicted_count=1000 eviction_rate=0.489956 and unsatisfied allocation rate=0.543426
I tensorflow/core/common_runtime/gpu/pool_allocator.cc:256] Raising pool_size_limit_ from 100 to 110
 1280/10000 [==>...........................] - ETA: 343s - g_loss: 5.1531 - d_loss: 2.9220 - d_loss_fake: 1.1781 - d_loss_legit: 1.7439 - time: 2.3984I tensorflow/core/common_runtime/gpu/pool_al$
ocator.cc:244] PoolAllocator: After 5407 get requests, put_count=5279 evicted_count=1000 eviction_rate=0.18943 and unsatisfied allocation rate=0.212872
I tensorflow/core/common_runtime/gpu/pool_allocator.cc:256] Raising pool_size_limit_ from 256 to 281
 3584/10000 [=========>....................] - ETA: 221s - g_loss: 6.1948 - d_loss: 2.4484 - d_loss_fake: 1.0048 - d_loss_legit: 1.4435 - time: 2.1499W tensorflow/core/framework/op_kernel.cc:936]
 Invalid argument: Nan in summary histogram for: HistogramSummary
         [[Node: HistogramSummary = HistogramSummary[T=DT_FLOAT, _device="/job:localhost/replica:0/task:0/cpu:0"](HistogramSummary/tag, autoencoder/add_28/_221)]]
W tensorflow/core/framework/op_kernel.cc:936] Invalid argument: Nan in summary histogram for: HistogramSummary
         [[Node: HistogramSummary = HistogramSummary[T=DT_FLOAT, _device="/job:localhost/replica:0/task:0/cpu:0"](HistogramSummary/tag, autoencoder/add_28/_221)]]
W tensorflow/core/framework/op_kernel.cc:936] Invalid argument: Nan in summary histogram for: HistogramSummary
         [[Node: HistogramSummary = HistogramSummary[T=DT_FLOAT, _device="/job:localhost/replica:0/task:0/cpu:0"](HistogramSummary/tag, autoencoder/add_28/_221)]]
W tensorflow/core/framework/op_kernel.cc:936] Invalid argument: Nan in summary histogram for: HistogramSummary
         [[Node: HistogramSummary = HistogramSummary[T=DT_FLOAT, _device="/job:localhost/replica:0/task:0/cpu:0"](HistogramSummary/tag, autoencoder/add_28/_221)]]
W tensorflow/core/framework/op_kernel.cc:936] Invalid argument: Nan in summary histogram for: HistogramSummary
         [[Node: HistogramSummary = HistogramSummary[T=DT_FLOAT, _device="/job:localhost/replica:0/task:0/cpu:0"](HistogramSummary/tag, autoencoder/add_28/_221)]]
W tensorflow/core/framework/op_kernel.cc:936] Invalid argument: Nan in summary histogram for: HistogramSummary
         [[Node: HistogramSummary = HistogramSummary[T=DT_FLOAT, _device="/job:localhost/replica:0/task:0/cpu:0"](HistogramSummary/tag, autoencoder/add_28/_221)]]
E tensorflow/core/client/tensor_c_api.cc:485] Nan in summary histogram for: HistogramSummary
         [[Node: HistogramSummary = HistogramSummary[T=DT_FLOAT, _device="/job:localhost/replica:0/task:0/cpu:0"](HistogramSummary/tag, autoencoder/add_28/_221)]]
Traceback (most recent call last):
  File "./train_generative_model.py", line 168, in <module>
    nb_epoch=args.epoch, verbose=1, saver=saver
  File "./train_generative_model.py", line 85, in train_model
    g_loss, samples, xs = g_train(x, z, counter)
  File "/home/kamal/Desktop/research/models/autoencoder.py", line 241, in train_g
    outs = sess.run(outputs + updates, feed_dict={Img: images, Z: z, Z2: z2, K.learning_phase(): 1})
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 382, in run
    run_metadata_ptr)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 655, in _run
    feed_dict_string, options, run_metadata)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 723, in _do_run
    target_list, options, run_metadata)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 743, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors.InvalidArgumentError: Nan in summary histogram for: HistogramSummary
         [[Node: HistogramSummary = HistogramSummary[T=DT_FLOAT, _device="/job:localhost/replica:0/task:0/cpu:0"](HistogramSummary/tag, autoencoder/add_28/_221)]]
Caused by op u'HistogramSummary', defined at:
  File "./train_generative_model.py", line 159, in <module>

   g_train, d_train, sampler, saver, loader, extras = get_model(sess=sess, name=args.name, batch_size=args.batch, gpu=args.gpu)
  File "/home/kamal/Desktop/research/models/autoencoder.py", line 204, in get_model
    sum_e_mean = tf.histogram_summary("e_mean", E_mean)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/logging_ops.py", line 125, in histogram_summary
    tag=tag, values=values, name=scope)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/gen_logging_ops.py", line 100, in _histogram_summary
    name=name)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/op_def_library.py", line 703, in apply_op
    op_def=op_def)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 2310, in create_op
    original_op=self._default_original_op, op_def=op_def)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 1232, in __init__
    self._traceback = _extract_stack()

Again, this happens randomly in different epochs (1,3, 18, or 23). I can only get so far in the training before I get this error. Any ideas? I tried setting the learning rate to 0.0001 but this error persisted.

view_generative_model.py fail to generate animation

Hi,

I have successfully trained the transition model.
Now I'm going to generate the gif of the predicted image.
I followed instructions:

$ ./server.py --time 60 --batch 64
$ ./view_generative_model.py transition --name transition

I got an error:

loading weights...
Traceback (most recent call last):
  File "./view_generative_model.py", line 71, in <module>
    x = X.reshape((sh[0], args.time, 80, 160, 3))
ValueError: cannot reshape array of size 147456000 into shape (64,1,80,160,3)

I thought I may need to make the input size matched, I add time and batch options as:

$ ./view_generative_model.py transition --name transition --batch 64 --time 60

I got another error:

loading weights...
Sampling...
  0%|                                                   | 0/128 [00:00<?, ?it/s]
Traceback (most recent call last):
  File "./view_generative_model.py", line 83, in <module>
    code = sess.run([E_out[0]], feed_dict={I: x, K.learning_phase(): 1})[0]
  File "/home/tajima/.local/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 766, in run
    run_metadata_ptr)
  File "/home/tajima/.local/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 943, in _run
    % (np_val.shape, subfeed_t.name, str(subfeed_t.get_shape())))
ValueError: Cannot feed value of shape (3840, 80, 160, 3) for Tensor u'transition/input_1:0', which has shape '(960, 80, 160, 3)'

How can I make it work?

autoencoder (ONLY) model refactored for TensorFlow 1.10+

Can be helpful for people like me that wanna try to run with latest tensorflow. Here is the refactored autoencoder model [1] that works with latest tensorflow/keras.

1 - https://github.com/r7vme/autoencoder

About the dividing of training set and testing set

Hi,
I am using your dataset to train my own steering angle prediction model. However, since you have not publicly divided the dataset into training set and testing set, I wonder can you specify the dividing criterion so that people who use your data can compare their model in a unified manner. Thanks.
Regards,
Cardwing

How can i get the camera parameter about the pic?

Hi every one.
Thanks for your great work.
I am a student, and I am learning about egomotion.
I have read about the dataset of kitti and cityscapes ,and i want to format the dataset as them in the research sfmlearner. However, i am so sad to find it that i can't find the message about camera parameter.
Would you like to tell me how can i get them?
Thx.

Hsck

How can I train a model?

I don't see a train.py file?

ValueError: Filter must not be larger than the input: Filter: (8, 8) Input: (3, 160)

Hi,

I'm having this issue when running train steering model:

(snakes) guy_pavlov84@instance-1:~/research$ sudo ./train_steering_model.py --port 5557 --val_port 5556
Using TensorFlow backend.
Traceback (most recent call last):
  File "./train_steering_model.py", line 62, in <module>
    model = get_model()
  File "./train_steering_model.py", line 31, in get_model
    model.add(Convolution2D(16, 8, 8, subsample=(4, 4), border_mode="same"))
  File "/usr/local/lib/python2.7/dist-packages/keras/models.py", line 308, in add
    output_tensor = layer(self.outputs[0])
  File "/usr/local/lib/python2.7/dist-packages/keras/engine/topology.py", line 514, in __call__
    self.add_inbound_node(inbound_layers, node_indices, tensor_indices)
  File "/usr/local/lib/python2.7/dist-packages/keras/engine/topology.py", line 572, in add_inbound_node
    Node.create_node(self, inbound_layers, node_indices, tensor_indices)
  File "/usr/local/lib/python2.7/dist-packages/keras/engine/topology.py", line 149, in create_node
    output_tensors = to_list(outbound_layer.call(input_tensors[0], mask=input_masks[0]))
  File "/usr/local/lib/python2.7/dist-packages/keras/layers/convolutional.py", line 466, in call
    filter_shape=self.W_shape)
  File "/usr/local/lib/python2.7/dist-packages/keras/backend/tensorflow_backend.py", line 1579, in conv2d
    x = tf.nn.conv2d(x, kernel, strides, padding=padding)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/gen_nn_ops.py", line 394, in conv2d
    data_format=data_format, name=name)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/op_def_library.py", line 704, in apply_op
    op_def=op_def)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 2262, in create_op
    set_shapes_for_outputs(ret)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 1702, in set_shapes_for_outputs
    shapes = shape_func(op)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/common_shapes.py", line 246, in conv2d_shape
    padding)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/common_shapes.py", line 184, in get2d_conv_output_size
    (row_stride, col_stride), padding_type)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/common_shapes.py", line 149, in get_conv_output_size
    "Filter: %r Input: %r" % (filter_size, input_size))
ValueError: Filter must not be larger than the input: Filter: (8, 8) Input: (3, 160)

Any idea what that might be?
This is the setup:

(snakes) guy_pavlov84@instance-1:~/research$ pip freeze
funcsigs==1.0.2
h5py==2.6.0
Keras==1.1.0
mock==2.0.0
numpy==1.11.2
pbr==1.10.0
PyYAML==3.12
pyzmq==15.4.0
scipy==0.18.1
six==1.10.0
tensorflow==0.5.0
Theano==0.8.2
zmq==0.0.0

can not find keys start with 'app_'

Hi dear

After I download the comma-dataset , I can not find keys start with 'app_' in the log directory file which you wrote in your log.md.
is there something wrong with the comma-data.zip?
because I want to find which keys represent the gyro's x,y,z speed .
and which keys represent the Accelerometer‘s x,y,z value in the log's h5 file

it will be very appreciate if anyone can answer

thanks
weizhen

TypeError: slice indices must be integers or None or have an __index__ method

when i run " ./view_steering_model.py ./outputs/steering_model/steering_angle.json", it prompt me this error message:
Traceback (most recent call last):
File "./view_steering_model.py", line 120, in
draw_path_on(img, speed_ms, -angle_steers/10.0)
File "./view_steering_model.py", line 84, in draw_path_on
draw_path(img, path_x, path_y, color)
File "./view_steering_model.py", line 59, in draw_path
draw_pt(img, x, y, color)
File "./view_steering_model.py", line 55, in draw_pt
img[row-sz:row+sz, col-sz:col+sz] = color
TypeError: slice indices must be integers or None or have an index method
How to solve this problem?

can not understand transition some code. Help,thanks

I don't understand the code below. I have two question

1.why do this operation : x[0, -1] = imgs[0]
imgs[0] Represents actual number of frames?
I think that imgs[0] is the first predicted frame by T(transition) function, why Assign imgs[0] to x[0,-1]

2. What does this operation mean?
   what's the different between video[:, 0] and video[:,1]
   video = video.reshape((batch_size, 2, 80, 160, 3))
   return video[:, 0], video[:, 1]

here is code : --------------------------

    def sampler(z, x):
            video = np.zeros((128, 80, 160, 3))
            print "Sampling..."
            for i in range(128):
                    print i
                    x = x.reshape((-1, 80, 160, 3))
        
                    code = sess.run([E(I)[0]], feed_dict={I: x, K.learning_phase(): 1})[0]
                    code = code.reshape((batch_size, time+out_leng, z_dim))
                    inp = code[:, :time]
                    outs = T.predict(inp, batch_size=batch_size)
                    # imgs = G.predict(out, batch_size=batch_size)
                    imgs = sess.run([G_dec], feed_dict={out: outs, K.learning_phase(): 1})[0]
                    video[i] = imgs[0]
                    x = x.reshape((batch_size, time+out_leng, 80, 160, 3))
                    x[0, :-1] = x[0, 1:]
                    x[0, -1] = imgs[0]

           video = video.reshape((batch_size, 2, 80, 160, 3))
           return video[:, 0], video[:, 1]

server.py file

zmq.error.ZMQError:Address in use
This error results when trying to execute the server.py file
any help?

Did you use two cameras to capture the pictures or videos ?

Two cameras will help you to build an 3d-vector space to help you software get more precise data.

Think about two eyes vs one eye when you are driving.

I think it's little bit waste time to ML one eye data.

PS: do you have a mail list or google group about research?

are trained models available?

has anyone made the trained models available online?

about How to get the predicted steer given a new image input

hi, I've read the train_steer_model.py, but I didn't find the relation between that output shape and steer, could you please explain? As far as I see, it still ouput some kind of image, thanks!

model.add(Lambda(lambda x: x/127.5 - 1.,
input_shape=(ch, row, col),
output_shape=(ch, row, col)))

TypeError: slice indices must be integers or None or have an __index__ method

understanding "hypersphere"/"high density region" from paper

Beyond that, assuming that an autoencoder is correctly learned respecting the latent space Gaussianity, we can generate realistic looking videos as long as the transition model never leaves the high density region of the embedding space. This high density region is a hypersphere of radius ⇢, which in its turn is function of the dimensionality of embedding space and variance of Gaussian prior.

that makes it sound like all latent space encodings which "look realistic" when decoded by the generator have equal distance from some centroid. am i misunderstanding something here? isn't it trivial to stay in the "high density region" in that case?

Corrupt zipfile from get_data

When I run ./get_data.sh on my macbook, I get the following error printout.

--2016-08-23 10:08:34--  https://archive.org/download/comma-dataset/comma-dataset.zip
Resolving archive.org... 207.241.224.2
Connecting to archive.org|207.241.224.2|:443... connected.
HTTP request sent, awaiting response... 302 Moved Temporarily
Location: https://ia601507.us.archive.org/7/items/comma-dataset/comma-dataset.zip [following]
--2016-08-23 10:08:34--  https://ia601507.us.archive.org/7/items/comma-dataset/comma-dataset.zip
Resolving ia601507.us.archive.org... 207.241.227.117
Connecting to ia601507.us.archive.org|207.241.227.117|:443... connected.
HTTP request sent, awaiting response... 416 Requested Range Not Satisfiable

    The file is already fully retrieved; nothing to do.

Archive:  ../comma-dataset.zip
warning [../comma-dataset.zip]:  43980607958 extra bytes at beginning or within zipfile
  (attempting to process anyway)
error [../comma-dataset.zip]:  start of central directory not found;
  zipfile corrupt.
  (please check that you have transferred or created the zipfile in the
  appropriate BINARY mode and that you have compiled UnZip properly)

Are there any workarounds to this? Could I unzip the file manually?

ValueError: Shapes are not compatible when training transition model after autoencoder trained

Hi,

I trained the autoencoder successfully. However, when I was doing followup steps in training the transition model, I had the problem below.

Thanks a lot for any help!

BTW, I didn't change any code.

How was Dask slow?

In the dask_generator.py file you have the following comment:

This file is named after dask for historical reasons. We first tried to
use dask to coordinate the hdf5 buckets but it was slow and we wrote our own
stuff.

I'm curious, how were you trying to use Dask and what performance limitations did you run into?

Feel free to close this issue immediately. This is just a question of curiosity.

NO module named keras.models

~/research-master$ ./train_steering_model.py --port 5557 --val_port 5556
Traceback (most recent call last):
File "./train_steering_model.py", line 8, in
from keras.models import Sequential
ImportError: No module named keras.models

How to use to results generated by the transition model.

Hi,

We need video datasets and the log datasets to train the steering model.
And I am confused that how to use the result images generated by the transition model to train the steering model since we do not have the log dataset such as steering angles for those generated images.

THX

from utils import load, save

i know this is a stupid doubt but i googled this and couldnt find an answer, line 20 in autoencoder.py has
from utils import load, save. I am getting an error in this line.
I tried conda install utils, pip install utils but none of these seem to work.
How can i install this utils package.
Thanks

cannot unzip comma-dataset.zip on osx

i've compared the sha1 checksum to what's specified at https://archive.org/download/comma-dataset/comma-dataset_files.xml

➜  research git:(master) ✗ shasum -a 1 comma-dataset.zip
ae425f5c125a27f3b1f7ef29aeeda7f000ab4b73  comma-dataset.zip

so i've definitely got the right file..

but still when i try to unzip:

➜  research git:(master) ✗ unzip comma-dataset.zip 
Archive:  comma-dataset.zip
warning [comma-dataset.zip]:  43980607958 extra bytes at beginning or within zipfile
  (attempting to process anyway)
error [comma-dataset.zip]:  start of central directory not found;
  zipfile corrupt.
  (please check that you have transferred or created the zipfile in the
  appropriate BINARY mode and that you have compiled UnZip properly)

can anyone tell me what they used?

segmentation fault

when type ./view_generative_model.py ./outputs/steering_model/steering_angle.json
or
(tensorflow) root@dl126:/data/zeng/research-master# ./train_generative_model.py
Using TensorFlow backend.
I tensorflow/stream_executor/dso_loader.cc:108] successfully opened CUDA library libcublas.so locally
I tensorflow/stream_executor/dso_loader.cc:108] successfully opened CUDA library libcudnn.so locally
I tensorflow/stream_executor/dso_loader.cc:108] successfully opened CUDA library libcufft.so locally
I tensorflow/stream_executor/dso_loader.cc:108] successfully opened CUDA library libcuda.so locally
I tensorflow/stream_executor/dso_loader.cc:108] successfully opened CUDA library libcurand.so locally
Segmentation fault (core dumped)

---

i get this:
Segmentation fault (core dumped)

can anyone explain how to run the code? thx

Logs.md incomplete

I've not finished downloading the data yet but gear_choice doesn't tell me the option for drive (i'm assuming its an automatic)

Is the model in Comma 3 open sourced?

I was looking for the model and training pipeline from Herald's talk, are they open sourced, where can I find them?
https://www.youtube.com/watch?v=7u9sdenbI5M

Hey am new to this and don't understand much of so any help would be much appreciated

Hey am new to this and don't understand much of so any help would be much appreciated

Originally posted by @Liverpoolj in commaai/openpilot#19498 (comment)

Can not train generative transition model: Tensorflow complains about dimensions

After training the autoencoder model, I close the server used for that training and open a new server as specified. I then follow the next step for training the generative transition model, but Tensorflow immediately complains about mismatched (or undefined?) shapes as can be seen in the last line of the trace-back.

>>>>>>: ~/Desktop/research$ ./train_generative_model.py transition --batch 64 --name transition
Using TensorFlow backend.
I tensorflow/stream_executor/dso_loader.cc:108] successfully opened CUDA library libcublas.so locally
I tensorflow/stream_executor/dso_loader.cc:108] successfully opened CUDA library libcudnn.so locally
I tensorflow/stream_executor/dso_loader.cc:108] successfully opened CUDA library libcufft.so locally
I tensorflow/stream_executor/dso_loader.cc:108] successfully opened CUDA library libcuda.so.1 locally
I tensorflow/stream_executor/dso_loader.cc:108] successfully opened CUDA library libcurand.so locally
I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:925] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node 
zero

I tensorflow/core/common_runtime/gpu/gpu_init.cc:102] Found device 0 with properties: 
name: GeForce GTX 1060 6GB
major: 6 minor: 1 memoryClockRate (GHz) 1.7085
pciBusID 0000:01:00.0
Total memory: 5.93GiB
Free memory: 5.59GiB
I tensorflow/core/common_runtime/gpu/gpu_init.cc:126] DMA: 0 
I tensorflow/core/common_runtime/gpu/gpu_init.cc:136] 0:   Y 
I tensorflow/core/common_runtime/gpu/gpu_device.cc:839] Creating TensorFlow device (/gpu:0) -> (device: 0, name: GeForce GTX 1060 6GB, pci bus id: 0000:01:00.0)
Traceback (most recent call last):
  File "./train_generative_model.py", line 159, in <module>
    g_train, d_train, sampler, saver, loader, extras = get_model(sess=sess, name=args.name, batch_size=args.batch, gpu=args.gpu)
  File "/home/kamal/Desktop/research/models/transition.py", line 65, in get_model
    T = transition(batch_size)
  File "/home/kamal/Desktop/research/models/transition.py", line 42, in transition
    activation="tanh", batch_input_shape=(batch_size, time, z_dim)))
  File "/usr/local/lib/python2.7/dist-packages/keras/models.py", line 276, in add
    layer.create_input_layer(batch_input_shape, input_dtype)
  File "/usr/local/lib/python2.7/dist-packages/keras/engine/topology.py", line 371, in create_input_layer
    self(x)
  File "/usr/local/lib/python2.7/dist-packages/keras/engine/topology.py", line 515, in __call__
    self.add_inbound_node(inbound_layers, node_indices, tensor_indices)
  File "/usr/local/lib/python2.7/dist-packages/keras/engine/topology.py", line 573, in add_inbound_node
    Node.create_node(self, inbound_layers, node_indices, tensor_indices)
  File "/usr/local/lib/python2.7/dist-packages/keras/engine/topology.py", line 150, in create_node
    output_tensors = to_list(outbound_layer.call(input_tensors[0], mask=input_masks[0]))
  File "/home/kamal/Desktop/research/models/layers.py", line 371, in call
    input_length=self.output_length)
  File "/usr/local/lib/python2.7/dist-packages/keras/backend/tensorflow_backend.py", line 1195, in rnn
    sequence_length=None)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/rnn.py", line 1006, in _dynamic_rnn_loop
    output.set_shape(shape)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 405, in set_shape
    self._shape = self._shape.merge_with(shape)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/tensor_shape.py", line 570, in merge_with
    (self, other))
ValueError: Shapes (?, 64, 512) and (9, 2, 512) are not compatible

any ideas?

the code cannot move forward

INFO:main:server started
INFO:dask_generator:Loading 9 hdf5 buckets.
x 52722 | t 263583 | f 52722
x 58993 | t 294919 | f 58993
x 19731 | t 98719 | f 19731
x 56166 | t 280785 | f 56166
x 25865 | t 129344 | f 25865
x 85296 | t 426596 | f 85296
x 78463 | t 392182 | f 78463
x 30538 | t 152650 | f 30538
x 51691 | t 258571 | f 51691
training on 437158/459465 examples
INFO:dask_generator:camera files 9
843.83 ms
X (64, 1, 3, 160, 320)
angle (64, 1, 1)
speed (64, 1, 1)

???

How to interpret velodyne imu ?

Hi,

I was wondering how to interpret the 12 dimensional velodyne imu measurement. Is it a 12-way split of the 360 degree surrounding ?

Thanks

when i train"train_generation_model",i meet the problem

when i train"train_generation_model",i meet the problem
Traceback (most recent call last):
File "./train_generative_model.py", line 160, in
g_train, d_train, sampler, saver, loader, extras = get_model(sess=sess, name=args.name, batch_size=args.batch, gpu=args.gpu)
File "/home/sues/Desktop/research-master/research-master/models/autoencoder.py", line 171, in get_model
d_loss_legit = tf.reduce_mean(tf.nn.sigmoid_cross_entropy_with_logits(D_legit, tf.ones_like(D_legit)))
File "/home/sues/anaconda2/lib/python2.7/site-packages/tensorflow/python/ops/nn_impl.py", line 146, in sigmoid_cross_entropy_with_logits
_sentinel, labels, logits)
File "/home/sues/anaconda2/lib/python2.7/site-packages/tensorflow/python/ops/nn_ops.py", line 1533, in _ensure_xent_args
"named arguments (labels=..., logits=..., ...)" % name)
ValueError: Only call sigmoid_cross_entropy_with_logits with named arguments (labels=..., logits=..., ...)
is someone do it successfully?
thanks

Error during training the transition model: ValueError: could not broadcast input array from shape (0,3,160,320) into shape (60,3,160,320)

Environment: Anaconda + Keras 1.0.6 + Tensorflow 1.0.0 (Unable to find TF0.9.0 )

I ran the codes according to the Drive Simulator tutorial: https://github.com/commaai/research/blob/master/DriveSim.md

Everything is OK before the 5th step "Train transition model".

I ran the server for the transition model first, in window 1, and train the transition model in window 2: ./train_generative_model.py transition --batch 64 --name transition

Window 1 (ran the server code:./server.py --time 60 --batch 64) outputs the above error from time to time. As is shown in the screen shot:

Could anyone help me with this issue?
Best Regards! ：）

GAN+VAE: error when assigning value when training

Hi,

first, thank you so much for your Autoencoding beyond pixels using a learned similarity metric code !
I adapted it to my needs, and I when I run it a get an error because of the lines 221-222:

      with tf.control_dependencies(outputs):
        updates = [tf.assign(p, new_p) for (p, new_p) in D.updates]

I get the error:

    updates = [tf.assign(p, new_p) for (p, new_p) in D.updates]
  File "/local/gensoft2/exe/tensorflow/1.0.0-py2/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 516, in __iter__
    raise TypeError("'Tensor' object is not iterable.")
TypeError: 'Tensor' object is not iterable.`

Indeed, D.updates - where D is a keras.models.Model instance - is a list of tensors and not a list of tuples (keras 2.0.3).
So my questions are:

• What these lines are supposed to do and how should it be corrected ?
• Is there anything I'm missing ?
• Should it just be replaced by:

    with tf.control_dependencies(outputs):
        updates = [tf.assign(p, new_p) for (p, new_p) in D.updates]

Thank in advance !

Training transition model is too resource intensive, uses too much memory. Possible bug

After training the autoencode, i try to train the transition model as described by the same document.

using

./server.py --time 60 --batch 64

and

./train_generative_model.py transition --batch 64 --name transition

on two different tmux sessions.

Soon (a minute) after running the training command, the process is killed because my memory and swap (16 + 10 GB) are used up, and I'm still on epoch one.

Here is a dump:

/train_generative_model.py transition --batch 64 --name transition                                                                                                  [0/0]
Using TensorFlow backend.
I tensorflow/stream_executor/dso_loader.cc:108] successfully opened CUDA library libcublas.so locally
I tensorflow/stream_executor/dso_loader.cc:108] successfully opened CUDA library libcudnn.so locally
I tensorflow/stream_executor/dso_loader.cc:108] successfully opened CUDA library libcufft.so locally
I tensorflow/stream_executor/dso_loader.cc:108] successfully opened CUDA library libcuda.so.1 locally
I tensorflow/stream_executor/dso_loader.cc:108] successfully opened CUDA library libcurand.so locally
I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:925] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
I tensorflow/core/common_runtime/gpu/gpu_init.cc:102] Found device 0 with properties: 
name: GeForce GTX 1060 6GB
major: 6 minor: 1 memoryClockRate (GHz) 1.7085
pciBusID 0000:01:00.0
Total memory: 5.93GiB
Free memory: 5.58GiB
I tensorflow/core/common_runtime/gpu/gpu_init.cc:126] DMA: 0 
I tensorflow/core/common_runtime/gpu/gpu_init.cc:136] 0:   Y 
I tensorflow/core/common_runtime/gpu/gpu_device.cc:838] Creating TensorFlow device (/gpu:0) -> (device: 0, name: GeForce GTX 1060 6GB, pci bus id: 0000:01:00.0)
T.shape:  (64, 14, 512)
Transition variables:
transition/dreamyrnn_1_W:0
transition/dreamyrnn_1_U:0
transition/dreamyrnn_1_b:0
transition/dreamyrnn_1_V:0
transition/dreamyrnn_1_ext_b:0
Epoch 1/200
Killed

how to see the video

hi. I had download this date,What should I do to see this video?

velodyne_gps

Is there a shift in the GPS data? When I locate the coordinates using some map pages, the satellite images do not seem to match the images taken by comma ai team camera. Please advise.

Regards,

Riyadh

Suggestion: use "pygame.event.pump()" to prevent not responding.

In the for loop that blits the images to the screen, use "pygame.event.pump()" to prevent not responding in windows.

try mxnet?

Hello George,

Appreciate that you can share your code. Just got one suggestion, hope it's useful. I know that Keras is quite good and simple, but tensorflow is slow compared with other tools such as mxnet and caffe, even slower than theano. Recently i'm converting to mxnet and found that it can save GPU memory, keep high speed as well as support distributed computation(which i thought would be a trend in future self-driving car), besides, its net-built method is quite like Keras.
I'm rewriting your network with mxnet now and will finish it in a couple of weeks. Will send you the speed and memory comparison results after that.

slip_factor view_steering_model

in the function view_steering_angle.py you guys used three parameter to calculate the curvature. steer_ratio and wheel base is depend on physical property of car and can be find online. However, for the slip_factor, the comment said "it obtained from real data." I was wondering if you can explain how that value was calculated.
thanks

Training of Steering Model doesn't takes speed into account

Hi,
I was just going through the code. In the file train_steering_model.py, the function "gen" yields only the camera frame and the steering angle which is passed to the model.

https://github.com/commaai/research/blob/master/train_steering_model.py#L21

But I was wondering that steering angle position (more importantly the rate of change) does depend on speed.

Steering angle values

I've downloaded the dataset and analyzed some of the test drives.
Right now I'm wondering about the steering_angle value ranges.

They are going from about -4500 to 4500.
What is the unit of the recorded angles?

The dataset contains random images

I found the dataset contains random images, which look like an interior image with charger and monitors. Is the dataset hacked by someone? It is not clean...looks like a serious issue

Steering model does not generalize

Hi,
I was able to train the model using the given code and data. The loss on the training data converges to ~350 however it appears that on Dev data the loss remains high (around 3500). This implies model overfitting and failure to generalize.
Am I doing something wrong or are these the expected results using your code+data?

bad marshal data?

Steering Angle Driving Data

Hey guys,

I'm playing around with your steering model. Will soon equip my old Miata as data acquisition machine, but will be really grateful if you could share some 20-30K steering angle driving data for testing purposes. (I did collect thousand miles data for chffr :) )