Cannot use segmentation images as data set. about autonomousdrivingcookbook

[shrimalmadhur]

I believe that map is not that great for segmentation since the mesh of road and the side walk is the same, so when you get the segmentation and train on that, you might loose features which the side walk also provides. I guess if the map has better segmentation you could train a better model but still I think normal RGB image has more deep features than segmentation.

[BrianN92]

Thanks! But as I mentioned, the shadow of trees is an issue when using normal RGB images as data set. So I think the segmentation might be a solution for that.

[shrimalmadhur]

Yup. I agree good segmentation image can be helpful. Maybe there's way we can utilize both of them, so that we get good features from RGB and use the segmentation to solve the shadow issue. But I don't think so that only segmentation can be that useful.

[mitchellspryn]

@BrianN92 What, precisely, are you trying to do? Are you trying to train the end-to-end model on a dataset you collected from the neighborhood map?

[BrianN92]

@mitchellspryn Yes, I am trying to train the E2E model on the neighborhood map. Sorry for the vagueness of this question.

[mitchellspryn]

Ok. In that case, using the segmentation images as input to your model is suspicious. Consider that in a real scenario, your vehicle will not have access to the segmentation images. By requiring that as an input to your net, you'd be developing a model that only works in a simulator and would not translate to the real world.

A more generalizable approach would be to create a transformation that adds random patches of shadows on existing images. A simple way of doing this could be to select a random set of connected pixels from the RGB image, transform the pixel to HSV space, decrease the value by a set multiplier (i.e. 'value = value * 0.9'), and transforming back to RGB space. By doing this, the model will learn that the shadows don't really mean anything, and will become more robust to the noise.

[BrianN92]

Thanks, I will try it!

[BrianN92]

Hi @mitchellspryn , after the image transformation, like the images I attached below, I have another question.

How can I make the names of images after transformation corresponding to the "ImageName" tag in every "airsim_rec.txt" file? Thank you!

[mitchellspryn]

I assume you wrote another script that created the transformations? If so, you'll probably be better off directly generating the .h5s for training. Look in "cooking.py" and mimic that logic in your preprocessing script.

In fact, if you really wanted to get slick, you could probably add this as an additional transform inside of Generator.py. You'll see the set of transforms we've already coded inside of random_transform_with_states(); you can just add yours there.