This repo contains several python file that can be used to automatically create 2D counding box for vehicles from a camera view point in CARLA Simulator. To use this, you need to import carla_vehicle_annotator.py
to your python code and use the available functions. The other provided files are used to do supporting tasks that will be explained in this file.
IMPORTANT. Some parts of the provided code are not made by me and copied from CARLA example code. CARLA Simulator and its example code are licenced under the terms of MIT Licence. For more information about CARLA, I highly recommend you to visit their website https://carla.org.
There are several functions that is provided in this module. However, you only need these main functions to make this module works properly.
result, removed = auto_annotate(vehicles, camera, depth_img, max_dist=100, depth_margin=-1, patch_ratio=0.5, resize_ratio=0.5, json_path=None)
This function process the vehicles and image information and retrieve the 2D bounding box pixel coordinates for each visible vehicle with the corresponding vehicle’s class
Arguments:
vehicles
(list ofcarla.Actor
orcarla.ActorSnapShot
): list that contains vehicle actors/snapshots that you want to consider. You can pass list of all vehicles that have been spawned in Carla world. If you pass vehicles' snapshot, make sure that you addbounding_box
andtype_id
instances to the snapshot. Usesnap_processing( )
function to add the instances automatically.camera
(carla.Sensor
): the RGB camera object (carla sensor).depth_img
(float
2D numpy array): the depth map taken from the same view of RGB camera in meter unit. You can get it from the output of depth camera manually or by usingextract_depth( )
function.max_dist
(float
; default 100): the maximum distance parameter for distance filter. Increasing this value will allow the function to return vehicles that is located in greater distance, and thus with smaller bounding box.depth_margin
(float
; defaullt -1): the depth margin parameter for occlusion filter. If you pass negative value, the depth margin will be vary according to the vehicle’s dimension. Increasing this value can increase the number of positive results, but with more chance of having false positive result.patch_ratio
(float
[0,1] ; default 0.5): the patch ratio parameter for occlusion filter. Incresing this value might reduce the number of positive results, but with more cance of having false negative results.resize_ratio
(float
[0,1] ; default 0.5): the resize ratio parameter for occlusion filter. Set this to 1 if you don’t want to resize the bounding box for depth measurement.json_path
(string
; default None): the JSON .txt path that contains vehicle types classification (ex: ‘/vehicle_class_json_file.txt’). This argument must be filled if you want to assign a class label to each vehicle. Read explanation about vehicle_class_json_creator.py for information about labeling. Set this to None if you don’t need the label.
Return:
result
(dictionary): python dictionary that contains the bounding boxes, vehicle actors, and the corresponding label of the visible vehicles.removed
(dictionary): python dictionary that contains the bounding boxes, vehicle actor, and the corresponding label of the vehicles that are removed by the occlusion filter.
The result and removed dictionaries contain the following keys - values:
- key:
"vehicles"
- value (list ofcarla.Actor
): the list of vehicle actors. - key:
"bbox"
- value (list ofint
2D numpy array): the list of bounding boxes where each bounding box represented by two (min and max) corner points (in (x,y) format) of the box. The i-th element of the list is the bounding box for i-th vehicle in"vehicles"
. - key:
"class"
- value (list ofint
): the list of vehicle class. The i-th element of the list is the class label for i-th vehicle in"vehicles"
.
Notes:
- Make sure that the depth camera and RGB camera have the same attributes and transformation, and their data are taken in the same time.
- Make sure that the vehicles' data (transformation) are taken in the same frame with the RGB and Depth image. Open
collectData.py
to see the tested implementation example. The trick is to pass vehicle snapshot that you can get fromcarla.world.on_tick( )
instead of vehicle actor tovehicles
argument. Usingcarla.world.on_tick( )
allows you to get vehicles' data in the same frame with the other sensor data. Again, make sure that you addboudning_box
andtype_id
instances to the snapshot by usingsnap_processing( )
function. - If you find that the bounding box algorithm's performance is not satisfying, try to change the value of filter parameters:
depth_margin
,patch_ratio
,resize_ratio
. - You might wonder why you need the removed bounding boxes. The occlusion filter is not 100% accurate. Therefore, you might want to have the list bounding boxes removed by the occlusion filter so that you can return the removed bounding box if you find that it is a false removal.
- For a quick review about how the algorithm works, I recommend you to visit my page.
void save_output(carla_img, bboxes, vehicle_class=None, old_bboxes=None, old_vehicle_class=None, cc_rgb=carla.ColorConverter.Raw, path=‘’, save_patched=False, add_data=None, out_format=‘pickle’)
Use this function to save the result of auto_annotate( )
into your local directory. This function will save the RGB image and RGB image with drawn bounding boxes in .jpg format and the bounding boxes information in JSON .txt or pickle format.
Arguments:
carla_img
(carla.Image
): object (carla image) returned by the RGB camera at the time the bounding boxes are calculated.bboxes
(list offloat
2D numpy array): list of bounding boxes coordinates that you want to save. You can get it from the returned valueresult
ofauto_annotate( )
with key"bbox"
.vehicle_class
(list ofint
; default None): the list of visible vehilcles’ class. You can get it from the returned valueresult
ofauto_annotate( )
with key"class"
. Set this to None if you don’t want to save this information.old_bboxes
(list ofint
2D numpy array ; default None): list of bounding boxes that are removed by the occlusion filter. You can get it from the returned valueremoved
ofauto_annotate( )
with key"bbox"
. Set this to None if you don’t want to save this information.old_vehicle_class
(list ofint
; default None): the list of vehicles’ class that are removed by the occlusion filter. You can get it from the returned valueremoved
ofauto_annotate( )
with key"class"
. Set this to None if you don’t want to save this information.cc_rgb
(carla.ColorConverter
; defaultcarla.ColorConverter.Raw
): image color style that you want to use for the RGB image.path
(string ; default ‘’): the folder path where you want to save the result (ex: ‘/this/path/‘).save_patched
(bool
; default False): set this to True if you want to save the image with drawn bounding boxes.add_data
(any ; default None): fill this argument if you want to save any other additional information.out_format
(string ; default ‘pickle’): the file format to save the result and removed bounding boxes and class and also the additional informationadd_data
. Only support ‘json’ (.txt) and ‘pickle’ (.pkl). Any format you choose, this information will be packed as python dictionary when you import the file to your python program.
This function will create three folders in path
, which are out_rgb that contains RGB image, out_bbox that contains bounding boxes and other data formatted in pickle or JSON, and out_rgb_bbox that contains RGB image with drawn bounding boxes. Folder out_rgb_bbox will only be created if you set parameter save_patched
to True. Data taken from the same moment are named with the same name in these three folders, so that you can find which bounding boxes file that corresponds to which image easily. Information that packed in out_bbox’s files are packed as python dictionary. This dictionary contains these keys - values:
- key:
"bboxes"
- value: data passed tobboxes
argument - key:
"vehicle_class"
- value: data passed tovehicle_class
argument - key:
"removed_bboxes"
- value: data passed toold_bboxes
argument - key:
"removed_vehicle_class"
- value: data passed toold_vehicle_class
argument - key:
"others"
- value: data passed toadd_data
argument
Notes:
- Make sure you put ending slash ‘/‘ behind the folder path that you pass into
path
argument. - If you want to save bounding boxes data in JSON format, make sure data that you passed to
add_data
argument (if any) is compatible with JSON format.
void save2darknet(bboxes, vehicle_class, carla_img, data_path = '', cc_rgb = carla.ColorConverter.Raw, save_train = False, customName = '')
This function will save your images and the corresponding bounding boxes according to darknet's training data format, so you can use it for ,as example, train your YOLOv4 model. For more information about darknet and its data format, I recommend you to visit darknet github.
Arguments:
bboxes
(list offloat
2D numpy array): list of bounding boxes coordinates that you want to save. You can get it from the returned valueresult
ofauto_annotate( )
with key"bbox"
.vehicle_class
(list ofint
; default None): the list of visible vehilcles’ class. You can get it from the returned valueresult
ofauto_annotate( )
with key"class"
.carla_img
(carla.Image
): object (carla image) returned by the RGB camera at the time the bounding boxes are calculated.data_path
(string
; default''
): path wheredarknet.exe
is located. It should be located in[DARKNET_PATH]/build/darknet/x64/
. This function willl create new folder/data
(relative todata_path
) that contains folderobj
and filetrain.txt
(if you setsave_train
to True).cc_rgb
(carla.ColorConverter
; defaultcarla.ColorConverter.Raw
): image color style that you want to use for the RGB image.save_train
(bool
; defaultFalse
): set this value toTrue
if you want to createtrain.txt
file in/data
. If set this to true, this function will scan all JPG files that exist in/data/obj/
and will recreatetrain.txt
according to the detected JPG files. Therefore I suggest you to use this feature only in the end of your data aggregation process. You can setbboxes
,vehicle_class
orcarla_img
to None if you want to use this feature withput creating new training data.customName
(string
; default''
): string passed to this argument will be added to the image name. The output image's name format will becustomName_frameNumber.jpg
. If this argument is set to default, the output image's name format will beframeNumber.jpg
.
Notes:
- You have to create
obj.names
andobj.names
manually. - Remember that the bounding boxes created by this module is not 100% accurate. Therefore, I recommend you to use this program provided by darknet to review or edit the bounding boxes.
Use this function to convert depth carla.Image
that you get from depth camera into depth map in meter unit, so that you can pass it to auto_annotate( )
function.
Argument:
depth_img
(carla.Image
): carla image object that come from depth sensor measurement
Return:
depth_meter
(float
2D numpy array): depth map of the corresponding input in meter unit.
Use this function to add bounding_box
and type_id
instances to vehicles snapshot based on actual vehicles actor data.
Argument:
vehiclesActor
(list ofcarla.Actor
): list of vehicle actors that you are interested in, which you can get fromcarla.world.get_actors( )
function.worldSnap
(carla.WorldSnapshot
): The world snapshot that contains your vehicles snapshot.
Return:
vehicles
(list ofcarla.ActorSnapshot
): List of snapshot of vehicles that exist in bothvehiclesActor
andworldSnap
. The snapshots have two additional instances, which arebounding_box
andtype_id
that are taken from actor information.
Run this program to test how the algorithm works. This program will load interactive window that contains view from a car camera perspective. You can control the car manually so that you can evaluate the performance of the algorithm directly. You can change the values of filter parameters and see how changing these parameters can effect the performance of the filter.
Run this program to create vehicle_class_json_file.txt. This file is a JSON formatted data that map each vehicle types available in Carla to one integer label. Definition of each label is also packed in the JSON file. The default label mapping file has been provided in my repo. But you can create your own label mapping file by running and edit this program.
This program has two variables that you can modify. One of them is autoFill
, which if you set to True, the program will automatically create mapping file that maps all vehicle types to class label 0. The other one is class_ref
, which is a dictionary that defines the definition of each class label. After you run the program (if autoFill
is set to False), check your Carla window and it will show each vehicle types. All you have to do is type the vehicle’s class in your python window. I think it is quite intuitive.
This program is an example of working carla_vehicle_annotator
implementation. This program will capture sensors data every 1 second in simulation time. The sensors are RGB camera (with Bouncing Box), depth camera, segmentation camera, and LIDAR camera. If you think that the bounding box results is poor, you can change the filter parameters in auto_annotate( )
function to get better result.
That’s all I have for you. Have fun with CARLA and keep supporting CARLA project. Thank you.