aicps / hydrafusion

Model code for our paper titled "HydraFusion: Context-Aware Selective Sensor Fusion for Robust and Efficient Autonomous Vehicle Perception"

HydraFusion

Code for our paper titled "HydraFusion: Context-Aware Selective Sensor Fusion for Robust and Efficient Autonomous Vehicle Perception," accepted to be published in ICCPS 2022.

This repository contains the algorithmic implementation of our HydraFusion model. Our model is intended to be used with the RADIATE dataset available here: https://pro.hw.ac.uk/radiate/

Model

hydranet.py -- contains the class HydraFusion, which defines our top-level model specification.

stem.py -- defines the stem modules in HydraFusion

branch.py -- defines the branches implemented in our model.

gate.py -- contains the gating module implementations.

fusion.py -- contains the definition of the fusion block along with the algorithms to fuse the bounding boxes output by each active branch.

The stems and branches are built using a split architecture implementation of Faster R-CNN with a ResNet-18 backbone. HydraFusion can be used with any image-based multi-modal dataset. In our evaluations we used two cameras, one radar sensor, and one lidar sensor as inputs to the model.

Requirements

PyTorch 1.9

hydrafusion's Issues

About training and eval

Dear author

I appreciate your work. I found some problems when I tried to refactor your code, may I ask how you get the boundingbox label for the 2D image. The official code given by radiate when getting the 3D boundingbox May I ask how this is converted. The second problem is that when I validate it, I find a lot of empty boundingboxes. Can you provide me with the training and validation code if possible?

Best whishes
Zhenrong

IndexError: too many indices for tensor of dimension 1

Hi sir,

Sorry to bother you.
I am also trying to train radiate's dataset using your model.
I followed one of your closed issues, 'We used the radiate sdk function get_from_timestamp to get the sensor data for each input modality and corresponding annotations. https://github.com/marcelsheeny/radiate_sdk/blob/master/radiate.py#L187. We used annotations['radar_cartesian'] as the radar_y and annotations['camera_right_rect'] as cam_y. However, since Faster R-CNN is a 2D bounding box predictor, we flatten the pseudo-3D camera annotations for cam_y into 2D boxes.'
I tried using this way but it does not seem to work.
Right now, I am passing in this 'data[iteration][instance].gt_box', code below

with EventStorage(start_iter) as storage:
for data, iteration in zip(data_loader, range(start_iter, max_iter)):
storage.iter = iteration
print("Printing Data")
#print(data[iteration]['image'])
print(iteration)
print(data[iteration]['instances'].gt_boxes)
loss_dict = model(None, None, data[iteration]['image'], None, None, None, data[iteration]['instances'].gt_boxes, None)

However, I am getting this error 'IndexError: too many indices for tensor of dimension 1'.
Could you give me some pointers on what I may be passing in wrong and what should be correct way to pass into the forward function?
Thank you.

Best Regards

How to use functions in gate

Could you offer a demo which shows how the functions in gate.py to use?
It doesn't look like it's been used in hydranet.py
Thanks.

Questions from newcomers

Dear author, hello. I am a newcomer in the field of autonomous driving technology. While reading the paper, I discovered your efforts in dynamic strategy algorithms, which I believe are valuable. I would like to integrate the algorithms into my fusion algorithm, but I currently do not know how to run your code or how to use the Kitti dataset as the correct input. I hope to be able to help answer any questions during my time.

config file

could you add an example config file? for the input to the class Hydrafusion(config).

so we know what all parameters should be in it? it doesn't look like it's the same as the RADIATE config.yaml file

Train and Test script files are missing

Train and test script files are missing in the repository. Can someone upload train and test scripts for HydraFusion model?