opengvlab / drivemlm

DriveMLM

Large language models (LLMs) have opened up new possibilities for intelligent agents, endowing them with human-like thinking and cognitive abilities. In this work, we delve into the potential of large language models (LLMs) in autonomous driving (AD). We introduce DriveMLM, an LLM-based AD framework that can perform close-loop autonomous driving in realistic simulators. To this end, (1) we bridge the gap between the language decisions and the vehicle control commands by standardizing the decision states according to the off-the-shelf motion planning module. (2) We employ a multi-modal LLM (MLLM) to model the behavior planning module of a module AD system, which uses driving rules, user commands, and inputs from various sensors (\eg, camera, lidar) as input and makes driving decisions and provide explanations; This model can plug-and-play in existing AD systems such as Apollo for close-loop driving. (3) We design an effective data engine to collect a dataset that includes decision state and corresponding explanation annotation for model training and evaluation. We conduct extensive experiments and show that our model achieves 76.1 driving score on the CARLA Town05 Long, and surpasses the Apollo baseline by 4.7 points under the same settings, demonstrating the effectiveness of our model. We hope this work can serve as a baseline for autonomous driving with LLMs.

🗓️ Schedule

• Release dataset and annotations
• Release code and models

🏠 Overview

🎁 Major Features

• Following human instruction.
  
  
• Handling more scenarios.
  
  
• Examples on nuScenes.
  
  

🎫 License

This project is released under the Apache 2.0 license.

🖊️ Citation

If you find this project useful in your research, please consider cite:

@article{wang2023drivemlm,
  title={DriveMLM: Aligning Multi-Modal Large Language Models with Behavioral Planning States for Autonomous Driving},
  author={Wang, Wenhai and Xie, Jiangwei and Hu, ChuanYang and Zou, Haoming and Fan, Jianan and Tong, Wenwen and Wen, Yang and Wu, Silei and Deng, Hanming and Li, Zhiqi and others},
  journal={arXiv preprint arXiv:2312.09245},
  year={2023}
}

drivemlm's Issues

Code

Hello, I really want to refer to your work, can you release the code?

About training resources

I wonder how many GPUs and how much time are needed to train your engine.

How is Apollo Baseline Implemented and Evaluated in CARLA town05 Long?

Hi,

Thanks for sharing your great work. I'd like to know more about how you implemented Apollo as the baseline in closed-loop driving on the CARLA Town05 Long benchmark.

Is there any material/tutorial that I can look up to reproduce the results reported in the paper?

Thanks.

Output format

Hi, thanks for sharing your excellent work here. May I know the output of your paper? It should be low-level control signals like course and throttle, right? But in your visualizations, I cannot see low-level controls so I am a little confused here.

Code release

Wonderful work, are there any open source plans for the code?