Language Prompt for Autonomous Driving

Dongming Wu*, Wencheng Han*, Tiancai Wang, Yingfei Liu, Xiangyu Zhang, Jianbing Shen

:fire: Introduction

This is the official implementation of Language Prompt for Autonomous Driving.

• We propose a new large-scale language prompt set for driving scenes, named NuPrompt. As far as we know, it is the first dataset specializing in multiple 3D objects of interest from video domain.
• We construct a new prompt-based driving perceiving task, which requires using a language prompt as a semantic cue to predict object trajectories.
• We develop a simple end-to-end baseline model, called PromptTrack, which effectively fuses cross-modal features in a newly built prompt reasoning branch to predict referent objects, showing impressive performance.

:boom: News

• [2024.12.10] This work is accepted by AAAI 2025. A new version of the paper will be released soon.
• [2024.06.27] Data and code are released. Welcome to try it!
• [2023.09.11] Our paper is released at arXiv.

:star: Benchmark

We expand nuScenes dataset with annotating language prompts, named NuPrompt. It is a large-scale dataset for language prompt in driving scenes, which contains 40,147 language prompts for 3D objects. Thanks to nuScenes, our descriptions are closed to real-driving nature and complexity, covering a 3D, multi-view, and multi-frame space.

The data can be downloaded from NuPrompt.

:hammer: Model

Our model is built upon PF-Track.

Please refer to data.md for preparing data and pre-trained models.

Please refer to environment.md for environment installation.

Please refer to training_inference.md for training and evaluation.

:rocket: Results

Method	AMOTA	AMOTP	RECALL	Model	Config
PromptTrack	0.200	1.572	32.5%	model	config

:point_right: Citation

If you find our work useful in your research, please consider citing them.

@inproceedings{wu2025language,
  title={Language prompt for autonomous driving},
  author={Wu, Dongming and Han, Wencheng and Liu, Yingfei and Wang, Tiancai and Xu, Cheng-zhong and Zhang, Xiangyu and Shen, Jianbing},
  booktitle={Proceedings of the AAAI Conference on Artificial Intelligence},
  volume={39},
  number={8},
  pages={8359--8367},
  year={2025}
}

@inproceedings{wu2023referring,
  title={Referring multi-object tracking},
  author={Wu, Dongming and Han, Wencheng and Wang, Tiancai and Dong, Xingping and Zhang, Xiangyu and Shen, Jianbing},
  booktitle={CVPR},
  year={2023}
}

:heart: Acknowledgements

We thank the authors that open the following projects.

• MMDetection3d
• nuScenes
• PF-Track
• PETR
• MOTR