DriveDreamer-2: LLM-Enhanced World Models for Diverse Driving Video Generation
December 18, 2024 · View on GitHub
Our team is actively working towards releasing the code for this project.
We appreciate your patience and understanding as we navigate the necessary processes.
Our new works, DriveDreamer4D and ReconDreamer, are released!
Project Page | Paper
Abstract
World models have demonstrated superiority in autonomous driving, particularly in the generation of multi-view driving videos. However, significant challenges still exist in generating customized driving videos. In this paper, we propose DriveDreamer-2, which builds upon the framework of DriveDreamer and incorporates a Large Language Model (LLM) to generate user-defined driving videos. Specifically, an LLM interface is initially incorporated to convert a user's query into agent trajectories. Subsequently, a HDMap, adhering to traffic regulations, is generated based on the trajectories. Ultimately, we propose the Unified Multi-View Model to enhance temporal and spatial coherence in the generated driving videos. DriveDreamer-2 is the first world model to generate customized driving videos, it can generate uncommon driving videos (e.g., vehicles abruptly cut in) in a user-friendly manner. Besides, experimental results demonstrate that the generated videos enhance the training of driving perception methods (e.g., 3D detection and tracking). Furthermore, video generation quality of DriveDreamer-2 surpasses other state-of-the-art methods, showcasing FID and FVD scores of 11.2 and 55.7, representing relative improvements of 30% and 50%.
News
- [2024/12/18] 🚀 Inference code and model weight for video generation are realsed!
- [2024/12/10] 🎉 DriveDreamer-2 is accepted for AAAI'25!.
- [2024/03/11] 🚀 We release the DriveDreamer-2 project! (Key features: multi-view video generation, user-friendly with LLM)
Getting Started
Download model weights and preprocessing file HERE.
Demo
Results with Gnerated Structural Information
Daytime / rainy day / at night, a car abruptly cutting in from the right rear of ego-car.
Rainy day, car abruptly cutting in from the left rear of ego-car. (long video)
Daytime, the ego-car changes lanes to the right side. (long video)
Rainy day, a person crosses the road in the front of the ego-car. (long video)
Results with nuScenes Structural Information
Daytime / rainy day / at night, ego-car drives through urban street, surrounded by a flow of vehicles on both sides.
Daytime / rainy day / at night, a bus is positioned to the left front of the ego-car, with a pedestrian near the bus.
Rainy day, the windshield wipers of the truck are continuously clearing the windshield.
Rainy day, the ego-car makes a left turn at the traffic signal, with vehicles behind proceeding straight through the intersection. (long video)
Daytime, the ego-car drives straight through the traffic light, with a truck situated to the left front and pedestrians crossing on the right side. (long video)
DriveDreamer-2 Framework
Bibtex
If this work is helpful for your research, please consider citing the following BibTeX entry.
@article{zhao2024drive,
title={DriveDreamer-2: LLM-Enhanced World Models for Diverse Driving Video Generation},
author={Zhao, Guosheng and Wang, Xiaofeng and Zhu, Zheng and Chen, Xinze and Huang, Guan and Bao, Xiaoyi and Wang, Xingang},
journal={arXiv preprint arXiv:2403.06845},
year={2024}
}