• Preparation
  • Installation
  • Datasets
• Getting Started
  • Training
  • Evaluation
• Customized Dataset
• Baseline Results
• References

Preparation

We implemented BEVerse as the baseline model for Track 2. The baseline model was trained on the official train split of the nuScenes dataset and evaluated on our robustness probing sets under different corruptions.

This codebase provides basic instructions for the reproduction of the baseline model in the RoboDrive Challenge.

:gear: Installation

Kindly refer to INSTALL.md to set up environments and download necessary checkpoints.

:hotsprings: Datasets

We use data under the nuScenes train split as the training set and the RoboDrive robustness probing data as the evaluation set. For training data preparation, kindly refer to PREPARE_DATASET.md.

For evaluation data preparation, kindly download the dataset from the following resources:

Type	Phase 1	Phase 2
Google Drive	link1 or link2	link1 or link2

Uncompress the downloaded dataset and organize the folder structure as follows:

.
├── ckpt
│   └── beverse_small.pth
├── data
│   ├── nuscenes
│   ├── robodrive-release
│   └── nuscenes_infos
│       ├── nuscenes_infos_train_mono3d.coco.json
│       ├── nuscenes_infos_train.pkl
│       ├── nuscenes_infos_trainval.pkl
│       ├── nuscenes_infos_val.pkl
│       └── robodrive_infos_test.pkl
├── projects
└── tools

Next, run the following command to generate the .pkl file for the evaluation sets:

bash tools/create_data.sh

:blue_car: Hint: You can download our generated rrobodrive_infos_test.pkl file from this Google Drive link.

The nuscenes folder should end up looking like this:

.
├── basemap
├── can_bus
├── can_bus.zip
├── expansion
├── lidarseg
├── maps
├── nuscenes_infos_temporal_train.pkl
├── nuscenes_infos_temporal_val.pkl
├── nuScenes-panoptic-v1.0-all
├── prediction
├── robodrive_infos_test.pkl
├── robodrive-v1.0-test
├── samples
├── sweeps
├── v1.0-mini
├── v1.0-test
└── v1.0-trainval

Getting Started

The training and evaluation instructions are summarized as follows.

:rocket: Training

Kindly refer to GET_STARTED.md for the details regarding model training.

:bar_chart: Evaluation

Simply run the following command to evaluate the trained baseline model on the RoboDrive robustness probing sets:

cd BEVerse
bash tools/dist_test_corruption.sh

The generated results will be saved in the folder structure as follows. Each results.pkl is a dictionary, its key is sample_idx and its value is np.ndarray.

.
├── brightness
│   └── results.pkl
├── color_quant
│   └── results.pkl
├── contrast
│   └── results.pkl
...
├── snow
└── zoom_blur

Next, kindly merge all the .pkl files into a single pred.pkl file and zip compress it.

You can merge the results using the following command:

python ./tools/convert_submit.py

:warning: Note: The prediction file MUST be named as pred.pkl. The .zip file can be named as you like.

Finally, upload the compressed file to Track 2's evaluation server for model evaluation.

:blue_car: Hint: We provided the baseline submission file at this Google Drive link. Feel free to download and check it for reference and learn how to correctly submit the prediction files to the server.

Customized Dataset

To customize your own dataset, simply build your dataset based on NuScenesCorruptionDataset from this link.

We mainly modified the data loading part. We only consider the subset of scenes for each corruption type, below is an example showing how to load a subset of scenes under each corruption type.

For more information, kindly refer to corruption_dataset.py.

data = mmcv.load(ann_file)
        
data_infos = data['infos']
sample_data_infos = []

for data_info in data_infos:
    if self.corruption is not None:
        if data_info['scene_token'] in self.sample_scenes_dict[self.corruption]:
            sample_data_infos.append(data_info)
        else:
            sample_data_infos.append(data_info)

You can modify the data path as follows from here:

if self.corruption is not None:
    for img_info in img_infos:
        for cam_name, cam_info in img_info.items():
            cur_path = cam_info['data_path']
            img_info[cam_name]['data_path'] = cur_path.replace('./data/nuscenes', osp.join(self.corruption_root, self.corruption))

Baseline Results

Phase 1

Corruption	mIoU
Bright	0.234
Dark	0.135
Fog	0.398
Frost	0.037
Snow	0.072
Contrast	0.057
Defocus Blur	0.194
Glass Blur	0.296
Motion Blur	0.212
Zoom Blur	0.105
Elastic Transform	0.406
Color Quant	0.140
Gaussian Noise	0.033
Impluse Noise	0.018
Shot Noise	0.028
ISO Noise	0.032
Pixelate	0.403
JPEG	0.319

Phase 2

Corruption	mIoU
Bright	0.214
Dark	0.141
Fog	0.193
Frost	0.068
Snow	0.032
Contrast	0.037
Defocus Blur	0.189
Glass Blur	0.279
Motion Blur	0.092
Zoom Blur	0.176
Elastic Transform	0.446
Color Quant	0.114
Gaussian Noise	0.052
Impluse Noise	0.021
Shot Noise	0.024
ISO Noise	0.096
Pixelate	0.368
JPEG	0.280

References

Kindly cite the corresponding paper(s) once you use the baseline model in this track.

@article{zhang2022beverse,
    title = {BEVerse: Unified Perception and Prediction in Bird's Eye View for Vision-Centric Autonomous Driving},
    author = {Zhang, Yunpeng and Zhu, Zheng and Zheng, Wenzhao and Huang, Junjie and Huang, Guan and Zhou, Jie and Lu, Jiwen},
    journal = {arXiv preprint arXiv:2205.09743},
    year = {2022}
}