Prepare Dataset

We prepare the nuScenes dataset similar to MagicDrive's instructions. Specifically,

1. Download the nuScenes dataset from the website and put them in data/. You should have these files:

data/nuscenes
├── maps
├── mini
├── samples
├── sweeps
├── v1.0-mini
└── v1.0-trainval

2. Generate ann_file for keyframes (samples) by:

python -m tools.create_data nuscenes \
--root-path data/nuscenes --out-dir data/nuscenes/nuscenes_mmdet3d-keyframes \
--extra-tag nuscenes --only_info

Or directly download our pre-generated .pkl files and place them into data/nuscenes/nuscenes_mmdet3d-keyframes/.

Either way, you will get the following structure:

data/nuscenes/nuscenes_mmdet3d-keyframes
├── nuscenes_infos_train.pkl
└── nuscenes_infos_val.pkl

3. Generate ann_file for 12Hz (sweeps) by:

• Firstly, follow ASAP to generate interp annotations for nuScenes.

  Note: The following codes in ASAP need to be modified:

  • In sAP3D/nusc_annotation_generator.py, please comment line357, and modify line101 to val_scene_ids = splits['val'] + splits['train'].

  • Modify the dataset path in scripts/ann_generator.sh to your custom dataset path.

  Then, you can run the following command in ASAP root:

  bash scripts/ann_generator.sh 12 --ann_strategy 'interp' 
  

  (Optional) Generate advanced annotations for sweeps. (We do not observe major difference between interp and advanced. You can refer to the implementation of ASAP. This step can be skipped.)

  Rename the generated folder to interp_12Hz_trainval and move it into your nuScenes dataset root data/nuscenes.

• Use the following command to generate ann_file with 12hz.

  python tools/create_data.py nuscenes --root-path data/nuscenes --out-dir data/nuscenes/nuscenes_mmdet3d-12Hz \
  --extra-tag nuscenes_interp_12Hz --max-sweeps -1 --version interp_12Hz_trainval
  

Or directly download our pre-generated .pkl files and place them into data/nuscenes/nuscenes_mmdet3d-12Hz/.

Either way, you will get the following structure:

data/nuscenes/nuscenes_mmdet3d-12Hz
├── nuscenes_interp_12Hz_infos_train.pkl
└── nuscenes_interp_12Hz_infos_val.pkl

4. (Optional but recommended) We recommend generating cache files in .h5 format of the BEV map to speed up the data loading process.

# generate map cache for val
python tools/prepare_map_aux.py +process=val +subfix=12Hz_interp

# generate map cache for train
python tools/prepare_map_aux.py +process=train +subfix=12Hz_interp

You will get these files:

data/nuscenes/nuscenes_map_aux_12Hz_interp
├── train_200x200_12Hz_interp.h5 (264G)
└── val_200x200_12Hz_interp.h5 (53G)

5. Download our pre-generated scene descriptions for each keyframe from here. Unzip the file and place the contents into the data/nuscenes/ directory.

Your directory structure should look like this:

data/nuscenes/description
├── scene-0001              # Scene Token
│   ├── 1e19d0a5189b46f4b62aa47508f2983e # Frame Token
│   │   └── description.json
│   ├── 2acfc4b99f084778bc484fa1eef1b79f # Frame Token
│   │   └── description.json
│   └── ...
├── scene-0002
└── ...

6. Download the nuscenes_interp_12Hz_infos_track2_eval.pkl and nuscenes_interp_12Hz_infos_track2_eval_long.pkl from w-coda2024/track2 or here for 16-frame videos FVD benchmark evaluation.

You will get these files:

data/nuscenes/workshop
├── nuscenes_interp_12Hz_infos_track2_eval.pkl
└── nuscenes_interp_12Hz_infos_track2_eval_long.pkl

7. Prepare Occupancy annotations as defined in Occ3D:

1. Download the occupancy annotations gts.tar.gz from here, then unzip it to the data/nuscenes directory.

2. Download the train/val pickle files and put them in data/nuscenes/

   • nuscenes_infos_train_temporal_v3_scene.pkl

     Main Link | Backup Link

   • nuscenes_infos_val_temporal_v3_scene.pkl

     Main Link | Backup Link

3. You will get the folder structure:

   data/nuscenes/
   ├── gts/
   |   |── scenes-0001/
   |   | ...
   │
   │──nuscenes_infos_train_temporal_v3_scene.pkl
   │──nuscenes_infos_val_temporal_v3_scene.pkl