Getting Started & Problem Definition

March 19, 2023 · View on GitHub

The purpose of Multi-Domain Fusion (MDF) is to train a unified model from multiple labeled domains sis_i, to obtain more generalizable representations FF, which would have minimum prediction error on the multiple different domains sis_i.

 

Getting Started & Task Challenges

  1. Data-level differences: Compared with 2D natural images that are composed of pixels with a consistent value range of [0, 255], 3D point clouds often are collected using different sensor types with different point cloud ranges and object-size distribution, which lead to distributional discrepancy among datasets.

  2. Taxonomy-level differences: Different autonomous driving manufacturers employ inconsistent class definitions and annotation granularity.

   

Getting Started & Training-Testing for MDF Setting

Here, we take Waymo-and-nuScenes consolidation as an example.

Joint-training stage: train a baseline detection model on the merged dataset:

  • Train with consistent point-cloud range (employing Waymo range) using multiple GPUs
  • Note that the multi-dataset 3D detection results will become very poor, if the point-cloud-range for different datasets is not aligned.
sh scripts/MDF/dist_train_multi_db_merge_loss.sh ${NUM_GPUs} \
--cfg_file ./cfgs/MDF/waymo_nusc/multi_db_pvrcnn_feat_3_merged.yaml
  • Train with consistent point-cloud range (employing Waymo range) using multiple machines
sh scripts/MDF/slurm_train_multi_db_merge_loss.sh ${PARTITION} ${JOB_NAME} ${NUM_NODES} \
--cfg_file ./cfgs/MDF/waymo_nusc/multi_db_pvrcnn_feat_3_merged.yaml
  • Train other baseline detectors such as PV-RCNN++ using multiple GPUs
sh scripts/MDF/dist_train_multi_db_merge_loss.sh ${NUM_GPUs} \
--cfg_file ./cfgs/MDF/waymo_nusc/multi_db_pvplus_feat_3_merged.yaml
  • Train other baseline detectors such as Voxel-RCNN using multiple GPUs
sh scripts/MDF/dist_train_multi_db_merge_loss.sh ${NUM_GPUs} \
--cfg_file ./cfgs/MDF/waymo_nusc/multi_db_voxel_rcnn_feat_3_merged.yaml

Joint-training stage: train a Uni3D detection model on the merged dataset:

  • Train with consistent point-cloud range (employing Waymo range) and C.A. (Coordinate-origin Alignment) using multiple GPUs
sh scripts/MDF/dist_train_mdf.sh ${NUM_GPUs} \
--cfg_file ./cfgs/MDF/waymo_nusc/waymo_nusc_pvrcnn_feat_3_uni3d.yaml \
--source_one_name waymo
  • Train with consistent point-cloud range (employing Waymo range) and C.A. (Coordinate-origin Alignment) using multiple machines
sh scripts/MDF/slurm_train_multi_db.sh ${PARTITION} ${JOB_NAME} ${NUM_NODES} \
--cfg_file ./cfgs/MDF/waymo_nusc/waymo_nusc_pvrcnn_feat_3_uni3d.yaml \
--source_one_name waymo
  • Train other baseline detectors such as PV-RCNN++ using multiple GPUs
sh scripts/MDF/dist_train_mdf.sh ${NUM_GPUs} \
--cfg_file ./cfgs/MDF/waymo_nusc/waymo_nusc_pvplus_feat_3_uni3d.yaml \
--source_one_name waymo
  • Train other baseline detectors such as Voxel-RCNN using multiple GPUs
sh scripts/MDF/dist_train_mdf.sh ${NUM_GPUs} \
--cfg_file ./cfgs/MDF/waymo_nusc/waymo_nusc_voxel_rcnn_feat_3_uni3d.yaml \
--source_one_name waymo

 

:fire: News :fire:

A simpler way to achieve the feature coupling and reuse for MDF setting:

  • Train with Domain Attention (DT) using multiple GPUs
sh scripts/MDF/dist_train_mdf.sh ${NUM_GPUs} \
--cfg_file ./cfgs/MDF/waymo_nusc/waymo_nusc_pvrcnn_feat_3_domain_attention.yaml \
--source_one_name waymo
  • Train with Domain Attention (DT) using multiple machines
sh scripts/MDF/slurm_train_multi_db.sh ${PARTITION} ${JOB_NAME} ${NUM_NODES} \
--cfg_file ./cfgs/MDF/waymo_nusc/waymo_nusc_pvrcnn_feat_3_domain_attention.yaml \
--source_one_name waymo
  • Train other baseline detectors such as PV-RCNN++ using multiple GPUs
sh scripts/MDF/dist_train_mdf.sh ${NUM_GPUs} \
--cfg_file ./cfgs/MDF/waymo_nusc/waymo_nusc_pvplus_feat_3_domain_attention.yaml \
--source_one_name waymo
  • Train other baseline detectors such as Voxel-RCNN using multiple GPUs
sh scripts/MDF/dist_train_mdf.sh ${NUM_GPUs} \
--cfg_file ./cfgs/MDF/waymo_nusc/waymo_nusc_voxel_rcnn_feat_3_domain_attention.yaml \
--source_one_name waymo

Training using another Backbone:

  • Train using multiple GPUs
sh scripts/MDF/dist_train_mdf.sh ${NUM_GPUs} \
--cfg_file ./cfgs/MDF/waymo_nusc/waymo_nusc_voxel_rcnn_resnet_feat_3_domain_attention.yaml \
--source_one_name waymo
  • Train multiple machines
sh scripts/MDF/slurm_train_multi_db.sh ${PARTITION} ${JOB_NAME} ${NUM_NODES} \
--cfg_file ./cfgs/MDF/waymo_nusc/waymo_nusc_voxel_rcnn_resnet_feat_3_domain_attention.yaml \
--source_one_name waymo

 

Evaluation stage: evaluate the unified detection model on different datasets:

  • Note that for the KITTI-related evaluation, please try --set DATA_CONFIG.FOV_POINTS_ONLY True to enable front view point cloud only. We report the best results on KITTI for testing all epochs on the validation set.

    • ${FIRST_DB_NAME} denotes that the fisrt dataset name of the merged two dataset, which is used to split the merged dataset into two individual datasets.

    • ${DB_SOURCE} denotes the dataset to be tested.

  • Test the models using multiple GPUs

sh scripts/MDF/dist_test_mdf.sh ${NUM_GPUs} \
--cfg_file ${CFG_FILE} \
--ckpt ${CKPT} \
--source_one_name ${FIRST_DB_NAME} \
--source_1 ${DB_SOURCE}
  • Test the models using multiple machines
sh scripts/MDF/slurm_test_mdb_mgpu.sh ${PARTITION} ${JOB_NAME} ${NUM_NODES} \
--cfg_file ${CFG_FILE} \
--ckpt ${CKPT} \
--source_one_name ${FIRST_DB_NAME} \
--source_1 ${DB_SOURCE}

 

All MDF Results:

Here, we report the Waymo-and-nuScenes consolidation results, and the model is evaluated on Waymo using the mAP/mAPH LEVEL_2 and nuScenes using the BEV/3D AP.

  • All LiDAR-based models are trained with 8 NVIDIA A100 GPUs and are available for download.
  • The multi-domain dataset fusion (MDF) training time is measured with 8 NVIDIA A100 GPUs and PyTorch 1.8.1.
  • For Waymo dataset training, we train the model using 20% training data for saving training time.
  • PV-RCNN-nuScenes represents that we train the PV-RCNN model only using nuScenes dataset, and PV-RCNN-DM indicates that we merge the Waymo and nuScenes datasets and train on the merged dataset. Besides, PV-RCNN-DT denotes the domain attention-aware multi-dataset training.
BaselineMDF MethodsWaymo@VehicleWaymo@PedestrianWaymo@CyclistnuScenes@CarnuScenes@PedestriannuScenes@Cyclist
PV-RCNN-nuScenesonly nuScenes35.59 / 35.213.95 / 2.550.94 / 0.9257.78 / 41.1024.52 / 18.5610.24 / 8.25
PV-RCNN-Waymoonly Waymo66.49 / 66.0164.09 / 58.0662.09 / 61.0232.99 / 17.553.34 / 1.940.02 / 0.01
PV-RCNN-DMDirect Merging57.82 / 57.4048.24 / 42.8154.63 / 53.6448.67 / 30.4312.66 / 8.121.67 / 1.04
PV-RCNN-Uni3DUni3D66.98 / 66.5065.70 / 59.1461.49 / 60.4360.77 / 42.6627.44 / 21.8513.50 / 11.87
PV-RCNN-DTDomain Attention67.27 / 66.7765.86 / 59.3861.38 / 60.3460.83 / 43.0327.46 / 22.0613.82 / 11.52
BaselineMDF MethodsWaymo@VehicleWaymo@PedestrianWaymo@CyclistnuScenes@CarnuScenes@PedestriannuScenes@Cyclist
Voxel-RCNN-nuScenesonly nuScenes31.89 / 31.653.74 / 2.572.41 / 2.3753.63 / 39.0522.48 / 17.8510.86 / 9.70
Voxel-RCNN-Waymoonly Waymo67.05 / 66.4166.75 / 60.8363.13 / 62.1534.10 / 17.312.99 / 1.690.05 / 0.01
Voxel-RCNN-DMDirect Merging58.26 / 57.8752.72 / 47.1150.26 / 49.5051.40 / 31.6815.04 / 9.995.40 / 3.87
Voxel-RCNN-Uni3DUni3D66.76 / 66.2966.62 / 60.5163.36 / 62.4260.18 / 42.2330.08 / 24.3714.60 / 12.32
Voxel-RCNN-DTDomain Attention66.96 / 66.5068.23 / 62.0062.57 / 61.6460.42 / 42.8130.49 / 24.9215.91 / 13.35
BaselineMDF MethodsWaymo@VehicleWaymo@PedestrianWaymo@CyclistnuScenes@CarnuScenes@PedestriannuScenes@Cyclist
PV-RCNN++ DMDirect Merging63.79 / 63.3855.03 / 49.7559.88 / 58.9950.91 / 31.4617.07 / 12.153.10 / 2.20
PV-RCNN++-Uni3DUni3D68.55 / 68.0869.83 / 63.6064.90 / 63.9162.51 / 44.1633.82 / 27.1822.48 / 19.30
PV-RCNN++-DTDomain Attention68.51 / 68.0569.81 / 63.5864.39 / 63.4362.33 / 44.1633.44 / 26.9421.64 / 18.52

   

Joint-training using Waymo, KITTI, and nuScenes:

  • Train with consistent point-cloud range (employing Waymo range) using multiple GPUs
sh scripts/MDF/dist_train_mdf_3db.sh ${NUM_GPUs} \
--cfg_file ./cfgs/MDF/KNW/knw_voxel_rcnn_feat_3_uni3d.yaml
  • Train with consistent point-cloud range (employing Waymo range) using multiple machines
sh scripts/MDF/slurm_train_multi_db_3db.sh ${PARTITION} ${JOB_NAME} ${NUM_NODES} \
--cfg_file ./cfgs/MDF/KNW/knw_voxel_rcnn_feat_3_uni3d.yaml
  • Train other baseline detectors such as PV-RCNN using multiple GPUs
sh scripts/MDF/dist_train_mdf_3db.sh ${NUM_GPUs} \
--cfg_file ./cfgs/MDF/KNW/knw_pvrcnn_feat_3_uni3d.yaml

Evaluation stage: evaluate the unified detection model on different datasets:

  • ${DB_SOURCE} denotes the dataset to be tested.

  • Test the models using multiple GPUs

sh scripts/MDF/dist_test_mdf_3db.sh ${NUM_GPUs} \
--cfg_file ${CFG_FILE} \
--ckpt ${CKPT} \
--source_1 ${DB_SOURCE}
  • Test the models using multiple machines
sh scripts/MDF/slurm_test_mdb_mgpu_3db.sh ${PARTITION} ${JOB_NAME} ${NUM_NODES} \
--cfg_file ${CFG_FILE} \
--ckpt ${CKPT} \
--source_1 ${DB_SOURCE}