Getting Started & Problem Definition

April 26, 2023 · View on GitHub

Different from UDA task, the purpose of an Active Domain Adaptation (ADA) task is to pick up a subset of unlabeled target domain tt to perform the manual annotation process, such that we can achieve a good trade-off between high performance and low annotation cost, where labeled training data from the source domain ss (such as point cloud or images) are assumed to be available for initializing the training model.

   

Getting Started & Training-Testing for ADA setting

Here, We take Waymo-to-KITTI adaptation as an example.

Pretraining stage: train the source-only model on the labeled source domain:

  • Train FEAT=3 (X,Y,Z) with SN (statistical normalization) using multiple GPUs

    sh scripts/dist_train.sh ${NUM_GPUs} \ 
 --cfg_file ./cfgs/DA/waymo_kitti/source_only/pvrcnn_old_anchor_sn_kitti.yaml

  • Train FEAT=3 (X,Y,Z) with SN (statistical normalization) using multiple machines

    sh scripts/slurm_train.sh ${PARTITION} ${JOB_NAME} ${NUM_NODES} \ 
 --cfg_file ./cfgs/DA/waymo_kitti/source_only/pvrcnn_old_anchor_sn_kitti.yaml

  • Train FEAT=3 (X,Y,Z) without SN (statistical normalization) using multiple GPUs

    sh scripts/dist_train.sh ${NUM_GPUs} \ 
 --cfg_file ./cfgs/DA/waymo_kitti/source_only/pvrcnn_feat_3_vehi.yaml

  • Train FEAT=3 (X,Y,Z) without SN (statistical normalization) using multiple machines

    sh scripts/slurm_train.sh ${PARTITION} ${JOB_NAME} ${NUM_NODES} \ 
 --cfg_file ./cfgs/DA/waymo_kitti/source_only/pvrcnn_feat_3_vehi.yaml

  • Train other baseline detectors such as Voxel R-CNN using multiple GPUs

    sh scripts/dist_train.sh ${NUM_GPUs} \ 
 --cfg_file ./cfgs/DA/waymo_kitti/source_only/voxel_rcnn_feat_3_vehi.yaml

  • Train other baseline detectors such as Voxel R-CNN using multiple machines

    sh scripts/slurm_train.sh ${PARTITION} ${JOB} ${NUM_NODES} \ 
 --cfg_file ./cfgs/DA/waymo_kitti/source_only/voxel_rcnn_feat_3_vehi.yaml

Evaluate the source-pretrained model:

  • Note that for the cross-domain setting where the KITTI dataset is regarded as the target domain, please try --set DATA_CONFIG_TAR.FOV_POINTS_ONLY True to enable front view point cloud only. We report the best model for all epochs on the validation set.

  • Test the source-only models using multiple GPUs

    sh scripts/dist_test.sh ${NUM_GPUs} \ 
 --cfg_file ./cfgs/DA/waymo_kitti/source_only/pvrcnn_feat_3_vehi.yaml \ 
 --ckpt ${CKPT}

  • Test the source-only models using multiple machines

    sh scripts/slurm_test_mgpu.sh ${PARTITION} ${NUM_NODES} \ 
 --cfg_file ./cfgs/DA/waymo_kitti/source_only/pvrcnn_feat_3_vehi.yaml \ 
 --ckpt ${CKPT}

  • Test the source-only models of all ckpts using multiple GPUs

    sh scripts/dist_test.sh ${NUM_GPUs} \ 
 --cfg_file ./cfgs/DA/waymo_kitti/source_only/pvrcnn_feat_3_vehi.yaml \ 
 --eval_all

  • Test the source-only models of all ckpts using multiple machines

    sh scripts/slurm_test_mgpu.sh ${PARTITION} ${NUM_NODES} \ 
 --cfg_file ./cfgs/DA/waymo_kitti/source_only/pvrcnn_feat_3_vehi.yaml \ 
 --eval_all

Bi3D Adaptation stage 1: active source domain data

  • You need to set the --pretrained_model ${PRETRAINED_MODEL} when finish the pretraining model stage

  • Train with SN (statistical normalization) using multiple GPUs

    sh scripts/ADA/dist_train_active_source.sh ${NUM_GPUs} \ 
 --cfg_file ./cfgs/ADA/waymo-kitti/pvrcnn/active_source_only.yaml \ 
 --pretrained_model ${PRETRAINED_MODEL}

  • Train with SN (statistical normalization) using multiple machines

    sh scripts/ADA/slurm_train_active_source.sh ${PARTITION} ${JOB_NAME} ${NUM_NODES} \ 
 --cfg_file ./cfgs/ADA/waymo-kitti/pvrcnn/active_source_only.yaml \ 
 --pretrained_model ${PRETRAINED_MODEL}

  • Train without SN (statistical normalization) using multiple GPUs

    sh scripts/ADA/dist_train_active_source.sh ${NUM_GPUs} \ 
 --cfg_file ./cfgs/ADA/waymo-kitti/pvrcnn/active_source_only_wosn.yaml \ 
 --pretrained_model ${PRETRAINED_MODEL}

  • Train without SN (statistical normalization) using multiple machines

    sh scripts/ADA/slurm_train_active_source.sh ${PARTITION} ${JOB_NAME} ${NUM_NODES} \ 
 --cfg_file ./cfgs/ADA/waymo-kitti/pvrcnn/active_source_only_wosn.yaml \ 
 --pretrained_model ${PRETRAINED_MODEL}

Bi3D Adaptation stage 2: active target domain data

  • You need to set the --pretrained_model ${PRETRAINED_MODEL} when finish the adaptation stage 1

  • Train with 1% annotation budget using multiple GPUs

    sh scripts/ADA/dist_train_active.sh ${NUM_GPUs} \ 
 --cfg_file ./cfgs/ADA/waymo-kitti/pvrcnn/active_dual_target_01.yaml \ 
 --pretrained_model ${PRETRAINED_MODEL}

  • Train with 1% annotation budget using multiple machines

    sh scripts/ADA/slurm_train_active.sh ${PARTITION} ${JOB_NAME} ${NUM_NODES} \ 
 --cfg_file ./cfgs/ADA/waymo-kitti/pvrcnn/active_dual_target_01.yaml \  
 --pretrained_model ${PRETRAINED_MODEL}

  • Train with 5% annotation budget using multiple GPUs

    sh scripts/ADA/dist_train_active.sh ${NUM_GPUs} \ 
 --cfg_file ./cfgs/ADA/waymo-kitti/pvrcnn/active_dual_target_05.yaml \ 
 --pretrained_model ${PRETRAINED_MODEL}

  • Train with 5% annotation budget using multiple machines

    sh scripts/ADA/slurm_train_active.sh ${PARTITION} ${JOB_NAME} ${NUM_NODES} \ 
 --cfg_file ./cfgs/ADA/waymo-kitti/pvrcnn/active_dual_target_05.yaml \ 
 --pretrained_model ${PRETRAINED_MODEL}

Evaluating the model on the labeled target domain

  • Test with a ckpt file:

    python test.py --cfg_file ${CONFIG_FILE} \ 
 --batch_size ${BATCH_SIZE} \ 
 --ckpt ${CKPT}

  • To test all the saved checkpoints of a specific training setting and draw the performance curve on the Tensorboard, add the --eval_all argument:

    python test.py \ 
 --cfg_file ${CONFIG_FILE} \ 
 --batch_size ${BATCH_SIZE} \ 
 --eval_all

  • Notice that if you want to test on the setting with KITTI as target domain, please add --set DATA_CONFIG_TAR.FOV_POINTS_ONLY True to enable front view point cloud only:

    python test.py \ 
 --cfg_file ${CONFIG_FILE} \ 
 --batch_size ${BATCH_SIZE} \ 
 --eval_all \ 
 --set DATA_CONFIG_TAR.FOV_POINTS_ONLY True

  • To test with multiple machines for S-Proj:

    sh scripts/slurm_test_mgpu.sh ${PARTITION} ${NUM_NODES} \ 
  --cfg_file ${CONFIG_FILE} \ 
  --batch_size ${BATCH_SIZE}

Train with other active domain adaptation / active learning methods

  • Train with TQS

    sh scripts/ADA/dist_train_active_TQS.sh ${NUM_GPUs} \ 
 --cfg_file ./cfgs/ADA/waymo-kitti/pvrcnn/active_TQS.yaml \ 
 --pretrained_model ${PRETRAINED_MODEL}

  • Train with CLUE

    sh scripts/ADA/dist_train_active_CLUE.sh ${NUM_GPUs} \ 
 --cfg_file ./cfgs/ADA/waymo-kitti/pvrcnn/active_CLUE.yaml \ 
 --pretrained_model ${PRETRAINED_MODEL}

Combine Bi3D and UDA

  • Train with multiple GPUs

    sh scripts/ADA/dist_train_active_st3d.sh ${NUM_GPUs} \ 
 --cfg_file ./cfgs/ADA/waymo-kitti/pvrcnn/active_st3d.yaml \ 
 --pretrained_model ${PRETRAINED_MODEL}

  • Train with multiple machines

    sh scripts/ADA/slurm_train_active_st3d.sh ${PARTITION} ${JOB_NAME} ${NUM_NODES} \ 
 --cfg_file ./cfgs/ADA/waymo-kitti/pvrcnn/active_st3d.yaml \ 
 --pretrained_model ${PRETRAINED_MODEL}

   

All ADA Results:

We report the cross-dataset adaptation results including Waymo-to-KITTI, nuScenes-to-KITTI, Waymo-to-nuScenes, and Waymo-to-Lyft.

  • All LiDAR-based models are trained with 2 NVIDIA A100 GPUs and are available for download.

ADA Results for Waymo-to-KITTI:

training timeAdaptationCar@R40download
PV-RCNN~23h@4 A100Source Only67.95 / 27.65-
PV-RCNN~1.5h@2 A100Bi3D (1% annotation budget)87.12 / 78.03Model-58M
PV-RCNN~10h@2 A100Bi3D (5% annotation budget)89.53 / 81.32Model-58M
PV-RCNN~1.5h@2 A100TQS82.00 / 72.04Model-58M
PV-RCNN~1.5h@2 A100CLUE82.13 / 73.14Model-50M
PV-RCNN~10h@2 A100Bi3D+ST3D87.83 / 81.23Model-58M
Voxel R-CNN~16h@4 A100Source Only64.87 / 19.90-
Voxel R-CNN~1.5h@2 A100Bi3D (1% annotation budget)88.09 / 79.14Model-72M
Voxel R-CNN~6h@2 A100Bi3D (5% annotation budget)90.18 / 81.34Model-72M
Voxel R-CNN~1.5h@2 A100TQS78.26 / 67.11Model-72M
Voxel R-CNN~1.5h@2 A100CLUE81.93 / 70.89Model-72M

ADA Results for nuScenes-to-KITTI:

training timeAdaptationCar@R40download
PV-RCNN~23h@4 A100Source Only68.15 / 37.17Model-150M
PV-RCNN~1.5h@2 A100Bi3D (1% annotation budget)87.00 / 77.55Model-58M
PV-RCNN~9h@2 A100Bi3D (5% annotation budget)89.63 / 81.02Model-58M
PV-RCNN~1.5h@2 A100TQS84.66 / 75.40Model-58M
PV-RCNN~1.5h@2 A100CLUE74.77 / 64.43Model-50M
PV-RCNN~7h@ 2 A100Bi3D+ST3D89.28 / 79.69Model-58M
Voxel R-CNN~16h@4 A100Source Only68.45 / 33.00Model-191M
Voxel R-CNN~1.5h@2 A100Bi3D (1% annotation budget)87.33 / 77.24Model-72M
Voxel R-CNN~5.5h@2 A100Bi3D (5% annotation budget)87.66 / 80.22Model-72M
Voxel R-CNN~1.5h@2 A100TQS79.12 / 68.02Model-73M
Voxel R-CNN~1.5h@2 A100CLUE77.98 / 66.02Model-65M

ADA Results for Waymo-to-nuScenes:

training timeAdaptationCar@R40download
PV-RCNN~23h@4 A100Source Only31.02 / 21.21-
PV-RCNN~4h@2 A100Bi3D (1% annotation budget)45.00 / 30.81Model-58M
PV-RCNN~12h@4 A100Bi3D (5% annotation budget)48.03 / 32.02Model-58M
PV-RCNN~4h@2 A100TQS35.47 / 25.00Model-58M
PV-RCNN~3h@2 A100CLUE38.18 / 26.96Model-50M
Voxel R-CNN~16h@4 A100Source Only29.08 / 19.42-
Voxel R-CNN~2.5h@2 A100Bi3D (1% annotation budget)45.47 / 30.49Model-72M
Voxel R-CNN~4h@4 A100Bi3D (5% annotation budget)46.78 / 32.14Model-72M
Voxel R-CNN~4h@2 A100TQS36.38 / 24.18Model-72M
Voxel R-CNN~3h@2 A100CLUE37.27 / 25.12Model-65M
SECOND~3h@2 A100Bi3D(1%)46.15 / 26.24Model-54M

ADA Results for Waymo-to-Lyft:

training timeAdaptationCar@R40download
PV-RCNN~23h@4 A100Source Only70.10 / 53.11-
PV-RCNN~7h@2 A100Bi3D (1% annotation budget)79.07 / 63.74Model-58M
PV-RCNN~22h@2 A100Bi3D (5% annotation budget)80.19 / 66.09Model-58M
PV-RCNN~7h@2 A100TQS70.87 / 55.25Model-58M
PV-RCNN~5h@2 A100CLUE75.23 / 62.17Model-50M
Voxel R-CNN~16h@4 A100Source Only70.52 / 53.48-
Voxel R-CNN~7h@2 A100Bi3D (1% annotation budget)77.00 / 61.23Model-72M
Voxel R-CNN~19h@2 A100Bi3D (5% annotation budget)79.15 / 65.26Model-72M
Voxel R-CNN~8h@2 A100TQS71.11 / 56.28Model-73M
Voxel R-CNN~5h@2 A100CLUE75.61 / 59.34Model-65M