TrajectoryFormer: 3D Object Tracking Transformer with Predictive Trajectory Hypotheses (ICCV 2023)

Preprocessing

Please follow the command at Data Preprocessing Documentations to process Waymo Open Dataset and install EFG.

The preprocessed detection boxes of CenterPoint and MPPNet can be download form Google Drive and put the download files in the EFG/datasets/waymo/ folder.

Finally, compile the evaluation metrics tool provided by Waymo officials by following Quick Guide to Waymo Open Dataset.

Training Motionpredictor

cd playground/tracking.3d/waymo/trajectoryformer.motionpred;
efg_run --nug-gpus 8 task train

The trained model will be saved at ./log/model_final.pth.

Training TrajectoryFormer

# CenterPoint
cd playground/tracking.3d/waymo/trajectoryformer.centerpoint;
efg_run --num-gpus 8 task train 

# MPPNet 
cd playground/tracking.3d/waymo/trajectoryformer.mppnet;
efg_run --num-gpus 8 task train 

Evaluation

Set the metrics tool path, such as /home/user/bazel_bin/waymo_open_dataset/metrics/tools/compute_tracking_main to the yaml.
For CenterPoint,

cd playground/tracking.3d/waymo/trajectoryformer.centerpoint;
# for vehicle
efg_run --num-gpus 8 task val \
trainer.eval_metrics_path /path/to/your/tools 
model.nms_thresh 0.1
model.eval_class VEHICLE

# for pedestrian
efg_run --num-gpus 8 task val \
trainer.eval_metrics_path /path/to/your/tools 
model.nms_thresh 0.7 \
model.eval_class PEDESTRIAN

# for cyclist
efg_run --num-gpus 8 task val \
trainer.eval_metrics_path /path/to/your/tools 
model.nms_thresh 0.7 \
model.eval_class CYCLIST

For MPPNet,

# eval vehicle, pedestrian, cyclist
cd playground/tracking.3d/waymo/trajectoryformer.mppnet;
efg_run --num-gpus 8 task val \
trainer.eval_metrics_path /path/to/your/tools
model.eval_class VEHICLE or PEDESTRIAN or CYCLIST

If you want the pretrained model, please contact chenxuesong@link.cuhk.edu.hk.

Citation

@article{chen2023trajectoryformer,
  title={TrajectoryFormer: 3D Object Tracking Transformer with Predictive Trajectory Hypotheses},
  author={Chen, Xuesong and Shi, Shaoshuai and Zhang, Chao and Zhu, Benjin and Wang, Qiang and Cheung, Ka Chun and See, Simon and Li, Hongsheng},
  journal={arXiv preprint arXiv:2306.05888},
  year={2023}
}


@inproceedings{chen2022mppnet,
  title={Mppnet: Multi-frame feature intertwining with proxy points for 3d temporal object detection},
  author={Chen, Xuesong and Shi, Shaoshuai and Zhu, Benjin and Cheung, Ka Chun and Xu, Hang and Li, Hongsheng},
  booktitle={European Conference on Computer Vision},
  pages={680--697},
  year={2022},
  organization={Springer}
}