WeakMotionPP

May 28, 2026 · View on GitHub

This is the PyTorch code for Weakly and Self-Supervised Class-Agnostic Motion Prediction for Autonomous Driving (T-PAMI 2026). You can also check out the arXiv version at WeakMotionPP-arXiv. The code is created by Ruibo Li (ruibo001@e.ntu.edu.sg).

Prerequisites
Data preprocess
Evaluation
- Trained models
- Testing
Training
Citation
Acknowledgement

Prerequisites

Create a conda environment for WeakMotionPP:

conda create -n WeakMotionPP python=3.7
conda activate WeakMotionPP
python -m pip install torch==1.9.0+cu102  -f https://download.pytorch.org/whl/torch_stable.html
python -m pip install pyquaternion==0.9.9 opencv-python==4.8.1.78 matplotlib==3.5.3 scikit-learn==1.0.2 tqdm==4.66.1
pip install waymo-open-dataset-tf-2-6-0

Data preprocess

We provide preprocessing scripts for both nuScenes and Waymo to prepare the input data, motion ground truth, Foreground/Background labels, and ground plane models for weak and self-supervision.

Please refer to gen_data_pp/README_data.md for the detailed instructions.

Evaluation

Trained models

The trained models can be downloaded from the following links.

nuScenes

Weakly Supervised

Annotation ratio	Trained PreSegNet (Stage1)	Trained WeakMotionNet (Stage2)
0.01% FG/BG masks	Not required	nuscenes_motion_0-0001
0.1% FG/BG masks	nuscenes_seg_0-001	nuscenes_motion_0-001
1.0% FG/BG masks	nuscenes_seg_0-01	nuscenes_motion_0-01
100% FG/BG masks	Not required	nuscenes_motion_1-0

Self-Supervised

Supervision	Trained SelfMotionNet
Self-supervised	nuscenes_selfmotion

Waymo

Weakly Supervised

Annotation ratio	Trained PreSegNet (Stage1)	Trained WeakMotionNet (Stage2)
0.001% FG/BG masks	Not required	waymo_motion_0-00001
0.1% FG/BG masks	waymo_seg_0-001	waymo_motion_0-001
1.0% FG/BG masks	waymo_seg_0-01	waymo_motion_0-01
100% FG/BG masks	Not required	waymo_motion_1-0

Self-Supervised

Supervision	Trained SelfMotionNet
Self-supervised	waymo_selfmotion

Testing

We provide two evaluation scripts: test_WeakMotionNet.py for the weakly supervised model, and test_SelfMotionNet.py for the self-supervised one. Both share the same arguments:

Argument	Description
`--evaldata`	Path to the evaluation BEV data
`--pretrained`	Path to the trained model checkpoint under `pretrained/`
`--datatype`	Dataset type; one of `nuScenes` or `Waymo`

Download the pretrained model from the Trained models tables and place it under pretrained/ before running the commands below.

Weakly Supervised

python test_WeakMotionNet.py --evaldata /path_to/nuScenes/motionnet-data/test/ --pretrained pretrained/nuscenes_motion_1-0.pth --datatype nuScenes

python test_WeakMotionNet.py --evaldata /path_to/Waymo/motionnet-data/val/ --pretrained pretrained/waymo_motion_1-0.pth --datatype Waymo

Self-Supervised

python test_SelfMotionNet.py --evaldata /path_to/nuScenes/motionnet-data/test/ --pretrained pretrained/nuscenes_selfmotion.pth --datatype nuScenes

python test_SelfMotionNet.py --evaldata /path_to/Waymo/motionnet-data/val/ --pretrained pretrained/waymo_selfmotion.pth --datatype Waymo

Training

In the following, we take the training on nuScenes data as an example.

Training with partially annotated Foreground/Background masks (per-frame partial annotation)

Training of PreSegNet (Stage1) Arguments:

Argument	Description
`--motiondata`	Path to the preprocessed BEV training data
`--evaldata`	Path to the preprocessed BEV validation data
`--raw_pc`	Path to the raw point clouds
`--weak_info`	Path to the weak-supervision meta info
`--raw_pc_label_root`	Path to the per-point FG/BG labels for the raw point clouds
`--datatype`	Dataset type; one of `nuScenes` or `Waymo`
`--annotation_ratio`	FG/BG annotation ratio; one of `0.001` (0.1%) or `0.01` (1.0%)
`--bg_weight`	Background loss weight: use `0.005` for nuScenes and `0.02` for Waymo
`--name`	Experiment name; used as the sub-folder under `logs/`

Examples:

nuScenes (1.0% FG/BG annotation):

python train_PreSegNet.py \
    --motiondata /path_to/nuScenes/motionnet-data/train/ \
    --evaldata   /path_to/nuScenes/motionnet-data/val/ \
    --raw_pc     /path_to/nuScenes/raw-pc/ \
    --weak_info  /path_to/nuScenes/weak-data-info/ \
    --raw_pc_label_root /path_to/nuScenes/raw-pc-label/ \
    --datatype   nuScenes \
    --annotation_ratio 0.01 \
    --bg_weight  0.005 \
    --batch      32 \
    --name       PreSegNet_nuscenes_1e-2

Waymo (1.0% FG/BG annotation):

python train_PreSegNet.py \
    --motiondata /path_to/Waymo/motionnet-data/train/ \
    --evaldata   /path_to/Waymo/motionnet-data/val/ \
    --raw_pc     /path_to/Waymo/raw-pc/ \
    --weak_info  /path_to/Waymo/weak-data-info/ \
    --raw_pc_label_root /path_to/Waymo/raw-pc-label/ \
    --datatype   Waymo \
    --annotation_ratio 0.01 \
    --bg_weight  0.02 \
    --batch      32 \
    --name       PreSegNet_waymo_1e-2

To train with the lower annotation ratio (0.1%), set --annotation_ratio 0.001 and rename --name accordingly (e.g., PreSegNet_nuscenes_1e-3).

Generate FG/BG masks for the training data using the trained PreSegNet Choose a PreSegNet model with higher Foreground Accuracy and Overall Accuracy to generate FG/BG masks for the training data. Arguments:

Argument	Description
`--save_FGBG_BEV`	Path to save the predicted FG/BG masks (BEV `.npy` files)
`--pretrained`	Path to the trained PreSegNet model from Stage1

Examples:

nuScenes (using the PreSegNet trained with 1.0% FG/BG annotation):

python FGBG_prediction_by_PreSegNet.py \
    --motiondata    /path_to/nuScenes/motionnet-data/train/ \
    --raw_pc        /path_to/nuScenes/raw-pc/ \
    --weak_info     /path_to/nuScenes/weak-data-info/ \
    --save_FGBG_BEV /path_to/nuScenes/FGBG-train-prediction/nuscenes_1e-2/ \
    --datatype      nuScenes \
    --pretrained    pretrained/PreSegNet/nuscenes_seg_0-01.pth

Waymo (using the PreSegNet trained with 1.0% FG/BG annotation):
```
python FGBG_prediction_by_PreSegNet.py \
    --motiondata    /path_to/Waymo/motionnet-data/train/ \
    --raw_pc        /path_to/Waymo/raw-pc/ \
    --weak_info     /path_to/Waymo/weak-data-info/ \
    --save_FGBG_BEV /path_to/Waymo/FGBG-train-prediction/waymo_1e-2/ \
    --datatype      Waymo \
    --pretrained    pretrained/PreSegNet/waymo_seg_0-01.pth
```
To use the PreSegNet trained with the lower annotation ratio (0.1%), set --pretrained to the corresponding *_seg_0-001.pth checkpoint and rename --save_FGBG_BEV accordingly (e.g., /path_to/nuScenes/FGBG-train-prediction/nuscenes_1e-3/).
In addition to training PreSegNet yourself, you can also use the pretrained PreSegNet models listed in the Trained models tables above (under the Trained PreSegNet (Stage1) column).

Training of WeakMotionNet (Stage2) Arguments:

Argument	Description
`--FGBGdata`	Path to the predicted FG/BG masks from Stage1 (the `--save_FGBG_BEV` directory from the previous step)
`--motion_data_back`	Path to the preprocessed backward BEV data
`--bg_weight`	Background loss weight: use `0.005` for nuScenes and `0.02` for Waymo
`--ratio`	FG/BG annotation ratio; must match the ratio used in Stage1 (e.g., `0.001` or `0.01`)

Examples:

nuScenes (1.0% FG/BG annotation):

python train_WeakMotionNet.py \
    --motiondata /path_to/nuScenes/motionnet-data/train/ \
    --evaldata   /path_to/nuScenes/motionnet-data/test/ \
    --FGBGdata   /path_to/nuScenes/FGBG-train-prediction/nuscenes_1e-2/ \
    --raw_pc     /path_to/nuScenes/raw-pc/ \
    --raw_pc_label_root /path_to/nuScenes/raw-pc-label/ \
    --weak_info  /path_to/nuScenes/weak-data-info/ \
    --motion_data_back /path_to/nuScenes/motionnet-data-back/ \
    --datatype   nuScenes \
    --bg_weight  0.005 \
    --ratio      0.01 \
    --batch      16 \
    --name       WeakMotion_nuscenes_1e-2

Waymo (1.0% FG/BG annotation):

python train_WeakMotionNet.py \
    --motiondata /path_to/Waymo/motionnet-data/train/ \
    --evaldata   /path_to/Waymo/motionnet-data/val/ \
    --FGBGdata   /path_to/Waymo/FGBG-train-prediction/waymo_1e-2/ \
    --raw_pc     /path_to/Waymo/raw-pc/ \
    --raw_pc_label_root /path_to/Waymo/raw-pc-label/ \
    --weak_info  /path_to/Waymo/weak-data-info/ \
    --motion_data_back /path_to/Waymo/motionnet-data-back/ \
    --datatype   Waymo \
    --bg_weight  0.02 \
    --ratio      0.01 \
    --batch      12 \
    --name       WeakMotion_waymo_1e-2

To train with the lower annotation ratio (0.1%), set --ratio 0.001, point --FGBGdata to the corresponding Stage1 output (e.g., /path_to/nuScenes/FGBG-train-prediction/nuscenes_1e-3/), and rename --name accordingly (e.g., WeakMotion_nuscenes_1e-3). Note that --ratio must be identical to the --annotation_ratio used in Stage1.

Training with fully annotated Foreground/Background masks

When fully annotated FG/BG masks are available, WeakMotionNet can be trained in a single stage. Stage1 (PreSegNet training) and the subsequent FG/BG mask prediction are no longer required. In this setting, the ground-truth FG/BG masks are loaded directly from --raw_pc_label_root, so --FGBGdata does not need to be specified, and --ratio should be set to 1.0.

Examples:

nuScenes (100% FG/BG annotation):

python train_WeakMotionNet.py \
    --motiondata /path_to/nuScenes/motionnet-data/train/ \
    --evaldata   /path_to/nuScenes/motionnet-data/test/ \
    --raw_pc     /path_to/nuScenes/raw-pc/ \
    --raw_pc_label_root /path_to/nuScenes/raw-pc-label/ \
    --weak_info  /path_to/nuScenes/weak-data-info/ \
    --motion_data_back /path_to/nuScenes/motionnet-data-back/ \
    --datatype   nuScenes \
    --bg_weight  0.005 \
    --ratio      1.0 \
    --batch      16 \
    --name       WeakMotion_nuscenes_1-0

Waymo (100% FG/BG annotation):

python train_WeakMotionNet.py \
    --motiondata /path_to/Waymo/motionnet-data/train/ \
    --evaldata   /path_to/Waymo/motionnet-data/val/ \
    --raw_pc     /path_to/Waymo/raw-pc/ \
    --raw_pc_label_root /path_to/Waymo/raw-pc-label/ \
    --weak_info  /path_to/Waymo/weak-data-info/ \
    --motion_data_back /path_to/Waymo/motionnet-data-back/ \
    --datatype   Waymo \
    --bg_weight  0.02 \
    --ratio      1.0 \
    --batch      12 \
    --name       WeakMotion_waymo_1-0

Training with partially annotated Foreground/Background masks (one annotated frame per sequence)

In this even weaker setting, only a single frame per sequence is annotated. The training is done in a single stage with train_WeakMotionNet_weaker.py, and Stage1 (PreSegNet) and the FG/BG mask prediction are not required.

Internally, the data loaders (data/weak_nuscenes_weaker.py and data/weak_waymo_weaker.py) first sub-sample one out of every 10 frames as the pool of candidate annotated frames, and then keep only --ratio of them. The effective annotation ratio is therefore --ratio × 1/10.

Additional arguments (on top of those used in the previous sections):

Argument	Description
`--raw_ground`	Path to the per-frame ground-plane models
`--scene_info`	Path to the scene-level meta info used to group frames into sequences

Examples:

nuScenes (one annotated frame per sequence, effective FG/BG ratio = 0.01%):

python train_WeakMotionNet_weaker.py \
    --motiondata /path_to/nuScenes/motionnet-data/train/ \
    --evaldata   /path_to/nuScenes/motionnet-data/test/ \
    --raw_pc     /path_to/nuScenes/raw-pc/ \
    --raw_pc_label_root /path_to/nuScenes/raw-pc-label/ \
    --weak_info  /path_to/nuScenes/weak-data-info/ \
    --motion_data_back /path_to/nuScenes/motionnet-data-back/ \
    --raw_ground /path_to/nuScenes/ground-data-info/ \
    --scene_info /path_to/nuScenes/scene-info/ \
    --datatype   nuScenes \
    --bg_weight  0.005 \
    --ratio      0.001 \
    --batch      16 \
    --name       WeakMotion_nuscenes_0-0001

Waymo (one annotated frame per sequence, effective FG/BG ratio = 0.001%):

python train_WeakMotionNet_weaker.py \
    --motiondata /path_to/Waymo/motionnet-data/train/ \
    --evaldata   /path_to/Waymo/motionnet-data/val/ \
    --raw_pc     /path_to/Waymo/raw-pc/ \
    --raw_pc_label_root /path_to/Waymo/raw-pc-label/ \
    --weak_info  /path_to/Waymo/weak-data-info/ \
    --motion_data_back /path_to/Waymo/motionnet-data-back/ \
    --raw_ground /path_to/Waymo/ground-data-info/ \
    --scene_info /path_to/Waymo/scene-info/ \
    --datatype   Waymo \
    --bg_weight  0.005 \
    --ratio      0.0001 \
    --batch      8 \
    --name       WeakMotion_waymo_0-00001

Self-Supervised Training with ground/non-ground masks

In the fully self-supervised setting, no human-annotated FG/BG masks are used. SelfMotionNet is instead supervised by automatically generated ground/non-ground masks (from --raw_ground), so the per-point FG/BG labels (--raw_pc_label_root) are no longer required. Training is done in a single stage with train_SelfMotionNet.py.

Examples:

nuScenes:

python train_SelfMotionNet.py \
    --motiondata /path_to/nuScenes/motionnet-data/train/ \
    --evaldata   /path_to/nuScenes/motionnet-data/test/ \
    --raw_pc     /path_to/nuScenes/raw-pc/ \
    --weak_info  /path_to/nuScenes/weak-data-info/ \
    --motion_data_back /path_to/nuScenes/motionnet-data-back/ \
    --raw_ground /path_to/nuScenes/ground-data-info/ \
    --datatype   nuScenes \
    --bg_weight  0.02 \
    --batch      16 \
    --name       SelfMotion_nuscenes

Waymo:

python train_SelfMotionNet.py \
    --motiondata /path_to/Waymo/motionnet-data/train/ \
    --evaldata   /path_to/Waymo/motionnet-data/val/ \
    --raw_pc     /path_to/Waymo/raw-pc/ \
    --weak_info  /path_to/Waymo/weak-data-info/ \
    --motion_data_back /path_to/Waymo/motionnet-data-back/ \
    --raw_ground /path_to/Waymo/ground-data-info/ \
    --datatype   Waymo \
    --bg_weight  0.005 \
    --batch      8 \
    --name       SelfMotion_waymo

Citation

If you find this code useful, please cite our paper:

@ARTICLE{11143881,
  author={Li, Ruibo and Shi, Hanyu and Wang, Zhe and Lin, Guosheng},
  journal={IEEE Transactions on Pattern Analysis and Machine Intelligence}, 
  title={Weakly and Self-Supervised Class-Agnostic Motion Prediction for Autonomous Driving}, 
  year={2026},
  volume={48},
  number={1},
  pages={1-16},
  doi={10.1109/TPAMI.2025.3604036}
}

@inproceedings{li2023weakly,
  title={Weakly Supervised Class-Agnostic Motion Prediction for Autonomous Driving},
  author={Li, Ruibo and Shi, Hanyu and Fu, Ziang and Wang, Zhe and Lin, Guosheng},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  pages={17599--17608},
  year={2023}
}

Acknowledgement

Our project references the codes in the following repos.