LIDAR-EDIT: Synthetic Generation of Faithful and Controllable Lidar Data for Autonomous Driving via Editing Real-World Scenes

May 4, 2025 · View on GitHub

Authors: Shing-Hei Ho, Bao Thach, Minghan Zhu

Accepted at ICRA 2025

Project Website

https://sites.google.com/view/lidar-edit/home

Interactive demo

https://huggingface.co/spaces/Shing-Hei/LiDAR-EDIT_DEMO_ICRA2025
You can also choose to run the interactive demo locally by running demo.sh or app.py (faster)

Our data and model checkpoints

https://drive.google.com/drive/folders/1QbJ2AKVYXiE5T0XK2vlV5iR_aS3sxnAH?usp=sharing

Arxiv link

https://arxiv.org/abs/2412.00592

File structure

Lidar_generation/
├── configs/
│   ├── nuscenes_config.py (configuration containing parameters for voxelization and transformer models' hyperparameters)
├── models/
│   ├── quantizer.py (for VQ-VAE)
│   ├── transformers.py (encoder, decoder, bidirectional transformer for MaskGIT)
│   ├── vqvae_transformers.py (VQ-VAE and MaskGIT)
├── train_transformer_models/
│   ├── train_vqvae_transformer.py
│   ├── test_vqvae_transformer.py
│   ├── train_maskgit_transformer.py
│   ├── test_maskgit_transformer.py
├── datasets/
│   ├── data_utils.py (general utilities for data processing and visualization)
│   ├── data_utils_nuscenes.py (utilities for NuScenes dataset)
│   ├── dataset_nuscenes.py (contains datasets for training, inpainting evaluation, foreground object extraction for point cloud completion and foreground object insertion)
│   ├── dataset.py (voxelizer and a general dataset class shared by all types of datasets in dataset_nuscenes.py)
├── evaluation/
│   ├── histogram_evaluate_allocentric_compare.py (evaluate in-painting performance via statistical metrics)
│   └── save_pc_for_feat_diff.py (save the in-painted backgrounds from different baselines)
├── actor_insertion/
│   ├── collect_foreground_objects.py (collect partial foreground vehicle (car,bus,truck) point clouds for point cloud completion)
│   ├── generate_insert_obj_dict_baselines.py (insert dense foreground vehicles for training (perturbed poses) and evaluation (inserted at original poses) of 3D detection models)
├── installation
│   ├── ultralidar.sh
│   ├── mmdet3d.sh
├── other_repo
│   ├── AnchorFormer
│   │      └── ...
│   └── OpenPCDet_minghan
│           └── ...
├── removal.sh
├── evaluation.sh
└── insertion.sh

other_repo
- AnchorFormer: Some scripts are customized to run point cloud completion on our extracted vehicles from NuScenes.
- OpenPCDet_minghan: 3D detection model's dataset creation, training and evaluation. We use VoxelNext trained on single-sweep lidar with intensity values (reflectance) set to zero.

Installation

For background in-painting training and evaluation and inserting dense object point cloud, run the ultralidar.sh script in the installation folder to install dependencies for our repo. You should install the conda environments ultralidar and pointTr separately by commenting and uncommenting the corresponding lines in the .sh file.
For 3D detection experiments, run mmdet3d.sh to install OpenPCDet's dependencies and go to OpenPCDet's repo to install their dependencies. Original OpenPCDet repo: https://github.com/open-mmlab/OpenPCDet/tree/master
Follow the instruction of AnchorFormer for point cloud completion. Original AnchorFormer repo: https://github.com/chenzhik/AnchorFormer. You may also refer to the PoinTr repo when setting up AnchorFormer https://github.com/yuxumin/PoinTr/blob/master/DATASET.md.
We had implemented additional functionalities in the AnchorFormer repo and OpenPCDet repo to customize them to the synthetic dataset generated by our method, and you would need to use the modified repos that are stored in ./other_repo

Setting up NuScenes dataset file structure

Download the dataset from https://www.nuscenes.org/nuscenes#download
Inside that dataset_path, follow the intsruction here to set up nuscenes trainval dataset enabled with lidarseg: https://www.nuscenes.org/tutorials/nuscenes_lidarseg_panoptic_tutorial.html

└── nuscenes  
    ├── Usual nuscenes folders (i.e. samples, sweep)
    │
    ├── lidarseg
    │   └── v1.0-{mini, test, trainval} <- Contains the .bin files; a .bin file 
    │                                      contains the labels of the points in a 
    │                                      point cloud (note that v1.0-test does not 
    │                                      have any .bin files associated with it)
    │
    ├── panoptic
    │   └── v1.0-{mini, test, trainval} <- Contains the *_panoptic.npz files; a .npz file 
    │                                      contains the panoptic labels of the points in a 
    │                                      point cloud (note that v1.0-test does not 
    │                                      have any .npz files associated with it) 
    └── v1.0-{mini, test, trainval}
        ├── Usual files (e.g. attribute.json, calibrated_sensor.json etc.)
        ├── lidarseg.json  <- contains the mapping of each .bin file to the token
        ├── panoptic.json  <- contains the mapping of each .npz file to the token       
        └── category.json  <- contains the categories of the labels (note that the 
                              category.json from nuScenes v1.0 is overwritten)

It is recommended to put NuScenes data at /mnt and create a symbolic link to it at arbitrary location (e.g. /home/shinghei/OpenPCDet/data) to use it.
The bash scripts (e.g. removal.sh) in this repo have a variable dataset_path, which is where I put the dataset. "version" specifies whether I am using v1.0-trainval or v1.0-mini. v1.0-mini is for quick testing and debugging. Use v1.0-trainval. You can customize dataset_path yourself.

OpenPCDet file structure

OpenPCDet assume the following file structure for NuScenes data.

OpenPCDet
├── data
│   ├── nuscenes
│   │   │── v1.0-trainval (or v1.0-mini if you use mini)
│   │   │   │── samples
│   │   │   │── sweeps
│   │   │   │── maps
│   │   │   │── v1.0-trainval

Assume that we are at the root directory of this repo.

1. Background in-painting training and evaluation

Train VQ-VAE and MaskGIT for background in-painting by running train_vqvae_transformer.py and train_maskgit_transformer.py
Visualize outputs from VQ-VAE and MaskGIT by running test_vqvae_transformer.py and test_maskgit_transformer.py

conda activate ultralidar
./removal.sh

evaluation

conda activate ultralidar
./evaluation.sh

2. Object Library Construction

Collect partial foreground objects' point clouds

You can use our pre-collected foreground object vehicles' partial point clouds and their dense point clouds here (TODO). If you do so, skip this section.
Collect foreground vehicles' point clouds from NuScenes lidar scans

conda activate ultralidar

# run this line that is in insertion.sh:
python actor_insertion/collect_foreground_objects.py --trainval_data_path=$dataset_path --data_version=$version --pc_path="./foreground_object_pointclouds" --visualize_dense=0 --save_as_pcd=1

# REMEMBER to remove all previously collected foreground objects first. otherwise, there would be 
# inconsistencies between the dictionaries and the point clouds we save.

After running the code above, ./foreground_object_pointclouds should have the following structure:

foreground_object_pointclouds
├── car
│    └── point clouds of cars ... each pointcloud has a name of the form "sample_<i>.pcd"
├── bus
│    └── poitn clouds of buses ... each pointcloud has a name of the form "sample_<i>.pcd"
├── truck
│    └── point clouds of trucks ... each pointcloud has a name of the form "sample_<i>.pcd"
├── allocentric_dict.pickle
├── sample_dict.pickle

# - let's say we have the following class_names for vehicles: {"car", "truck", "bus"}
# - each point cloud we collected is uniquely identified by (class_name, pc_name) e.g. ("car", "sample_14.pcd")
# - allocentric_dict.pickle: a dictionary mapping from class_name to 2D list. Each row of the 2D list has the same length, each entry i of each row corresponds to the same object point cloud and the entry is a property of that object (e.g. pc_name, bounding box, allocentric angle, etc.)
# - sample_dict.pickle: a dictionary mapping from (class_name, pc_name) to the 2D list same as above.

Point cloud completion

Run AnchorFormer: run point cloud completion and save the resultant dense point clouds of collected foreground vehicles
- Before running the following commands, make sure to rename some variables in:
  - AnchorFormer/datasets/customDataset.py: rename the variable pc_root in the _get_file_list method as the path to foreground_object_pointclouds
  - AnchorFormer/tools/runner.py: rename the variable save_root in the test method as the path to foreground_object_pointclouds

cd AnchorFormer # go to where the AnchorFormer directory is
conda activate pointTr
CUDA_VISIBLE_DEVICES=0 python main.py --ckpts anchorformer_dict.pth --config ./cfgs/custom_models/AnchorFormer.yaml --exp_name test_ckpt --test

The directory dense_nusc storing the corresponding dense point clouds should appear in the directory foreground_object_pointclouds

3. Generate synthetic LiDAR scenes via LiDAR Editing

Preparing synthetic dataset

You can uncomment certain lines in actor_insertion/nuscenes_utils.py and actor_insertion/generate_insert_obj_dict_baselines.py to visualize the original, inpainted and edited point clouds.
Run the following code in ./insertion.sh to collect synthetic LiDAR scenes:
- Modify the following variables:
  - root in insertion.sh: where you save the data
  - For generating synthetic LiDAR scenes based on the NuScenes validation set:
    - set split="val" in insertion.sh and use the method insertion_vehicles_driver in generate_insert_obj_dict_baselines
  - For generating synthetic LiDAR scenes based on the NuScenes training set:
    - set split="train" and use the method insertion_vehicles_driver_perturbed in generate_insert_obj_dict_baselines

python actor_insertion/generate_insert_obj_dict_baselines.py --trainval_data_path=$dataset_path --data_version=$version --pc_path="./foreground_object_pointclouds" --dense=$dense --vqvae_path="$vqvae_path/epoch_$vqvae_epoch" --maskgit_path="$maskgit_path/epoch_$maskgit_epoch" --save_lidar_path=$root --split=$split --intensity_model_path="$intensity_model_path/epoch_$intensity_vqvae_epoch"

Data format

The generated data should be saved inside <path to OpenPCDet repo>/data/nuscenes/v1.0-trainval/ours. In my code, the path is /home/shinghei/lidar_generation/OpenPCDet_minghan/data/nuscenes/v1.0-trainval/ours.
The file structure of ours is

ours
├── v1.0-trainval
│    ├── lidar_point_clouds
│           ├── a bunch of .pcd files which are synthetic LiDAR point clouds .......
│    ├── token2sample.pickle

token2sample.pickle is a dictionary mapping from the lidar sample token to

(full_path_to_synthetic_lidar, bounding_boxes_in_synthetic_lidar, list_of_annotation_tokens_in_synthetic_lidar, sample_records, list_of_annotation_info_in_synthetic_lidar)

It is generated according to the following procedure in actor_insertion/insertion_utils.py:

points_xyz = new_scene_points_xyz
bounding_boxes = new_bboxes
lidar_sample_token = dataset.lidar_sample_token

sample_records = dataset.obj_properties[12]
cs_record, sensor_record, pose_record = sample_records["cs_record"], sample_records["sensor_record"], sample_records["pose_record"]

############# save point cloud 
pc_name = f'{args.split}_{lidar_sample_token}.bin'
os.makedirs(os.path.join(save_lidar_path, "lidar_point_clouds"), exist_ok=True)
lidar_full_path = os.path.join(save_lidar_path, "lidar_point_clouds", pc_name)
#assert(not os.path.exists(lidar_full_path))
new_points_xyz.astype(np.float32).tofile(lidar_full_path)

############## Save the data needed to build the new database
token2sample_dict[lidar_sample_token] = (lidar_full_path, bounding_boxes+other_foreground_boxes, new_obj_ann_token_list+other_foreground_obj_ann_token_list, sample_records, new_ann_info_list+other_foreground_obj_ann_info_list)

each lidar_sample_token is data_token in the get_path_infos method of datasets/dataset_nuscenes.py:

for sample in nusc.sample:
        #### This is a sample token from visualization of eval of voxel nexr in OpenPCNet
        # if sample['token']!='5d773ca713f54023b34cd4718a5ee293':
        #     continue
        scene_token = sample['scene_token']
        data_token = sample['data']['LIDAR_TOP']

4. 3D object detection training and evaluation

run_openpcdet.sh contains example commands I ran

Synthetic and NuScenes Datasets

Files customized for synthetic data

dataset class: <path to OpenPcDet>/pcdet/datasets/nuscenes/custom_nuscenes_dataset.py
dataset configuration: <path to OpenPcDet>/tools/cfgs/dataset_configs/custom_nuscenes_dataset.yaml
voxelnext configuration: <path to OpenPcDet>/tools/cfgs/nuscenes_models/custom_cbgs_voxel0075_voxelnext.yaml

Files customized for NuScenes data

dataset class: <path to OpenPcDet>/pcdet/datasets/nuscenes/nuscenes_dataset.py
dataset configuration: <path to OpenPcDet>/tools/cfgs/dataset_configs/nuscenes_dataset.yaml
voxelnext configuration: <path to OpenPcDet>/tools/cfgs/nuscenes_models/cbgs_voxel0075_voxelnext.yaml

Data processing for OpenPCDet framework

Before training or evaluation, you need to generate processed data for the OpenPCDet framework and model to consume. For example, you may want to obtain:
- a concatenation of processed sythetic and NuScenes data
- processed synthetic data
- processed NuScenes data
We assume that the synthetic data is in <path to OpenPCDet repo>/data/nuscenes/v1.0-trainval/ours. Rename the directory as ours if necessary.
You may need to modify the following variables before you process training and evaluation data for each experiment:
- In <path to OpenPCDet>/pcdet/datasets/nuscenes/nuscenes_utils.py:
  - root: the path to the directory where our synthetic dataset ours is located
- In <path to OpenPCDet>/pcdet/datasets/nuscenes/custom_nuscenes_dataset.py:
  - my_root: same as root
  - TRAIN_WITH_MY_AUGMENT: True for concatenating synethtic dataset with the original NuScenes dataset. Otherwise, only get the synthetic data.
- For example, root = "/home/shinghei/lidar_generation/OpenPCDet_minghan/data/nuscenes/v1.0-trainval"
To generate processed synthetic data or concatneation of synthetic and NuScenes data, run these commands at <path to OpenPCDet>

VERSION=v1.0-trainval

############ export the cloned nuscenes-devkit repo path
export PYTHONPATH=/home/shinghei/lidar_generation/nuscenes-devkit/python-sdk:$PYTHONPATH

python -m pcdet.datasets.nuscenes.custom_nuscenes_dataset --func create_nuscenes_infos --cfg_file tools/cfgs/dataset_configs/custom_nuscenes_dataset.yaml --version $VERSION

Some files and directories should be generated at <path to OpenPCDet>/data/nuscenes/v1.0-trainval containing the processed data. When generating another processed data, you should move the existing processed data to another directory to avoid overwritting them.
To generate processed NuScenes data, run these commands at the <path to OpenPCDet>

VERSION=v1.0-trainval

############ export the cloned nuscenes-devkit repo path
export PYTHONPATH=/home/shinghei/lidar_generation/nuscenes-devkit/python-sdk:$PYTHONPATH

python -m pcdet.datasets.nuscenes.nuscenes_dataset --func create_nuscenes_infos --cfg_file tools/cfgs/dataset_configs/nuscenes_dataset.yaml --version $VERSION

The code above use token2sample_dict; for each lidar scan (1-1 correspondence to lidar sample token), the full path to its corresponding synthetic lidar scan is not a relative path. If you are using some pre-generated synthetic data not generated by yourself, make sure each full path is correct. If necessary, please modify the full path when processing data in fill_trainval_infos in nuscenes_utils.py. Use ctrl-F to search for "#### TODO: adjust the lidar path if necessary:" in that file, and uncomment and modify that part accordingly

Training

CONFIG_FILE=<voxel next configuration file>
CHECKPOINT_FILE=<the path to a model checkpoint>

##### if you want to initialize the model as the checkpoint model, reset the optimizer state and learning rate scheduler state and then train
python tools/train.py --pretrained_model ${CHECKPOINT_FILE} --cfg_file ${CONFIG_FILE}

##### if you want to train the model from scratch
python tools/train.py --cfg_file ${CONFIG_FILE}

##### if you want to resume model training from the checkpoint model (load model weight, resume optimizer state, learning rate scheduler state and current epoch)
python tools/train.py --ckpt ${CHECKPOINT_FILE} --cfg_file ${CONFIG_FILE}

You should see training log, tensorboard logging and model weights saved inside directories in <path to OpenPCDet>/output.

Evaluation

CONFIG_FILE=<voxel next configuration file>
CHECKPOINT_FILE=<the path to a model checkpoint>

python tools/test.py --cfg_file ${CONFIG_FILE} --batch_size 1 --ckpt ${CHECKPOINT_FILE}

You should see the evaluated performance printed on the terminal and possibly some visualization saved inside directories in <path to OpenPCDet>/output.

License

MIT License