README_EN.md

Robust and Efficient 3D Semantic Occupancy Prediction with Calibration-free Spatial Transformation

[中文|EN]

Introduction

Existing methods mainly rely on spatial transformation with sensor calibration information. However, this operation also results in two significant issues: First, the model performance is degraded due to the calibration noise (e.g., changing of sensor position during driving ). Second, the spatial transformation is computationally expensive and limits the efficiency of existing methods. In this work, we exploit a calibration-free spatial transformation scheme based on the vanilla attention scheme, to project the 2D images and point clouds into a compact BEV plane. Besides, we also introduce extra auxiliary training tasks and an efficient decoder to improve the model performance and efficiency, respectively.

Main Results

• Efficiency analysis on OpenOccupancy nuScenes

  

• Results on OpenOccupancy nuScenes

• Results on OCC3D-nuScenes

  

Results on Test Sets

• SemanticKITTI Scene Completion

  

• OCC3D-nuScenes

How to use

This work is based on our prior work, i.e., PMF. If you are familiar with our PMF project, you can run this project easily.

• Prepare Environment

## env for pmf/epmf
conda create -n pmf python=3.8
conda activate pmf
pip install torch==1.13.1+cu116 torchvision==0.14.1+cu116 --extra-index-url https://download.pytorch.org/whl/cu116
pip install -r requirements.txt

## install pybind11
pip install pybind11

## compile .so for data preprocessing
# please check the Python3_ROOT_DIR in CMakeLists.txt. eg. set(Python3_ROOT_DIR "/usr/local/lib/python3.8/dist-packages")
cd pc_processor/dataset/cpp_api/
mkdir build
cd build
cmake .. && make -j

• training: please refer to pmf/epmf

• evaluation

python infer_smk.py

Pretrained models

• TBD

Citation

@article{zhuang2024robust3dsemanticoccupancy,
      title={Robust 3D Semantic Occupancy Prediction with Calibration-free Spatial Transformation}, 
      author={Zhuangwei Zhuang and Ziyin Wang and Sitao Chen and Lizhao Liu and Hui Luo and Mingkui Tan},
      journal={arXiv preprint arXiv:2411.12177},
      year={2024}
}