SPHERE: Semantic-PHysical Engaged REpresentation for 3D Semantic Scene Completion.

[Arxiv]

News

• [2025/09]: We release the evaluation and training code for SemanticKITTI and SSCBench-KITTI-360.
• [2025/09]: Our paper is on arxiv.
• [2025/07]: Accepted by ACM MM 2025!
  

Abstract

Camera-based 3D Semantic Scene Completion (SSC) is a critical task in autonomous driving systems, assessing voxel-level geometry and semantics for holistic scene perception. While existing voxel-based and plane-based SSC methods have achieved considerable progress, they struggle to capture physical regularities for realistic geometric details. On the other hand, neural reconstruction methods like NeRF and 3DGS demonstrate superior physical awareness, but suffer from high computational cost and slow convergence when handling large-scale, complex autonomous driving scenes, leading to inferior semantic accuracy. To address these issues, we propose the Semantic-PHysical Engaged REpresentation (SPHERE) for camera-based SSC, which integrates voxel and Gaussian representations for joint exploitation of semantic and physical information. First, the Semantic-guided Gaussian Initialization (SGI) module leverages dual-branch 3D scene representations to locate focal voxels as anchors to guide efficient Gaussian initialization. Then, the Physical-aware Harmonics Enhancement (PHE) module incorporates semantic spherical harmonics to model physical-aware contextual details and promote semantic-geometry consistency through focal distribution alignment, generating SSC results with realistic details. Extensive experiments and analyses on the popular SemanticKITTI and SSCBench-KITTI-360 benchmarks validate the effectiveness of SPHERE

Method

Getting Started

Installation

Please refer to Voxformer to create base environment. Some extra packages are needed to be installed:

• spconv-cu111==2.1.25
• torch-scatter==2.0.8
• tochmetrics>=0.9.0

Prepare Dataset

Please refer to the README in the preprocess folder for details.

Run and Eval

Train SPHERE with 4 GPUs

./tools/dist_train.sh ./projects/configs/sphere/sphere-T.py 4

Eval SPHERE with 4 GPUs

./tools/dist_test.sh ./projects/configs/sphere/sphere-T.py ./path/to/ckpts.pth 4

Acknowledgement

Many thanks to these excellent open source projects:

• SGN
• HASSC
• VoxFormer
• mmdet3d
• MonoScene
• AICNet

Ciatation

If you find this project helpful, please consider citing the following paper:

@inproceedings{yang2024sphere,
  title={SPHERE: Semantic-PHysical Engaged REpresentation for 3D Semantic Scene Completion},
  author={Yang, Zhiwen and Peng, Yuxin},
  booktitle={ACM Multimedia 2025},
  year={2025}
}