📖 About

This repository represents the official implementation of the paper titled "OcRFDet: Object-Centric Radiance Fields for Multi-View 3D Object Detection in Autonomous Driving".

We propose Object-Centric Radiance Fields (OcRF) to enhance multi-view 3D object detection by focusing rendering on foreground objects and filtering out background noise. An auxiliary rendering task improves 3D voxel features, while the generated opacity maps are used to refine BEV features through Height-aware Opacity-based Attention (HOA). This explicit geometry-aware design significantly boosts the detector’s ability to understand 3D structure from multi-view RGB images.

GeoEnhDet: External Geometry-Driven Feature Enhancement for Multi-View 3D Object Detection

📖 About

This repository presents the journal extension of OcRFDet, extending the original object-centric radiance field framework with global geometry distillation and more comprehensive evaluations to further improve geometry-aware 3D object detection.

💾 Main Results

nuScenes val set

Config	Frames	mAP	NDS	Model
OcRFDet	2	40.0	50.9	GoogleDrive
GeoEnhDet	2	40.4	51.3	-

nuScenes test set

Config	Frames	mAP	NDS	Results
OcRFDet	8	57.2	64.8	GoogleDrive
GeoEnhDet	8	58.6	66.2	-

Get Started

🛠️ Installation and Data Preparation

1. Please refer to getting_started.md for installing OcRFDet as mmdetection3d. Docker is recommended for environment preparation.
2. Prepare nuScenes dataset as introduced in nuscenes_det.md and create the pkl for OcRFDet by running:

python tools/create_data_bevdet.py

3. Please follow the baseline (DualBEV) to download nuScenes-lidarseg from nuScenes official site and put it under data/nuscenes/. Create depth and semantic labels from point cloud by running:

python tools/generate_point_label.py

4. Please load vggt pretrained model in vggt model and res50 in resnet50. Please combine vggt and resnet50 as the pretrained model:

python tools/combine_res50_vggt.py

Notice: arrange the folder as:

OcRFDet
    └──data
        └── nuscenes
            ├── v1.0-trainval
            ├── sweeps 
            ├── samples
            ├── samples_point_label
            ├── bevdetv2-nuscenes_infos_train.pkl
            └── bevdetv2-nuscenes_infos_val.pkl

4. Please follow the diff-gaussian-rasterization-w-depth to install the Gaussian Splatting renderer by running:

git clone https://github.com/JonathonLuiten/diff-gaussian-rasterization-w-depth.git
cd diff-gaussian-rasterization-w-depth
python setup.py install
pip install .

🏋️ Train OcRFDet model

bash tools/dist_train.sh configs/ocrfdet/ocrfdet.py 8

📋 Test OcRFDet model

bash tools/dist_test.sh configs/ocrfdet/ocrfdet.py  work_dirs/ocrfdet/epoch_20_ema.pth 8 --no-aavt --eval mAP

🏋️ Train GeoEnhDet model

bash tools/dist_train.sh configs/ocrfdet/ocrfdet_vggt.py 8

📋 Test GeoEnhDet model

bash tools/dist_test.sh configs/ocrfdet/ocrfdet_vggt.py  work_dirs/ocrfdet_vggt/epoch_20_ema.pth 8 --no-aavt --eval mAP

👀 Visualize the predicted result

python tools/test.py ./configs/ocrfdet/ocrfdet.py work_dirs/ocrfdet/epoch_20_ema.pth --format-only --eval-options jsonfile_prefix='./work_dirs/visual/'
python tools/analysis_tools/vis0.py ./work_dirs/visual/pts_bbox/results_nusc.json --save_path ./work_dirs/visual/save_dir --draw-gt --vis-thred 0.3 --format image --vis-frames 5 --scale-factor 2
python tools/test.py ./configs/ocrfdet/ocrfdet_vggt.py work_dirs/ocrfdet_vggt/epoch_20_ema.pth --format-only --eval-options jsonfile_prefix='./work_dirs/visual/'
python tools/analysis_tools/vis0.py ./work_dirs/visual/pts_bbox/results_nusc.json --save_path ./work_dirs/visual/save_dir --draw-gt --vis-thred 0.3 --format image --vis-frames 5 --scale-factor 2

❛❛❞ Citation

@article{ji2025ocrfdet,
  title={OcRFDet: Object-Centric Radiance Fields for Multi-View 3D Object Detection in Autonomous Driving},
  author={Ji, Mingqian and Yang, Jian and Zhang, Shanshan},
  journal={arXiv preprint arXiv:2506.23565},
  year={2025}
}

❤️ Acknowledgement

We thank these great works and open-source codebases: MMDetection3D, BEVDet, DualBEV, MVSGaussian, Grendel-GS, diff-gaussian-rasterization-w-depth, vggt.