🔥 News

• [2026-02-09] 🎉 OmniHD-Scenes has been accepted to IEEE TPAMI.

• [2025-07-28] 🚀 Our codebase and detection models have been released.

• [2025-04-15] 🎉 OmniHD-Scenes dataset v1.0 (~1.3TB) is now accessible. For research access, simply download the Data Use Agreement and submit the signed document via email to contact@2077ai.com. You will receive a JSON configuration file. You can then use the provided Python script to download the full dataset from Alibaba Cloud OSS using the JSON key.

• [2024-12-31] 🌐 The project page is now online.

🛠️ Abstract

We present OmniHD-Scenes, a large-scale multimodal dataset that provides comprehensive omnidirectional high-definition data. The OmniHD-Scenes dataset combines data from 128-beam LiDAR, six cameras, and six 4D imaging radar systems to achieve full environmental perception. To date, we have annotated 200 clips with more than 514K precise 3D bounding boxes. These clips also include semantic segmentation annotations for static scene elements. Alongside the proposed dataset, we establish comprehensive evaluation metrics, baseline models, and benchmarks for 3D detection and semantic occupancy prediction.

⚙️ Dataset Structure

OmniHD-Scenes is structured in clips, drawing inspiration from nuScenes' data composition format. The dataset is organized as follows.

OmniHD-Scenes
├── 1693714828633418               # Clip Scene
│   ├── cameras                    # Six Cameras
│   │   ├── camera_back                
│   │   │   ├── xxxxxxxxx.jpg         
│   │   │   └── ... 
│   │   ├── camera_front                
│   │   │   ├── xxxxxxxxx.jpg          
│   │   │   └── ... 
│   │   ├──...
│   │   ├── camera_right_front
│   │   │   ├── xxxxxxxxx.jpg          
│   │   │   └── ...                
│   ├── lidar                    # LiDAR
│   │   ├── lidar_top_compensation             
│   │   │   ├── xxxxxxxxx.bin          
│   │   │   └── ... 
│   ├── radars                    # Six 4D Radars
│   │   ├── radar_back                
│   │   │   ├── xxxxxxxxx.bin         
│   │   │   └── ... 
│   │   ├── radar_front                
│   │   │   ├── xxxxxxxxx.bin          
│   │   │   └── ... 
│   │   ├──...
│   │   ├── radar_right_front
│   │   │   ├── xxxxxxxxx.bin          
│   │   │   └── ...
├──...
├── 1693922406733409               
├── v1.0-trainval
│   ├── annotations.json           # 3D box label
│   ├── ego_pose.json                # ego pose
│   ├── imu.json                   # ego status
│   ├── meta.json                   # scene description
│   ├── sample_data.json           # index of all frames
│   ├── sample.json                # key frames
│   ├── scene_split.json           # train/test split
│   └── sensor_calibration.json    # calib parameters

🔨 Quick Start

Download Dataset

python download_oss.py --json-file xxxxx.json --download-dir [your path]

Installation

You can install the whole repository by following these steps:

Clone

git clone https://github.com/TJRadarLab/OmniHD-Scenes.git

Create environment

conda create -n omnihd python=3.8 -y
conda activate omnihd

Install pytorch

pip install torch==1.9.1+cu111 torchvision==0.10.1+cu111 torchaudio==0.9.1 -f https://download.pytorch.org/whl/torch_stable.html

Install mmcv/mmdet/mmseg

pip install mmcv-full==1.4.0 -f https://download.openmmlab.com/mmcv/dist/cu111/torch1.9.0/index.html
pip install mmdet==2.14.0
pip install mmsegmentation==0.14.1

Install mmdet3d

git clone https://github.com/open-mmlab/mmdetection3d.git
cd mmdetection3d
git checkout v0.17.1 
pip install -v -e .  

Install others

pip install scikit-image==0.19.3  
pip install einops fvcore seaborn iopath==0.1.9 timm==0.6.13  typing-extensions==4.5.0 pylint ipython==8.12  numpy==1.19.5 matplotlib==3.5.2 numba==0.48.0 pandas==1.4.4 scikit-image==0.19.3 setuptools==59.5.0 torch_scatter==2.1.1
python -m pip install 'git+https://github.com/facebookresearch/detectron2.git'
pip install yapf==0.40.1
python projects/setup.py develop
python projects/setup_bevpool2.py develop

Generate PKL

💡 Note: We provide a suggested train/val split for hyperparameter tuning. For final benchmarking on the Test set, please train on the full trainval set.

Generate PKL file for only 3D object detection

python ./newscenes_devkit/newscenes_converter_final.py

Generate PKL file for Occupancy&OD

python ./tools/merge_data_with_occ.py

Or you can download our generated pkl_files.

Test

Test a baseline model

./tools/dist_test.sh ./projects/configs/XXX/XXX.py ./work_dirs/XXX/XXX.pth 2

🍁 Baseline Results

🚀 Model Zoo

In this repository, we release baseline models for 3D object detection and occupancy prediction.

Methods	Modality	Image Size	Backbone	mAP	ODS	Models
PointPillars	LiDAR	---	---	61.15	55.54	Link
PointPillars	4D Radar	---	---	23.82	37.21	Link
RadarPillarNet	4D Radar	---	---	24.88	37.81	Link
LSS	Camera	544×960	R50	22.44	26.01	Link
BEVFormer-T	Camera	544×960	R50	29.17	30.54	Link
BEVFormer-T	Camera	864×1536	R101-DCN	32.22	32.57	Link
BEVFusion	Camera+4D Radar	544×960	R50	33.95	43.00	Link
RCFusion	Camera+4D Radar	544×960	R50	34.88	41.53	Link

Methods	Modality	Image Size	Backbone	SC IoU	mIoU	Models
BEVFusion-OCC	Camera+4D Radar	544×960	R50	27.02	16.24	Link

⏳ To Do

• Release the CodeBase
• Release the Evaluation Devkit (For historical reasons, it is referred to by its original project name newscenes_devkit within the code)
• Release OD baseline model
• Release the OCC label
• Release OCC baseline model

⭐ Others

If you have any questions about the dataset, feel free to cantact us with tjradarlab@163.com & contact@2077ai.com.

🎬 Video Demo

😙 Acknowledgement

Many thanks to these exceptional open source projects:

• BEVFormer
• mmdet3d
• nuScenes-devkit
• SurroundOcc
• BEVFusion

As it is not possible to list all the projects of the reference papers. If you find we leave out your repo, please contact us and we'll update the lists.

📃Citation

@article{zheng2024omnihd,
  title={OmniHD-Scenes: A next-generation multimodal dataset for autonomous driving},
  author={Zheng, Lianqing and Yang, Long and Lin, Qunshu and Ai, Wenjin and Liu, Minghao and Lu, Shouyi and Liu, Jianan and Ren, Hongze and Mo, Jingyue and Bai, Xiaokai and others},
  journal={arXiv preprint arXiv:2412.10734},
  year={2024}
}

⭐️ Star History