Introduction

This repository is an official implementation of PreSight: Enhancing Autonomous Vehicle Perception with City-Scale NeRF Priors.

News

• [2024/07/20]: :tada: We have released the code of PreSight!

• [2024/07/09]: :confetti_ball: Our paper has been accepted by the The 18th European Conference on Computer Vision (ECCV 2024)! Our code will be release this month. Stay tuned!

Main Results

Online HD Mapping

Model	Metric	w. Prior	Ped Crossing	Divider	Boundary	All	Runtime (FPS)
StreamMapNet	AP	×	10.19	11.26	11.87	11.10	22.4
StreamMapNet	AP	✓	21.11	23.73	32.31	25.72 (+14.62)	21.9
MapTR	AP	×	4.97	8.20	9.83	7.67	25.2
MapTR	AP	✓	16.18	19.04	34.14	23.12 (+15.45)	23.2
BEVFormer	IoU	×	14.90	29.88	32.74	25.84	15.5
BEVFormer	IoU	✓	16.37	34.82	51.66	34.28 (+8.44)	14.3

Occupancy

Method	w. Priors	mIoU	Dynamic	Static	others	barrier	bicycle	bus	car	constr. vehicle	motorcycle	pedestrian	traffic cone	truck	drive surface	other flat	sidewalk	terrain	manmade	vegetation	Runtime (FPS)
BEVDet	×	29.3	24.4	38.2	1.5	42.4	11.0	43.0	47.1	19.1	23.3	23.4	19.5	37.8	72.9	11.6	30.9	48.6	32.7	32.5	5.1
BEVDet	✓	33.7 (+4.4)	24.4	50.5 (+12.3)	1.2	40.1	14.8	42.1	48.3	15.7	26.4	24.4	18.7	37.2	81.8	15.2	40.3	60.5	50.4	54.9	4.9
FB-Occ	×	30.0	25.1	39.2	9.2	37.2	21.8	41.6	43.4	15.8	27.3	25.4	23.8	30.3	74.7	17.3	33.0	50.6	28.2	31.1	9.1
FB-Occ	✓	34.3 (+4.3)	25.4	50.7 (+11.5)	9.3	38.3	21.0	40.3	45.0	15.9	29.9	26.0	23.8	30.2	82.3	18.5	39.1	61.2	48.0	54.7	8.6

Getting Started

To get started, please follow the instructions below step-by-step.

• Installation
• Prepare Data
• Building Priors
• Training Perception Models with Priors

Pretrained Weights

Extracted Priors

	Boston-Seaport	Singapore-Onenorth	Singapore-Queenstown	Singapore-Hollandvillage
Google Drive	Download	Download	Download	Download

Perception Models

	Vectorized Online Mapping	Occupancy Prediction
Google Drive	Download	Download

TODO

• Add scripts to inference per-image monocular depth using Depth-Anything to enable training NeRFs with monocular-depth loss. Monocular-depth loss improves visualization quality but do not improve downstream perception metrics.

Acknowledgement

This project builds upon the outstanding work of several open-source projects. We extend our sincere thanks to the following codebases:

• Nerfstudio
• EmerNeRF
• BEVDet
• FB-BEV
• MapTR
• StreamMapNet
• SegFormer
• SUDS

Citation

If you find our work useful in your research, please consider citing:

@article{yuan2024presight,
  title={PreSight: Enhancing Autonomous Vehicle Perception with City-Scale NeRF Priors},
  author={Yuan, Tianyuan and Mao, Yucheng and Yang, Jiawei and Liu, Yicheng and Wang, Yue and Zhao, Hang},
  journal={arXiv preprint arXiv:2403.09079},
  year={2024}
}