RoboBEV Benchmark

SurroundDepth

Corruption	Abs Rel	Sq Rel	RMSE	RMSE log	a1	a2	a3
Clean	0.280	4.400	7.466	0.364	0.661	0.844	0.917
Camera	0.485	7.288	12.105	0.934	0.322	0.440	0.544
Frame	0.497	7.230	12.560	0.983	0.285	0.402	0.513
Bright	0.339	6.520	8.433	0.413	0.623	0.807	0.886
Dark	0.354	4.078	9.837	0.575	0.430	0.647	0.776
Motion	0.338	4.246	10.443	0.607	0.433	0.636	0.758
Quant	0.334	4.207	8.712	0.516	0.481	0.694	0.811
Fog	0.320	4.544	8.203	0.427	0.578	0.781	0.876
Snow	0.423	5.781	12.775	0.802	0.331	0.516	0.640

Experiment Log

Camera Crash

Severity	Abs Rel	Sq Rel	RMSE	RMSE log	a1	a2	a3
Easy	0.425	6.729	10.617	0.762	0.454	0.587	0.668
Moderate	0.515	7.358	12.738	1.041	0.262	0.365	0.472
Hard	0.516	7.778	12.960	1.001	0.251	0.368	0.493
Average	0.485	7.288	12.105	0.934	0.322	0.440	0.544

Frame Lost

Severity	Abs Rel	Sq Rel	RMSE	RMSE log	a1	a2	a3
Easy	0.398	5.879	10.240	0.706	0.458	0.598	0.691
Moderate	0.519	7.522	13.044	1.042	0.249	0.360	0.475
Hard	0.576	8.291	14.397	1.201	0.148	0.248	0.374
Average	0.497	7.230	12.560	0.983	0.285	0.402	0.513

Color Quant

Severity	Abs Rel	Sq Rel	RMSE	RMSE log	a1	a2	a3
Easy	0.282	3.988	7.558	0.378	0.634	0.830	0.908
Moderate	0.313	4.070	8.223	0.458	0.526	0.750	0.857
Hard	0.409	4.564	10.355	0.714	0.285	0.503	0.668
Average	0.334	4.207	8.712	0.516	0.481	0.694	0.811

Motion Blur

Severity	Abs Rel	Sq Rel	RMSE	RMSE log	a1	a2	a3
Easy	0.274	3.338	8.212	0.415	0.588	0.792	0.884
Moderate	0.345	4.177	10.870	0.633	0.395	0.606	0.740
Hard	0.397	5.224	12.247	0.773	0.316	0.510	0.650
Average	0.338	4.246	10.443	0.607	0.433	0.636	0.758

Brightness

Severity	Abs Rel	Sq Rel	RMSE	RMSE log	a1	a2	a3
Easy	0.303	5.281	7.832	0.387	0.647	0.826	0.901
Moderate	0.341	6.568	8.414	0.413	0.623	0.807	0.886
Hard	0.374	7.711	9.053	0.439	0.600	0.789	0.873
Average	0.339	6.520	8.433	0.413	0.623	0.807	0.886

Low Light

Severity	Abs Rel	Sq Rel	RMSE	RMSE log	a1	a2	a3
Easy	0.349	4.065	9.411	0.566	0.432	0.654	0.783
Moderate	0.349	3.941	9.710	0.568	0.434	0.653	0.782
Hard	0.364	4.229	10.391	0.593	0.425	0.636	0.764
Average	0.354	4.078	9.837	0.575	0.430	0.647	0.776

Fog

Severity	Abs Rel	Sq Rel	RMSE	RMSE log	a1	a2	a3
Easy	0.308	4.681	7.884	0.390	0.620	0.817	0.901
Moderate	0.319	4.527	8.165	0.424	0.579	0.784	0.880
Hard	0.333	4.425	8.562	0.469	0.535	0.743	0.849
Average	0.320	4.544	8.203	0.427	0.578	0.781	0.876

Snow

Severity	Abs Rel	Sq Rel	RMSE	RMSE log	a1	a2	a3
Easy	0.378	4.804	11.444	0.677	0.395	0.592	0.715
Moderate	0.421	5.748	12.869	0.789	0.344	0.525	0.647
Hard	0.470	6.792	14.013	0.940	0.256	0.432	0.560
Average	0.423	5.781	12.775	0.802	0.331	0.516	0.640

References

@inproceedings{wei2023surrounddepth,
  title={Surrounddepth: Entangling surrounding views for self-supervised multi-camera depth estimation},
  author={Wei, Yi and Zhao, Linqing and Zheng, Wenzhao and Zhu, Zheng and Rao, Yongming and Huang, Guan and Lu, Jiwen and Zhou, Jie},
  booktitle={Conference on Robot Learning},
  pages={539--549},
  year={2023},
  organization={PMLR}
}