Benchmark Result

Corruption Error (CE)

:triangular_ruler: Metric Setting

The Corruption Error (CE) for a model $A$ under corruption $i$ across 3 severity levels is calculated as:

$$\text{CE}_i^{\text{Model}A} = \frac{\sum_{l=1}((1 - \text{NDS})_{i,l}^{\text{Model}A})}{\sum_{l=1}((1 - \text{NDS})_{i,l}^{\text{Baseline}})} .$$

The average CE for a model $A$ on all corruptions, i.e., mCE, is calculated as:

$$\text{mCE} = \frac{1}{N}\sum^N_{i=1}\text{CE}_i ,$$

where $N=8$ denotes the number of corruption types in our benchmark. We choose DETR3D as the baseline to calculate CE metric.

:bar_chart: Benchmark

Model	NDS	mCE (%) $\downarrow$	Cam Crash	Frame Lost	Color Quant	Motion Blur	Bright	Low Light	Fog	Snow
DETR3D:star:	0.4224	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00
DETR3DCBGS	0.4341	99.21	98.15	98.90	99.15	101.62	97.47	100.28	98.23	99.85
BEVFormerSmall	0.4787	102.40	101.23	101.96	98.56	101.24	104.35	105.17	105.40	101.29
BEVFormerBase	0.5174	97.97	95.87	94.42	95.13	99.54	96.97	103.76	97.42	100.69
PETRR50-p4	0.3665	111.01	107.55	105.92	110.33	104.93	119.36	116.84	117.02	106.13
PETRVoV-p4	0.4550	100.69	99.09	97.46	103.06	102.33	102.40	106.67	103.43	91.11
ORA3D	0.4436	99.17	97.26	98.03	97.32	100.19	98.78	102.40	99.23	100.19
BEVDetR50	0.3770	115.12	105.22	109.19	111.27	108.18	123.96	123.34	123.83	115.93
BEVDetR101	0.3877	113.68	103.32	107.29	109.25	105.40	124.14	123.12	123.28	113.64
BEVDetR101-pt	0.3780	112.80	105.84	108.68	101.99	100.97	123.39	119.31	130.21	112.04
BEVDetSwinT	0.4037	116.48	103.50	106.61	113.18	107.26	130.19	131.83	124.01	115.25
BEVDepthR50	0.4058	110.02	103.09	106.26	106.24	102.02	118.72	114.26	116.57	112.98
BEVerseSwinT	0.4665	110.67	95.49	94.15	108.46	100.19	122.44	130.40	118.58	115.69
BEVerseSwinS	0.4951	107.82	92.93	101.61	105.42	100.40	110.14	123.12	117.46	111.48
PolarFormerR101	0.4602	96.06	96.16	97.24	95.13	92.37	94.96	103.22	94.25	95.17
PolarFormerVoV	0.4558	98.75	96.13	97.20	101.48	104.32	95.37	104.78	97.55	93.14
SRCN3DR101	0.4286	99.67	98.77	98.96	97.93	100.71	98.80	102.72	99.54	99.91
SRCN3DVoV	0.4205	102.64	99.78	100.34	105.13	107.06	101.93	101.10	102.27	92.75
Sparse4DR101	0.5438	100.01	99.80	99.91	98.05	102.00	100.30	103.83	100.46	95.72
SOLOFusionshort	0.3907	108.68	104.45	105.53	105.47	100.79	117.27	110.44	115.01	110.47
SOLOFusionlong	0.4850	97.99	95.80	101.54	93.83	89.11	100.00	99.61	98.70	105.35
SOLOFusionfusion	0.5381	92.86	86.74	88.37	87.09	86.63	94.55	102.22	90.67	106.64
FCOS3Dfinetune	0.3949	107.82	100.14	101.69	108.84	101.24	113.07	118.27	109.71	109.61
BEVFusionCam	0.4122	109.02	101.15	104.72	106.07	98.27	118.34	123.70	114.31	105.59
BEVFusionLiDAR	0.6928	-	-	-	-	-	-	-	-	-
BEVFusionC+L	0.7138	43.80	42.53	41.50	43.32	41.19	49.72	44.54	-	-

Resilience Rate (RR)

:triangular_ruler: Metric Setting

The Resilience Rate (RR) for a model $A$ under corruption $i$ across 3 severity levels is calculated as:

$$\text{RR}_i^{\text{Model}A} = \frac{\sum_{l=1}(\text{NDS}_{i,l}^{\text{Model}A})}{3\times \text{NDS}_{\text{clean}}^{\text{Model}A}} .$$

The average RR for a model $A$ on all corruptions, i.e., mRR, is calculated as:

$$\text{mRR} = \frac{1}{N}\sum^N_{i=1}\text{RR}_i ,$$

where $N=8$ denotes the number of corruption types in our benchmark.

:bar_chart: Benchmark

Model	NDS	mRR (%) $\uparrow$	Cam Crash	Frame Lost	Color Quant	Motion Blur	Bright	Low Light	Fog	Snow
DETR3D:star:	0.4224	70.77	67.68	61.65	75.21	63.00	94.74	65.96	92.61	45.29
DETR3DCBGS	0.4341	70.02	68.90	61.85	74.52	58.56	95.69	63.72	92.61	44.34
BEVFormerSmall	0.4787	59.07	57.89	51.37	68.41	53.69	78.15	50.41	74.85	37.79
BEVFormerBase	0.5174	60.40	60.96	58.31	67.82	52.09	80.87	48.61	78.64	35.89
PETRR50-p4	0.3665	61.26	63.30	59.10	67.45	62.73	77.52	42.86	78.47	38.66
PETRVoV-p4	0.4550	65.03	64.26	61.36	65.23	54.73	84.79	50.66	81.38	57.85
ORA3D	0.4436	68.63	68.87	61.99	75.74	59.67	91.86	58.90	89.25	42.79
BEVDetR50	0.3770	51.83	65.94	51.03	63.87	54.67	68.04	29.23	65.28	16.58
BEVDetR101	0.3877	53.12	67.63	53.26	65.67	58.42	65.88	28.84	64.35	20.89
BEVDetR101-pt	0.3780	56.35	64.60	51.90	80.45	68.52	68.76	36.85	54.84	24.84
BEVDetSwinT	0.4037	46.26	64.63	52.39	56.43	52.71	54.27	12.14	60.69	16.84
BEVDepthR50	0.4058	56.82	65.01	52.76	67.79	61.93	70.95	43.30	71.54	21.27
BEVerseSwinT	0.4665	48.60	68.19	65.10	55.73	56.74	56.93	12.71	59.61	13.80
BEVerseSwinS	0.4951	49.57	67.95	50.19	56.70	53.16	68.55	22.58	57.54	19.89
PolarFormerR101	0.4602	70.88	68.08	61.02	76.25	69.99	93.52	55.50	92.61	50.07
PolarFormerVoV	0.4558	67.51	68.78	61.67	67.49	51.43	93.90	53.55	89.10	54.15
SRCN3DR101	0.4286	70.23	68.76	62.55	77.41	60.87	95.05	60.43	91.93	44.80
SRCN3DVoV	0.4205	67.95	68.37	61.33	67.23	50.96	92.41	54.08	89.75	59.43
Sparse4DR101	0.5438	55.04	52.83	48.01	60.87	46.23	73.26	46.16	71.42	41.54
SOLOFusionshort	0.3907	61.45	65.04	56.18	71.77	66.62	75.92	52.03	76.73	27.28
SOLOFusionlong	0.4850	64.42	65.13	51.34	74.19	71.34	82.52	58.02	82.29	30.52
SOLOFusionfusion	0.5381	64.53	70.73	64.37	75.41	67.68	80.45	48.80	83.26	25.57
FCOS3Dfinetune	0.3949	62.09	72.14	62.78	65.18	65.08	81.49	37.17	84.10	28.77
BEVFusionCam	0.4122	57.81	67.37	54.71	67.03	67.64	70.40	26.10	73.77	35.44
BEVFusionLiDAR	0.6928	-	-	-	-	-	-	-	-	-
BEVFusionC+L	0.7138	97.41	97.55	97.10	98.68	97.74	98.32	95.08	-	-