README.md

April 22, 2022 · View on GitHub

With recent advances in stereo vision, dual cameras are commonly adopted in mobile phones and autonomous vehicles. Using the complementary information provided by binocular systems, the resolution of image pairs can be enhanced by stereo image super-resolution (SR) algorithms. In this repository, we first present a collection of datasets and papers on stereo image SR, together with their codes or repos. Then, we develop a benchmark to comprehensively evaluate milestone and state-of-the-art methods. Welcome to raise issues regarding our survey and submit novel results (better together with original files and source codes) to our benchmark.

Note: This repository will be updated on a regular basis, so stay tuned~~🎉🎉🎉

News:

Challenges:

NTIRE 2022 Stereo Image Super-Resolution Challenge

Datasets:

DatasetPublication
Flickr1024Flickr1024: A Large-Scale Dataset for Stereo Image Super-Resolution, ICCVW 2019
MiddleburyHigh-resolution stereo datasets with subpixel-accurate ground truth, GCPR 2014
KITTI 2012Are we ready for Autonomous Driving? The KITTI Vision Benchmark Suite, CVPR 2012
KITTI 2015Object Scene Flow for Autonomous Vehicles, CVPR 2015

Methods:

MethodPublicationOfficial Repository
StereoSREnhancing the Spatial Resolution of Stereo Images using a Parallax Prior, CVPR 2018PeterZhouSZ/
stereosr
PASSRnetLearning Parallax Attention for Stereo Image Super-Resolution, CVPR 2019LongguangWang/
PASSRnet
SAMA Stereo Attention Module for Stereo Image Super-Resolution, SPL 2020XinyiYing/SAM
SPAMnetStereoscopic image super-resolution with stereo consistent feature, AAAI 2020.--
DASSRDisparity-Aware Domain Adaptation in Stereo Image Restoration, CVPR 2020.--
IMSSRnetDeep Stereoscopic Image Super-Resolution via Interaction Module, TCSVT 2020.--
CPASSRnetCross Parallax Attention Network for Stereo Image Super-Resolution, TMM 2021.canqChen/
CPASSRnet
BSSRnetDeep Bilateral Learning for Stereo Image Super-Resolution, SPL 2021.xuqingyu26/
BSSRnet
iPASSRSymmetric Parallax Attention for Stereo Image Super-Resolution, CVPRW 2021.YingqianWang/
iPASSR
DFAMA Disparity Feature Alignment Module for Stereo Image Super-Resolution, SPL 2021.JiawangDan/DFAM
CVCnetCross View Capture for Stereo Image Super-Resolution, TMM 2021.xyzhu1/CVCnet
SSRDE-FNetFeedback Network for Mutually Boosted Stereo Image Super-Resolution and Disparity Estimation, ACM MM 2021.MIVRC/SSRDEFNet-PyTorch
SVSRNetStereo video super-resolution via exploiting view-temporal correlations, ACM MM 2021.--
PSSRPerception-Oriented Stereo Image Super-Resolution, ACM MM 2021.--
NAFSSRNAFSSR: Stereo Image Super-Resolution Using NAFNet, CVPRW 2022.megvii-research/
NAFNet
Trans-SVSRA New Dataset and Transformer for Stereoscopic Video Super-Resolution, CVPRW 2022.H-deep/
Trans-SVSR

Benchmark

We benchmark several methods on the KITTI 2012, KITTI 2015, Middlebury and Flickr1024 datasets. The test sets used below can be downloaded from Google Drive and Baidu Drive (Key: NUDT). Note that, All these methods have been retrained on the same training set (60 images from the Middlebury dataset and 800 images from the Flickr1024 dataset) for fair comparison.

PSNR and SSIM metrics are used for quantitative evaluation, which are first calculated on each view (without boundary cropping) independently, then averaged between left and right views to generate the score of a scene. Finally, the score of a dataset is obtained by averaging the scores of all its scenes.

PSNR and SSIM values achieved by different methods for 2xSR:

MethodScale#Params.KITTI 2012KITTI 2015MiddleburyFlickr1024
Bicubic2×—28.51/0.884228.61/0.897330.60/0.899024.94/0.8186
VDSR2×0.66M30.30/0.908929.78/0.915032.77/0.910225.60/0.8534
EDSR2×38.6M30.96/0.922830.73/0.933534.95/0.949228.66/0.9087
RDN2×22.0M30.94/0.922730.70/0.933034.94/0.949128.64/0.9084
RCAN2×15.3M31.02/0.923230.77/0.933634.90/0.948628.63/0.9082
StereoSR2×1.08M29.51/0.907329.33/0.916833.23/0.934825.96/0.8599
PASSRnet2×1.37M30.81/0.919030.60/0.930034.23/0.942228.38/0.9038
BSSRnet2×1.89M31.03/0.924130.74/0.934434.74/0.947528.53/0.9090
iPASSR2×1.37M31.11/0.924030.81/0.934034.51/0.945428.60/0.9097
SSRDE-FNet2×2.10M31.23/0.925430.90/0.935235.09/0.951128.85/0.9132
NAFSSR-T2×0.45M31.26/0.925430.99/0.935535.01/0.949528.94/0.9128
NAFSSR-S2×1.54M31.38/0.926631.08/0.936735.30/0.951429.19/0.9160
NAFSSR-B2×6.77M31.55/0.928331.22/0.938035.68/0.954429.54/0.9204
NAFSSR-L2×23.8M31.60/0.929131.25/0.938635.88/0.955729.68/0.9221

PSNR and SSIM values achieved by different methods for 4xSR:

MethodScale#Params.KITTI 2012KITTI 2015MiddleburyFlickr1024
Bicubic4×—24.58/0.737224.38/0.734026.40/0.757221.82/0.6293
VDSR4×0.66M25.60/0.772225.32/0.770327.69/0.794122.46/0.6718
EDSR4×38.9M26.35/0.801526.04/0.803929.23/0.839723.46/0.7285
RDN4×22.0M26.32/0.801426.04/0.804329.27/0.840423.47/0.7295
RCAN4×15.4M26.44/0.802926.22/0.806829.30/0.839723.48/0.7286
StereoSR4×1.08M24.53/0.755524.21/0.751127.64/0.802221.70/0.6460
PASSRnet4×1.42M26.34/0.798126.08/0.800228.72/0.823623.31/0.7195
SRRes+SAM4×1.73M26.44/0.801826.22/0.805428.83/0.829023.27/0.7233
BSSRnet4×1.91M26.47/0.804926.17/0.807529.08/0.836223.40/0.7289
iPASSR4×1.42M26.56/0.805326.32/0.808429.16/0.836723.44/0.7287
SSRDE-FNet4×2.24M26.70/0.808226.43/0.811829.38/0.841123.59/0.7352
NAFSSR-T4×0.46M26.79/0.810526.62/0.815929.32/0.840923.69/0.7384
NAFSSR-S4×1.56M26.93/0.814526.76/0.820329.72/0.849023.88/0.7468
NAFSSR-B4×6.80M27.08/0.818126.91/0.824530.04/0.856824.07/0.7551
NAFSSR-L4×23.8M27.12/0.819426.96/0.825730.20/0.860524.17/0.7589

Recources

We provide some original super-resolved images and useful resources to facilitate researchers to reproduce the above results.