Video restoration based on deep learning: a comprehensive survey

Claudio Rota, Marco Buzzelli, Simone Bianco, Raimondo Schettini

Artificial Intelligence Review, Springer Nature, 2022

[Open Access PDF]

Abstract

Video restoration concerns the recovery of a clean video sequence starting from its degraded version. Different video restoration tasks exist, including denoising, deblurring, super-resolution, and reduction of compression artifacts. In this paper, we provide a comprehensive review of the main features of existing video restoration methods based on deep learning. We focus our attention on the main architectural components, strategies for motion handling, and loss functions. We analyze the standard benchmark datasets and use them to summarize the performance of video restoration methods, both in terms of effectiveness and efficiency. In conclusion, the main challenges and future research directions in video restoration using deep learning are highlighted.

Citation

@article{rota2023video,
  title={Video restoration based on deep learning: a comprehensive survey},
  author={Rota, Claudio and Buzzelli, Marco and Bianco, Simone and Schettini, Raimondo},
  journal={Artificial Intelligence Review},
  volume={56},
  number={6},
  pages={5317--5364},
  year={2023},
  publisher={Springer}
}

Methods for video restoration

Name	Task(s)	Paper	Code
VESPCN	Super-resolution	Real-time video super-resolution with spatio-temporal networks and motion compensation (CVPR 2017)	[Pytorch]
DBN	Deblurring	Deep Video Deblurring for Hand-held Cameras (CVPR 2017)	[Torch]
STRCNN	Deblurring	Online video deblurring via dynamic temporal blending network (ICCV 2017)	:x:
DBLRGAN	Deblurring	Adversarial spatio-temporal learning for video deblurring (TIP 2018)	:x:
DUF	Super-resolution	Deep Video Super-Resolution Network Using Dynamic Upsampling FiltersWithout Explicit Motion Compensation (CVPR 2018)	[TensorFlow]
MFQE	Compression artifact reduction	Multi-Frame Quality Enhancement for Compressed Video (CVPR 2018)	[TensorFlow]
ToFlow	Denoising, Super-resolution, Compression artifact reduction	Video Enhancement with Task-Oriented Flow (IJCV 2019)	[Torch]
DVDNet	Denoising	Dvdnet: A fast network for deep video denoising (ICIP 2019)	[Pytorch]
MFQE2.0	Compression artifact reduction	MFQE 2.0: A New Approach for Multi-frame Quality Enhancement on Compressed Video (TPAMI 2019)	[TensorFlow]
STFAN	Deblurring	Spatio-Temporal Filter Adaptive Network for Video Deblurring (ICCV 2019)	[Pytorch]
EDVR	Deblurring, Super-resolution	Edvr: Video restoration with enhanced deformable convolutional networks (CVPR 2019)	[Pytorch]
PFNL	Super-resolution	Progressive Fusion Video Super-Resolution Network via Exploiting Non-Local Spatio-Temporal Correlations (ICCV 2019)	[Pytorch/TensorFlow]
ViDeNN	Denoising	ViDeNN: Deep Blind Video Denoising (CVPR 2019)	[TensorFlow]
IFI-RNN	Deblurring	Recurrent neural networks with intra-frame iterations for video deblurring (CVPR 2019)	:x:
FITVNet	Denoising	First image then video: A two-stage network for spatiotemporal video denoising (2019)	:x:
VNLNet	Denoising	A Non-Local CNN for Video Denoising (ICIP 2019)	:x:
NL-ConvLSTM	Compression artifact reduction	Non-local convlstm for video compression artifact reduction (ICCV 2019)	[Pytorch]
MB2D	Deblurring	Blur more to deblur better: Multi-blur2deblur for efficient video deblurring (2020)	:x:
FastDVDNet	Denoising	Fastdvdnet: Towards real-time deep video denoising without flow estimation (CVPR 2020)	[Pytorch]
TDAN	Super-resolution	Tdan: Temporally-deformable alignment network for video super-resolution (CVPR 2020)	[Pytorch]
ESTRNN	Deblurring	Efficient spatio-temporal recurrent neural network for video deblurring (ECCV 2020)	[Pytorch]
STDF	Compression artifact reduction	Spatio-temporal deformable convolution for compressed video quality enhancement (AAAI 2020)	[Pytorch]
MuCAN	Super-resolution	Mucan: Multi-correspondence aggregation network for video super-resolution (ECCV 2020)	[Pytorch]
RViDeNet	Denoising	Supervised raw video denoising with a benchmark dataset on dynamic scenes (CVPR 2020)	[Pytorch]
RSDN	Super-resolution	Video super-resolution with recurrent structure-detail network (ECCV 2020)	[Pytorch]
CDVD-TSP	Deblurring	Cascaded deep video deblurring using temporal sharpness prior (CVPR 2020)	[Pytorch]
Evrnet	Denoising, Super-resolution, Compression artifact reduction	Evrnet: Efficient video restoration on edge devices (ICM 2021)	:x:
MMNet	Denoising	Multiframe-to-multiframe network for video denoising (TOM 2021)	:x:
RFDA	Compression artifact reduction	Recursive fusion and deformable spatiotemporal attention for video compression artifact reduction (ICM 2021)	[Pytorch]
PVDNet	Deblurring	Recurrent video deblurring with blur-invariant motion estimation and pixel volumes (TOG 2021)	[Pytorch]
MaskDNGAN	Denoising	Multi-Stage Raw Video Denoising with Adversarial Loss and Gradient Mask (ICCP 2021)	[Pytorch]
PaCNet	Denoising	Patch craft: Video denoising by deep modeling and patch matching (ICCV 2021)	[Pytorch]
BasicVSR	Super-resolution	BasicVSR: The search for essential components in video super-resolution and beyond (CVPR 2021)	[Pytorch]
BasicVSR++	Super-resolution	BasicVSR++: Improving video super-resolution with enhanced propagation and alignment (CVPR 2022)	[Pytorch]

Datasets for video restoration

Name	Task(s)	Usage	Repository
GOPRO	Deblurring	Train/Test	[Link]
DVD	Deblurring	Train/Test	[Link]
BSD	Deblurring	Train/Test	[Link]
REDS	Deblurring, Super-resolution	Train/Test	[Link]
Vid4	Super-resolution	Test	[Link]
UDM10	Super-resolution	Test	[Link]
SPMCS	Super-resolution	Test	[Link]
Vimeo90K	Denoising, Super-resolution, Compression artifact reduction	Train/Test	[Link]
MFQEv2	Compression artifact reduction	Train/Test	[Link]
CRVD	Denoising	Train/Test	[Link]
Set8	Denoising	Test	[Link]
DAVIS 2017	Denoising	Train/Test	[Link]

Contacts

If you have any question, please contact Claudio Rota at c.rota30@campus.unimib.it