BasicLFSR

September 22, 2023 · View on GitHub

BasicLFSR is a PyTorch-based open-source and easy-to-use toolbox for Light Field (LF) image Super-Ressolution (SR). This toolbox introduces a simple pipeline to train/test your methods, and builds a benchmark to comprehensively evaluate the performance of existing methods. Our BasicLFSR can help researchers to get access to LF image SR quickly, and facilitates the development of novel methods. Welcome to contribute your own methods to the benchmark.

Note: This repository will be updated on a regular basis. Please stay tuned!


Contributions

  • We provide a PyTorch-based open-source and easy-to-use toolbox for LF image SR.
  • We re-implement a number of existing methods on the unified datasets, and develop a benchmark for performance evaluation.
  • We share the codes, models and results of existing methods to help researchers better get access to this area.

News & Updates

  • Jul 14, 2023: EPIT is accepted to ICCV 2023.
  • Jul 01, 2023: Update the pre-trained models of benchmark methods to Releases.
  • Jul 01, 2023: Add a new work LF-DET, accepted to IEEE TMM.
  • Mar 31, 2023: Add a new benchmark NTIRE-2023.
  • Mar 29, 2023: Add a new work HLFSR-SSR, accepted to IEEE TCI.
  • Feb 16, 2023: Add a new work EPIT.
  • Dec 10, 2022: Add a new work LFSSR_SAV, accepted to IEEE TCI.

Datasets

We used the EPFL, HCInew, HCIold, INRIA and STFgantry datasets for both training and test. Please first download our datasets via Baidu Drive (key:7nzy) or OneDrive, and place the 5 datasets to the folder ./datasets/.

  • Our project has the following structure: 
    ├──./datasets/
│    ├── EPFL
│    │    ├── training
│    │    │    ├── Bench_in_Paris.mat
│    │    │    ├── Billboards.mat
│    │    │    ├── ...
│    │    ├── test
│    │    │    ├── Bikes.mat
│    │    │    ├── Books__Decoded.mat
│    │    │    ├── ...
│    ├── HCI_new
│    ├── ...
  • Run Generate_Data_for_Training.m or Generate_Data_for_Training.py to generate training data. The generated data will be saved in ./data_for_train/ (SR_5x5_2x, SR_5x5_4x).
  • Run Generate_Data_for_Test.m or Generate_Data_for_Test.py to generate test data. The generated data will be saved in ./data_for_test/ (SR_5x5_2x, SR_5x5_4x).

Commands for Training

  • Run train.py to perform network training. Example for training [model_name] on 5x5 angular resolution for 2x/4x SR: 
    $ python train.py --model_name [model_name] --angRes 5 --scale_factor 2 --batch_size 8
$ python train.py --model_name [model_name] --angRes 5 --scale_factor 4 --batch_size 4
  • Checkpoints and Logs will be saved to ./log/, and the ./log/ has the following structure: 
    ├──./log/
│    ├── SR_5x5_2x
│    │    ├── [dataset_name]
│    │         ├── [model_name]
│    │         │    ├── [model_name]_log.txt
│    │         │    ├── checkpoints
│    │         │    │    ├── [model_name]_5x5_2x_epoch_01_model.pth
│    │         │    │    ├── [model_name]_5x5_2x_epoch_02_model.pth
│    │         │    │    ├── ...
│    │         │    ├── results
│    │         │    │    ├── VAL_epoch_01
│    │         │    │    ├── VAL_epoch_02
│    │         │    │    ├── ...
│    │         ├── [other_model_name]
│    │         ├── ...
│    ├── SR_5x5_4x

Commands for Test

  • Run test.py to perform network inference. Example for test [model_name] on 5x5 angular resolution for 2x/4xSR:

    $ python test.py --model_name [model_name] --angRes 5 --scale_factor 2  
$ python test.py --model_name [model_name] --angRes 5 --scale_factor 4

  • The PSNR and SSIM values of each dataset will be saved to ./log/, and the ./log/ has the following structure:

    ├──./log/
│    ├── SR_5x5_2x
│    │    ├── [dataset_name]
│    │        ├── [model_name]
│    │        │    ├── [model_name]_log.txt
│    │        │    ├── checkpoints
│    │        │    │   ├── ...
│    │        │    ├── results
│    │        │    │    ├── Test
│    │        │    │    │    ├── evaluation.xls
│    │        │    │    │    ├── [dataset_1_name]
│    │        │    │    │    │    ├── [scene_1_name]
│    │        │    │    │    │    │    ├── [scene_1_name]_CenterView.bmp
│    │        │    │    │    │    │    ├── [scene_1_name]_SAI.bmp
│    │        │    │    │    │    │    ├── views
│    │        │    │    │    │    │    │    ├── [scene_1_name]_0_0.bmp
│    │        │    │    │    │    │    │    ├── [scene_1_name]_0_1.bmp
│    │        │    │    │    │    │    │    ├── ...
│    │        │    │    │    │    │    │    ├── [scene_1_name]_4_4.bmp
│    │        │    │    │    │    ├── [scene_2_name]
│    │        │    │    │    │    ├── ...
│    │        │    │    │    ├── [dataset_2_name]
│    │        │    │    │    ├── ...
│    │        │    │    ├── VAL_epoch_01
│    │        │    │    ├── ...
│    │        ├── [other_model_name]
│    │        ├── ...
│    ├── SR_5x5_4x

Benchmark

We benchmark several methods on the above datasets. PSNR and SSIM metrics are used for quantitative evaluation. To obtain the metric score for a dataset with M scenes, we first calculate the metric on AxA SAIs on each scene separately, then obtain the score for each scene by averaging its A^2 scores, and finally obtain the score for this dataset by averaging the scores of all its M scenes.

Note: A detailed review of existing LF image SR methods can be referred to YingqianWang/LF-Image-SR.

PSNR and SSIM values achieved by different methods on 5x5 LFs for 2xSR:

MethodsScale#Params.EPFLHCInewHCIoldINRIASTFgantry
Bilinearx2--28.480/0.918030.718/0.919236.243/0.970930.134/0.945529.577/0.9310
Bicubicx2--29.740/0.937631.887/0.935637.686/0.978531.331/0.957731.063/0.9498
VDSRx20.665M32.498/0.959834.371/0.956140.606/0.986734.439/0.974135.541/0.9789
EDSRx238.62M33.089/0.962934.828/0.959241.014/0.987434.985/0.976436.296/0.9818
RCANx215.31M33.159/0.963435.022/0.960341.125/0.987535.046/0.976936.670/0.9831
resLFx27.982M33.617/0.970636.685/0.973943.422/0.993235.395/0.980438.354/0.9904
LFSSRx20.888M33.671/0.974436.802/0.974943.811/0.993835.279/0.983237.944/0.9898
LF-ATOx21.216M34.272/0.975737.244/0.976744.205/0.994236.170/0.984239.636/0.9929
LF_InterNetx25.040M34.112/0.976037.170/0.976344.573/0.994635.829/0.984338.435/0.9909
LF-DFnetx23.940M34.513/0.975537.418/0.977344.198/0.994136.416/0.984039.427/0.9926
MEG-Netx21.693M34.312/0.977337.424/0.977744.097/0.994236.103/0.984938.767/0.9915
LF-IINetx24.837M34.732/0.977337.768/0.979044.852/0.994836.566/0.985339.894/0.9936
DPTx23.731M34.490/0.975837.355/0.977144.302/0.994336.409/0.984339.429/0.9926
LFTx21.114M34.804/0.978137.838/0.979144.522/0.994536.594/0.985540.510/0.9941
DistgSSRx23.532M34.809/0.978737.959/0.979644.943/0.994936.586/0.985940.404/0.9942
LFSSR_SAVx21.217M34.616/0.977237.425/0.977644.216/0.994236.364/0.984938.689/0.9914
EPITx21.421M34.826/0.977538.228/0.981045.075/0.994936.672/0.985342.166/0.9957
HLFSR-SSRx213.72M35.310/0.980038.317/0.980744.978/0.995037.060/0.986740.849/0.9947
LF-DETx21.588M35.262/0.979738.314/0.980744.986/0.995036.949/0.986441.762/0.9955

PSNR and SSIM values achieved by different methods on 5x5 angular resolution for 4xSR:

MethodsScale#Params.EPFLHCInewHCIoldINRIASTFgantry
Bilinearx4--24.567/0.815827.085/0.839731.688/0.925626.226/0.875725.203/0.8261
Bicubicx4--25.264/0.832427.715/0.851732.576/0.934426.952/0.886726.087/0.8452
VDSRx40.665M27.246/0.877729.308/0.882334.810/0.951529.186/0.920428.506/0.9009
EDSRx438.89M27.833/0.885429.591/0.886935.176/0.953629.656/0.925728.703/0.9072
RCANx415.36M27.907/0.886329.694/0.888635.359/0.954829.805/0.927629.021/0.9131
resLFx48.646M28.260/0.903530.723/0.910736.705/0.968230.338/0.941230.191/0.9372
LFSSRx41.774M28.596/0.911830.928/0.914536.907/0.969630.585/0.946730.570/0.9426
LF-ATOx41.364M28.514/0.911530.880/0.913536.999/0.969930.711/0.948430.607/0.9430
LF_InterNetx45.483M28.812/0.916230.961/0.916137.150/0.971630.777/0.949130.365/0.9409
LF-DFnetx43.990M28.774/0.916531.234/0.919637.321/0.971830.826/0.950331.147/0.9494
MEG-Netx41.775M28.749/0.916031.103/0.917737.287/0.971630.674/0.949030.771/0.9453
LF-IINetx44.886M29.038/0.918831.331/0.920837.620/0.973431.034/0.951531.261/0.9502
DPTx43.778M28.939/0.917031.196/0.918837.412/0.972130.964/0.950331.150/0.9488
LFTx41.163M29.255/0.921031.462/0.921837.630/0.973531.205/0.952431.860/0.9548
DistgSSRx43.582M28.992/0.919531.380/0.921737.563/0.973230.994/0.951931.649/0.9535
LFSSR_SAVx41.543M29.368/0.922331.450/0.921737.497/0.972131.270/0.953131.362/0.9505
EPITx41.470M29.339/0.919731.511/0.923137.677/0.973731.372/0.952632.179/0.9571
HLFSR-SSRx413.87M29.196/0.922231.571/0.923837.776/0.974231.241/0.953431.641/0.9537
LF-DETx41.687M29.473/0.923031.558/0.923537.843/0.974431.389/0.953432.139/0.9573

We provide the result files generated by the aforementioned methods, and researchers can download the results via this link.



NTIRE 2023 LF Image SR Challenge

NTIRE 2023 LFSR Challenge introduces a new LF dataset (namely, NTIRE-2023) for validation and test. Both the validation and testset sets contain 16 synthetic scenes rendered by the 3DS MAX software and 16 real-world images captured by Lytro Illum cameras. For synthetic scenes, all virtual cameras in the camera array have identical internal parameters and are co-planar with the parallel optical axes.

All scenes in the test set have an angular resolution of $5\times 5. The spatial resolutions of synthetic LFs and real-world LFs are \500\times500 and \624\times432, respectively. All the LF images in the test set are bicubicly downsampled by a factor of \4. The participants are required to apply their models to the LR LF images released via [OneDrive](https://stuxidianeducn-my.sharepoint.com/:f:/g/personal/zyliang_stu_xidian_edu_cn/EiEJwlGY3SZDi0FMvHsIMUMB2c73kFsAqELkeidVGoOsKA?e=xHhOEG), and submit their \4\times$ super-resolved LF images to the CodaLab platform for test.

Only the LR versions are released to the participants, and the test server is still online.

Benchmark on NTIRE-2023 Test

MethodsScale#Params.LytroSyntheticAverage
Bicubicx4--25.109/0.840426.461/0.835225.785/0.8378
VDSRx40.665M27.052/0.888827.936/0.870327.494/0.8795
EDSRx438.89M27.540/0.898128.206/0.875727.873/0.8869
RCANx415.36M27.606/0.900128.308/0.877327.957/0.8887
resLFx48.646M28.657/0.926029.245/0.896828.951/0.9114
LFSSRx41.774M29.029/0.933729.399/0.900829.214/0.9173
LF-ATOx41.364M29.087/0.935429.401/0.901229.244/0.9183
LF_InterNetx45.483M29.233/0.936929.446/0.902829.340/0.9198
MEG-Netx41.775M29.203/0.936929.539/0.903629.371/0.9203
LF-IINetx44.886M29.487/0.940329.786/0.907129.636/0.9237
DPTx43.778M29.360/0.938829.771/0.906429.566/0.9226
LFTx41.163M29.657/0.942029.881/0.908429.769/0.9252
DistgSSRx43.582M29.389/0.940329.884/0.908429.637/0.9244
LFSSR_SAVx41.543M29.713/0.942529.850/0.907529.782/0.9250
EPITx41.470M29.718/0.942030.030/0.909729.874/0.9259
HLFSR-SSRx413.87M29.714/0.942929.945/0.909729.830/0.9263
LF-DETx41.687M29.911/0.942029.976/0.910129.944/0.9260

We provide the result files generated by the aforementioned methods, and researchers can download the results via this link.



Citiation

@InProceedings{NTIRE2023LFSR,
  author    = {Wang, Yingqian and Wang, Longguang and Liang, Zhengyu and Yang, Jungang and Timofte, Radu and Guo, Yulan and Jin, Kai and Wei, Zeqiang and Yang, Angulia and Guo, Sha and Gao, Mingzhi and Zhou, Xiuzhuang and Duong, Vinh Van and Huu, Thuc Nguyen and Yim, Jonghoon and Jeon, Byeungwoo and Liu, Yutong and Cheng, Zhen and Xiao, Zeyu and Xu, Ruikang and Xiong, Zhiwei and Liu, Gaosheng and Jin, Manchang and Yue, Huanjing and Yang, Jingyu and Gao, Chen and Zhang, Shuo and Chang, Song and Lin, Youfang and Chao, Wentao and Wang, Xuechun and Wang, Guanghui and Duan, Fuqing and Xia, Wang and Wang, Yan and Xia, Peiqi and Wang, Shunzhou and Lu, Yao and Cong, Ruixuan and Sheng, Hao and Yang, Da and Chen, Rongshan and Wang, Sizhe and Cui, Zhenglong and Chen, Yilei and Lu, Yongjie and Cai, Dongjun and An, Ping and Salem, Ahmed and Ibrahem, Hatem and Yagoub, Bilel and Kang, Hyun-Soo and Zeng, Zekai and Wu, Heng},
  title     = {NTIRE 2023 Challenge on Light Field Image Super-Resolution: Dataset, Methods and Results},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)},
  year      = {2023},
}

Resources

  • The pre-trained models of the aforementioned methods can be downloaded via Releases.

Acknowledgement

We thank Yingqian Wang for the helpful discussions and insightful suggestions regarding this repository.

Contact

Welcome to raise issues or email to zyliang@nudt.edu.cn for any question regarding our BasicLFSR.