RAW-Diffusion: RGB-Guided Diffusion Models for High-Fidelity RAW Image Generation
February 17, 2025 ยท View on GitHub
RAW-Diffusion: RGB-Guided Diffusion Models for High-Fidelity RAW Image Generation
Christoph Reinders, Radu Berdan*, Beril Besbinar*, Junji Otsuka, and Daisuke Iso
Official implementation of the paper "RAW-Diffusion: RGB-Guided Diffusion Models for High-Fidelity RAW Image Generation" [WACV 2025].
Overview
Abstract: Current deep learning approaches in computer vision primarily focus on RGB data sacrificing information. In contrast, RAW images offer richer representation, which is crucial for precise recognition, particularly in challenging conditions like low-light environments. The resultant demand for comprehensive RAW image datasets contrasts with the labor-intensive process of creating specific datasets for individual sensors. To address this, we propose a novel diffusion-based method for generating RAW images guided by RGB images. Our approach integrates an RGB-guidance module for feature extraction from RGB inputs, then incorporates these features into the reverse diffusion process with RGB-guided residual blocks across various resolutions. This approach yields high-fidelity RAW images, enabling the creation of camera-specific RAW datasets. Our RGB2RAW experiments on four DSLR datasets demonstrate state-of-the-art performance. Moreover, RAW-Diffusion demonstrates exceptional data efficiency, achieving remarkable performance with as few as 25 training samples or even fewer. We extend our method to create BDD100K-RAW and Cityscapes-RAW datasets, revealing its effectiveness for object detection in RAW imagery, significantly reducing the amount of required RAW images.
Citation
If you find our work useful, please consider citing our paper:
@inproceedings{reinders2025rawdiffusion,
title = {{{RAW-Diffusion}}: {{RGB-Guided Diffusion Models}} for {{High-Fidelity RAW Image Generation}}},
author={Reinders, Christoph and Berdan, Radu and Besbinar, Beril and Otsuka, Junji and Iso, Daisuke},
booktitle={IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)},
year={2025},
}
Installation
git clone git@github.com:SonyResearch/RAW-Diffusion.git
cd RAW-Diffusion
conda create -n rawdiffusion python=3.11
conda activate rawdiffusion
# Install PyTorch (https://pytorch.org/get-started/locally/)
# The results are produced with PyTorch 2.0.1 and CUDA 11.8
conda install pytorch torchvision pytorch-cuda=11.8 -c pytorch -c nvidia
pip install -r requirements.txt
Datasets
We perform experiments on FiveK Nikon, FiveK Canon, NOD Nikon, and NOD Sony. The details for preparing the datasets are provided in datasets/README.md.
Running Experiments
You can run the experiments either locally or via SLURM. Below are examples for both methods.
Running Locally
To run an experiment locally, use the following command:
python train.py dataset=DATASET general.seed=SEED
# for example
python train.py dataset=fivek_nikon general.seed=0
Running via SLURM
To run an experiment using SLURM, specify the SLURM partition by replacing SLURM_PARTITION with the appropriate partition name, and execute the following command:
python train.py dataset=DATASET general.seed=SEED "hydra.launcher.partition=SLURM_PARTITION" -m
# for example
python train.py dataset=fivek_nikon general.seed=0 "hydra.launcher.partition=SLURM_PARTITION" -m
RGB2RAW
For training and evaluation of the main experiments can be performed as follows.
FiveK Nikon
python train.py dataset=fivek_nikon general.seed=0
python sample.py dataset=fivek_nikon general.seed=0
FiveK Canon
python train.py dataset=fivek_canon general.seed=0
python sample.py dataset=fivek_canon general.seed=0
NOD Nikon
python train.py dataset=nod_nikon general.seed=0
python sample.py dataset=nod_nikon general.seed=0
NOD Sony
python train.py dataset=nod_sony general.seed=0
python sample.py dataset=nod_sony general.seed=0
Small Data RGB2RAW
Training and evaluation of RAW-Diffusion with limited data.
FiveK Nikon
python train.py dataset=fivek_nikon_subsample dataset.train.max_items=25 general.seed=0
python sample.py dataset=fivek_nikon_subsample dataset.train.max_items=25 general.seed=0
FiveK Canon
python train.py dataset=fivek_canon_subsample dataset.train.max_items=25 general.seed=0
python sample.py dataset=fivek_canon_subsample dataset.train.max_items=25 general.seed=0
NOD Nikon
python train.py dataset=nod_nikon_subsample dataset.train.max_items=25 general.seed=0
python sample.py dataset=nod_nikon_subsample dataset.train.max_items=25 general.seed=0
NOD Sony
python train.py dataset=nod_sony_subsample dataset.train.max_items=25 general.seed=0
python sample.py dataset=nod_sony_subsample dataset.train.max_items=25 general.seed=0
Cityscapes-RAW and BDD100K-RAW
Training of RAW-Diffusion on limited amount of RAW images and generation of Cityscapes-RAW and BDD100K-RAW for NOD Nikon and NOD Sony.
NOD Nikon
python train.py dataset=nod_nikon_h416_subsample dataset.train.max_items=100
python sample.py dataset/train=nod_nikon_h416_subsample dataset.train.max_items=100 dataset/val=cityscapes_h416 save_pred=true rgb_only=true
python sample.py dataset/train=nod_nikon_h416_subsample dataset.train.max_items=100 dataset/val=bdd_h416 save_pred=true rgb_only=true diffusion_val.timestep_respacing=ddim6
NOD Sony
python train.py dataset=nod_sony_h416_subsample dataset.train.max_items=100
python sample.py dataset/train=nod_sony_h416_subsample dataset.train.max_items=100 dataset/val=cityscapes_h416 save_pred=true rgb_only=true
python sample.py dataset/train=nod_sony_h416_subsample dataset.train.max_items=100 dataset/val=bdd_h416 save_pred=true rgb_only=true diffusion_val.timestep_respacing=ddim6
Object Detection
The object detection pipeline is provided under object_detection. Please see object_detection/README for details.
Acknowledgements
We are grateful for the following, but not limited to these, wonderful open-source repositories.