Region-to-Region

August 4, 2025 · View on GitHub

Official implementation of the paper "Region-to-Region: Enhancing Generative Image Harmonization with Adaptive Regional Injection".

Note:

  1. This repository is released solely for anonymous review purposes. The paper is currently under review.
  2. To ensure a double-blind review process, the issue section has been disabled during the review period.
  3. All download links are anonymous.

Installation

cd Region_to_Region
conda env create -f environment.yml -n r2r
conda activate r2r

Download Checkpoints

Download LDM checkpoint: here
Download Clear-VAE checkpoint: here

Place them under inference/checkpoints. forming the following structure:

inference/
└── checkpoints/
    ├── clear_vae/
    ├── stable-diffusion-inpainting/

Dataset Preparation

iHarmony4

Please refer to the DiffHarmony to prepare data. Place the iHarmony4 dataset inside the data folder, forming the following structure:

data/iHarmony4
|- HAdobe5k/
    |- composite_images
    |- masks
    |- real_images
    |- ...
|- HCOCO/
|- HFlickr/
|- Hday2night/
|- train.jsonl
|- test.jsonl

RPHarmony

Please download it from here and place it under the data folder. Make sure the structure is just like that:

data/RPHarmony
|- R-ADE20K
    |- composite_images
    |- masks
    |- real_images
    |- ...
|- R-DUTS
|- train.jsonl
|- test.jsonl
|- train.txt
|- test.txt

The RPHarmony dataset is built using our proposed Random Poisson Blending method.
The implementation of Poisson Blending is based on pytorch-poisson-image-editing.

Inference

cd inference

For using pretrained checkpoints, see the Download Checkpoints section for instructions.

Test the model performance on the iHarmony4 dataset:

sh scripts/inference.sh

To test on other datasets, simply change the parameters in the inference.sh. You can customize the testing settings by editing the inference.sh script.

Train

The weights of our model are derived from DiffHarmony.
We offer converted weights compatible with ControlNet-style training.

Please download the pretrained model weights and place them in the ./ckpt directory. Alternatively, you can manually convert the weights using the tool_add_control.py

Please modify the configuration files in the ./configs, such as datasets.yaml, and then:

sh train.sh

For detailed training options and configurations, see train_control.py.

Train Clear-VAE

The VAE is initialized with weights from Stable Diffusion. Please download the pretrained VAE weights from here and place them in the ./ckpt directory.

python train_VAE.py

For detailed training options and configurations, see train_VAE.py.

Model weights convert

If you use the weights obtained from training, please run inference/convert.py to convert them from Stable Diffusion (SD) format to Diffusers format.
For parameter settings, please refer to the convert.py.

cd inference
python convert.py  --ckpt_path *** --save_path *** --vae_path *** --vae_save_path ***

The convert.py provides weight conversion for the core components (e.g., ControlNet, clear-VAE). However, you still need to download the remaining weights (e.g., safety_checker) from here.

VAE inference

Download our pretrained Clear-VAE checkpoint (Stable Diffusion format) from here, modify the ckpt_path in configs/vae.yaml, and then run:

python infer_VAE.py

Acknowledgements

This project is developped on the codebase of ControlNet, AnyDoor, DiffHarmony, FinetuneVAE-SD and pytorch-poisson-image-editing. We appreciate these great work!