Structure Guided Diffusion for Portrait Shadow Removal

September 26, 2025 · View on GitHub

Environment

pip install -r requirement.txt

Data Prepare

  • Download the FFHQ 128×128 or FFHQ 512×512 dataset and place all images in the train_C folder.
  • Using DPR, perform random relighting on the original images in train_C; then, combine the relit images with the original images using randomly generated masks. Save the masks to the train_B folder and the synthesized images to the train_A folder.
  • Process corresponding images from train_B and train_C using ./structure_guided_diffusion_models/data/prepare_data.py; the results will be saved in the train_A_1 folder.
  • Use pre-trained models from Unpaired-Portrait-Drawing to extract structure maps from train_C (saved to train_edge for Stage 1 & 2). Then, extract gradient maps from train_A (saved to train_detail for Stage 3).

Training Pipeline

###Stage 1: SE-Net Required folder structure:

dataset/train/
├── train_A/          # Synthesized images
├── train_edge/       # structure maps
└── train_C/          # Original FFHQ images

Commands:

cd /SE-Net
# Training
bash ./scripts/train.sh

# Testing
python test_seq_style.py --dataroot [input_folder] --savefolder [save_folder_name]

###Stage 2: Structure-Guided Portrait Inpainting Required folder structure:

dataset/train/
├── train_A_1/        # Processed images from prepare_data.py
├── train_B/          # Random masks
├── train_C/          # Original FFHQ images
└── train_edge/       # structure maps

Commands:

cd /structure_guided_diffusion_models
# Training
python sr.py -p train -c config/sample_sr3.json

# Testing
python sr.py -p val -c config/sample_sr3.json
# Save results to result_0 folder

###Stage 3: Gradient-guided Detail Restoration Required folder structure:

dataset/train/
├── result_0/         # Output from Stage 2
├── train_B/          # Random masks
├── train_C/          # Original FFHQ images
└── train_detail/     # Gradient maps

Commands:

cd /structure_guided_diffusion_models
# Training
python sr.py -p train -c config/sample_sr3.json

# Testing
python sr.py -p val -c config/sample_sr3.json

Acknowledgements

Our work is based on the following theoretical works: