📌 Release

[03/2025] Code and paper are publicly available.

📑 Abstract

TL;DR: Conceptrol is a free lunch that elicits the personalized ability of zero-shot adapter by transforming image condition to visual specification contrained by textual concept, even outperforming fine-tuning methods.

CLICK for the full abstract

Personalized image generation with text-to-image diffusion models generates unseen images based on reference image content. Zero-shot adapter methods such as IP-Adapter and OminiControl are especially interesting because they do not require test-time fine-tuning. However, they struggle to balance preserving personalized content and adherence to the text prompt. We identify a critical design flaw resulting in this performance gap: current adapters inadequately integrate personalization images with the textual descriptions. The generated images, therefore, replicate the personalized content rather than adhere to the text prompt instructions. Yet the base text-to-image has strong conceptual understanding capabilities that can be leveraged.

We propose Conceptrol, a simple yet effective framework that enhances zero-shot adapters without adding computational overhead. Conceptrol constrains the attention of visual specification with a textual concept mask that improves subject-driven generation capabilities. It achieves as much as 89% improvement on personalization benchmarks over the vanilla IP-Adapter and can even outperform fine-tuning approaches such as Dreambooth LoRA. The source code will be released upon acceptance.

🚗 Quick Start

1. Environment Setup

conda create -n conceptrol python=3.10
conda activate conceptrol
pip install -r requirements.txt

2. Go to demo_sd.ipynb / demo_sdxl.ipynb / demo_flux.py for fun!

🛳️ Local Setup using Gradio

1. Start Gradio Interface

pip install gradio
gradio gradio_src/app.py

2. Use the GUI for fun!

📝 Supporting Models

Model Name	Link
Stable Diffusion 1.5	stable-diffusion-v1-5/stable-diffusion-v1-5
Realistic Vision V5.1	SG161222/Realistic_Vision_V5.1_noVAE
Stable Diffusion XL-1024	stabilityai/stable-diffusion-xl-base-1.0
Animagine XL v4.0	cagliostrolab/animagine-xl-4.0
Realistic Vision XL V5.0	SG161222/RealVisXL_V5.0
FLUX-schnell	black-forest-labs/FLUX.1-schnell

Adapter Name	Link
IP-Adapter	h94/IP-Adapter
OminiControl	Yuanshi/OminiControl

✒️Citation

If you found this repository/our paper useful, please consider citing:

@article{he2025conceptrol,
  title={Conceptrol: Concept Control of Zero-shot Personalized Image Generation},
  author={Qiyuan He and Angela Yao},
  journal={arXiv preprint arXiv:2503.06568},
  year={2025}
}

❤️ Acknowledgement

We thank the following repositories for their great work:

diffusers, transformers, IP-Adapter, OminiControl