IT-Blender

Imagine for Me: Creative Conceptual Blending of Real Images and Text via Blended Attention
Wonwoong Cho¹, Yanxia Zhang², Yan-Ying Chen² David Inouye^1
1 Elmore Family School of Electrical and Computer Engineering, Purdue University
² Toyota Research Institute

Key Contributions

IT-Blender is a T2I diffusion adapter that can automate the blending process of visual and textual concepts to enhance human creativity.

• What representations are blended? The latent representations of a real image based on a pretrained diffusion model.

• How are these representations blended? Blended attention, which learns to bridge the gap between the clean reference stream and the noisy generating stream.

• Conceptual blending module that can be generally applicable to diverse creative applications with a single IT-Blender (e.g., Product, Character, Graphic, Fashion, Interior, and Architectural design, Art).

News

• 2025-07-09: ⭐️ Sampling codes of IT-Blender on both SD 1.5 and FLUX have been released.

Quick Start

Setup

1. Environment setup

conda create -n itblender python=3.12
conda activate itblender

2. Requirements installation

pip install -r requirements.txt

3. Accessing the pretrained Models & CLI Login

• Make sure you are logged into your Hugging Face account.

• Accept the specific license terms.

  • For SD 1.5, go to the runwayml/stable-diffusion-v1-5 model page.

  • For FLUX, navigate to the black-forest-labs/FLUX.1-dev model page.

• Read and accept the license agreement.

huggingface-cli login

Usage example

1. FLUX (FLUX.1-dev)

python sample_flux.py

• Peak VRAM usage is 26726MiB with a single gpu.
• Options
  • --scale: A scale for Blended Attention. The default value is 0.6. A value between 0.5 and 0.8 is recommended. (float, default=0.6)
  • --num_ref: The number of reference images. (int, default=1)
  • --num_samples_per_ref : The number of samples to generate per a reference image. (int, default=3)
  • --seed : A random seed. (int, default=42)
  • --obj : An object to generate, e.g., monster cartoon character, dragon, sneakers, and handbag. (str, default="", which is replaced with ["monster cartoon character", "owl cartoon character"])
  • --temperature : A temperature before softmax. Only used if num_ref > 1. Set greater than 1.0 (low temp) if the result does not clearly apply multiple reference images. This sharpens the softmax distribution, possibly helping to prevent ambiguous mixtures of visual concepts. Setting greater than 1.0 can negatively affect the generation quality. A value less than 1.5 is recommended. See appendices of our paper for further details. (float, default=1.0)
  • --ref_preprocessing : Two image preprocessing algorithms are provided to deal with both square and rectangular reference images; Select either one of "resize_centercrop" or "resize_addmargin". (str, default=resize_addmargin)

2. StableDiffusion (SD 1.5)

python sample_sd15.py

• Peak VRAM usage is 6364 MiB with a single gpu.
• Options
  • --scale: A scale for Blended Attention. The default value is 0.25. A value between 0.2 and 0.3 is recommended. (float, default=0.25)
  • --num_samples_per_ref : The number of samples to generate per a reference image. (int, default=4)
  • --seed : A random seed. (int, default=42)
  • --obj : An object to generate, e.g., monster cartoon character, dragon, sneakers, and handbag. (str, default="", which is replaced with "motorcycle")
  • --ref_preprocessing : Two image preprocessing algorithms are provided to deal with both square and rectangular reference images; Select either one of "resize_centercrop" or "resize_addmargin". (str, default=resize_addmargin)

Pretrained Models

Model	Base model	Description	Resolution
IT-Blender FLUX	FLUX.1-dev	The model used in the paper (1.43GB).	(512, 512)
IT-Blender StableDiffusion	SD 1.5	The model used in the paper (99.1MB).	(512, 512)

Feasible Design

The blended results are more practical and feasible when the given cross-modal concepts are semantically close. More examples are provided in our project page.

Societal Impact

• Positive societal impact: IT-Blender can augment human creativity, especially for people in creative industries, e.g., design and marketing. With IT-Blender, designers might be able to have better final design outcome by exploring wide design space in the ideation stage.
• Negative societal impact: IT-Blender can be used to apply the design of an existing product to the new products. The user must be aware of the fact that they can infringe on the company’s intellectual property if a specific texture pattern or material combination is registered. We encourage users to use IT-Blender to augment creativity in the ideation stage, rather than directly having a final design outcome.

License

This project is licensed under the Purdue University.
See the LICENSE file for full license terms.

Citation

@article{tan2024ominicontrol,
  title={OminiControl: Minimal and Universal Control for Diffusion Transformer},
  author={Tan, Zhenxiong and Liu, Songhua and Yang, Xingyi and Xue, Qiaochu and Wang, Xinchao},
  journal={arXiv preprint arXiv:2411.15098},
  year={2024}
}