coadapter.md

March 22, 2023 ยท View on GitHub

Overview

We introduce CoAdapter (Composable Adapter) by jointly training T2I-Adapters and an extra fuser. The fuser allows different adapters with various conditions to be aware of each other and synergize to achieve more powerful composability, especially the combination of element-level style and other structural information.

CoAdapter is inspired by Composer. However, instead of training the whole model, it only trains extra light-weight adapters based on T2I-Adapter. But CoAdapter can also show the capability of generating creative images with composibility. Note that the model is still in training and this release is only a preview.

Demos

SketchCannyDepthColor (Spatial)StyleResults
imageimageimage
imageimageimage
imageimageimageimage
imageimageimageimage

Benefits from CoAdapter

CoAdapter offers two advantages over the original T2I-Adapter. You can try CoAdapter in Huggingface CoAdapter.

  1. CoAdapter has improved composability, especially for the style modality, due to the joint training of multiple adapters.
Input 1Input 2Input3Prompt and seedT2I-AdapterCoAdapter
image
Canny: 1.0		image
Style: 1.0		"tower"
seed=42		imageimage
image
Skecth: 1.0		image
Style: 1.0		image
Color: 1.0		"motorbike"
seed=993		imageimage
image
Skecth: 1.0		image
Style: 1.0		"a corgi"
seed=42		imageimage
  1. The joint training of CoAdapter can also enhance the generation quality of each individual adapter. TODO

The above results are based on coadapter-sd15v1 and t2iadapter-sd14v1.

Useful Tips

  • Condition weight is important. If the generated image is not well aligned with the condition, increase the corresponding Condition weight. If increasing Condition weight is ineffective or degrades image quality, try decreasing the Condition weight of other conditions.
  • Start with fewer conditions. If you plan to use more than two conditions to control the model, it is recommended to start with only one or two conditions. Once you have found the suitable Condition weight for existing conditions, gradually append the new conditions.
  • It is recommended to use a step size of 0.1 to adjust Condition weight. From experience, Condition weight will not be less than 0.5 or greater than 1.5

Training

python train.py -t --base configs/pl_train/coadapter-v1-train.yaml --gpus 0,1,2,3,4,5,6,7 --scale_lr False --num_nodes 1 --sd_finetune_from models/v1-5-pruned-emaonly.ckpt --name coadapter-v1-train --auto_resume