Training for Identity, Inference for Controllability: A Unified Approach to Tuning-Free Face Personalization

Official Implementation of "Training for Identity, Inference for Controllability: A Unified Approach to Tuning-Free Face Personalization" by Lianyu Pang, Ji Zhou, Qiping Wang, Baoquan Zhao, Zhenguo Yang, Li Qing and Xudong Mao.

Introduction

We introduce UniID, a unified tuning-free framework that synergistically combines text embedding and adapter approaches while preserving both identity fidelity and text controllability. Our key insight is that when merging the two branches, they should mutually reinforce only identity information, while non-identity aspects such as scene composition are controlled by the original diffusion model's prior knowledge. Specifically, during training, we employ an identity-focused learning scheme that guides both the text embedding and adapter branches to capture exclusively identity-relevant features. At inference, we introduce a normalized rescaling strategy that recovers the text controllability of the original diffusion model in both branches while enabling their complementary identity signals to mutually reinforce each other. Through this strategic training-inference paradigm, UniID achieves superior identity fidelity while preserving the text controllability of the original model.

Release

• 2025/12/04: We have released the code and model weights!

Setup

Setting Up the Environment

To set up the environment, run the following commands:

conda env create -f environment.yaml
conda activate uniid

Setting Up Accelerate

Initialize an Accelerate environment with:

accelerate config

Download

Datasets

Our model was trained using the training sets of CelebA-HQ, FFHQ-Portrait and a filtered version of FaceID-6M.

Face Recognition Model

We use Partial-FC as our face recognition model.

Model Weights

For the trained model presented in our paper, please visit our HuggingFace repository:

🔗 HuggingFace Repository: UniID

Usage

Running Inference

You can run the bash_scripts/inference.sh script to generate images. Before running the script, configure the following parameters in inference.sh:

• Line 3: text_branch_checkpoint_dir. Path to the directory containing the checkpoints of the text branch.
• Line 4: adapter_branch_checkpoint_dir. Path to the directory containing the checkpoints of the adapter branch.
• Line 5: pretrained_face_recog_model_path. Path to the face recognition model. Specify this path if you are using the model weights we provide. If you trained the model yourself, you can omit this argument; it will be loaded from the training configuration.
• Line 6: ref_image_path. Path to the reference image.
• Line 7: save_dir. Directory where the generated images will be saved.
• Line 8: prompt. The input prompt. For convenience, you can also specify a path to a text file containing prompts (one per line) using --prompt_file. For example:

  A photo of a <class word>
  A <class word> eating bread in front of the Eiffel Tower
  A <class word> latte art
  

  The token <class word> in the prompts will be automatically replaced with man or woman.

To run the inference script:

bash bash_scripts/inference.sh

• We recommend using RealVisXL as the base model in inference instead of SDXL for better results.
• The generated images will be saved to the directory {save_dir}/{prompt}
• For a full list of parameters, please refer to inference.py and inference.sh.

Training

You can run the bash_scripts/train_uniid.sh script to train your own model. Before running the script, configure the following parameters in train_uniid.sh:

• Line 3: pretrained_face_recog_model_path. Path to the face recognition model.
• Line 4: data_path_file. This file should list the paths to all training images, with one path per line. For example:

  dataset/CelebA-HQ/train/00000.jpg
  dataset/CelebA-HQ/train/00001.jpg
  ...
  dataset/ffhq/train/00001.jpg
  

  You can generate this file using the write_dataset_to_file function in utils.py.
• Line 5: output_root. Directory where the trained model will be saved.

To run the training script:

bash bash_scripts/train_uniid.sh

Notes:

• All training arguments are listed in train_uniid.sh and are set to the default values used in the paper.
• Please refer to the script for more details on each parameter.

Acknowledgements

We would like to express our sincere gratitude to the creators and maintainers of CelebA-HQ, FFHQ-Portrait, FaceID-6M, Diffusers, Celeb Basis and IP-Adapter. Our project builds upon their foundational work and is made possible by their dedication to the open-source community.

Citation

@article{UniID,
  title = {Training for Identity, Inference for Controllability: A Unified Approach to Tuning-Free Face Personalization},
  author = {Lianyu Pang, Ji Zhou, Qiping Wang, Baoquan Zhao, Zhenguo Yang, Qing Li, Xudong Mao},
  journal = {arXiv preprint arXiv:2512.03964},
  year = {2025}
}