Overview

MixSA (Mixture-of-Self-Attention) is a training-free sketch extraction method that leverages strong diffusion priors for enhanced sketch perception. It overcomes the limitations of both data-driven methods and diffusion-based style transfer approaches, providing consistent and high-quality sketch extraction without requiring extensive training.

Demos

Method Comparison

Data-driven sketch extraction methods (a) struggle to adapt to unseen reference styles, while diffusion-based style transfer methods (b) fail to disentangle overall styles from sketch-specific styles, resulting in inconsistent sketch transfers. MixSA effectively addresses both limitations without requiring training.

Performance Comparison

Comparison with existing training-based and training-free methodologies. MixSA consistently surpasses state-of-the-art methods (Ref2Sketch, Semi-ref2sketch, StyleID, InstantStyle, and IP-Adapter) in preserving the reference sketch's brush strokes and artistic style across various input images.

Setup

Our codebase is built on StyleID and has similar dependencies and model architecture.

Create a Conda Environment

conda env create -f environment.yaml
conda activate mixsa

Download Pre-trained Weights

StableDiffusion

Download the StableDiffusion weights from the CompVis organization at Hugging Face (download the sd-v1-4.ckpt file), and link them:

ln -s <path/to/model.ckpt> models/ldm/stable-diffusion-v1/model.ckpt 

U2NET

Download the u2net.pth from:

• Google Drive
• Baidu Pan (提取码: chgd)

Place the model in the directory: ./U2Net_/saved_models/

HED

Download the network-bsds500.pytorch from Hugging Face and place it in the directory: ./checkpoint/

Usage

After setting up the environment and downloading the required weights, you can run the model using:

python run.py --content <content_image_path> --reference <reference_sketch_path> --output <output_path>

Citation

If you find MixSA useful for your research and applications, please cite our work:

@ARTICLE{10758215,
  author={Yang, Rui and Wu, Xiaojun and He, Shengfeng},
  journal={IEEE Transactions on Visualization and Computer Graphics}, 
  title={MixSA: Training-Free Reference-Based Sketch Extraction via Mixture-of-Self-Attention}, 
  year={2025},
  volume={31},
  number={9},
  pages={6208-6222},
  keywords={Sketch extraction;image representations;image generation;image-to-image translation},
  doi={10.1109/TVCG.2024.3502395}}

Contact

For any questions, please feel free to contact us via:

• rane@snnu.edu.cn
• rui.xjtu@gmail.com

License

This project is released under the MIT License. See LICENSE for details.