[CVPR'25 Oral] LoRASculpt

July 24, 2025 ยท View on GitHub

This repository is built for the paper LoRASculpt: Sculpting LoRA for Harmonizing General and Specialized Knowledge in Multimodal Large Language Models.

method

News

  • [2025-05] Repo created. Code will be released soon.
  • [2025-07] Code released.

Preparation

  1. Clone this repository and navigate to LoRASculpt folder:

    git clone https://github.com/LiangJian24/LoRASculpt
cd LoRASculpt

  2. Install package:

    conda create -n lorasculpt python=3.10 -y
conda activate lorasculpt
pip install --upgrade pip
pip install -e .

  3. Install additional packages for training cases:

    pip install -e ".[train]"
pip install flash-attn --no-build-isolation

  4. Download the required datasets and place them in the corresponding folder.

Usage

  1. Set the correct paths in the scripts under ./scripts/v1_5.

  2. Run the following training script to train on downstream task:

    bash ./scripts/v1_5/train/ours-train.sh

  3. Run the following script to evaluate upstream and downstream performance:

    bash ./scripts/v1_5/eval/eval_all.sh

Citation

If you find LoRASculpt useful for your research and applications, please cite using this BibTeX:

@InProceedings{Liang_2025_CVPR,
    author    = {Liang, Jian and Huang, Wenke and Wan, Guancheng and Yang, Qu and Ye, Mang},
    title     = {LoRASculpt: Sculpting LoRA for Harmonizing General and Specialized Knowledge in Multimodal Large Language Models},
    booktitle = {CVPR},
    year      = {2025}
}

Acknowledgement

Our repo is built on LLaVA. We thank the authors for sharing their code.

Keeping Yourself is Important in Downstream Tuning Multimodal Large Language Model [Paper][Project Page]