DGG: Multimodal Continual Instruction Tuning with Dynamic Gradient Guidance

April 4, 2026 · View on GitHub

This is the official repository of CVPR 2026 paper: Multimodal Continual Instruction Tuning with Dynamic Gradient Guidance.

Multimodal Continual Instruction Tuning with Dynamic Gradient Guidance

Songze Li, Mingyu Gao, Tonghua Su*, Xu-Yao Zhang, Zhongjie Wang

Install
Dataset
Train
Evaluation
Citation

Install

1. Clone this repository with submodules

git clone --recursive https://github.com/lisongze/DGG.git
cd DGG

If you have already cloned without --recursive, initialize submodules manually:

git submodule update --init --recursive

Note: This project depends on two custom submodules (DGG-transformers and DGG-deepspeed) that contain our modified implementations of transformers and DeepSpeed. Make sure they are properly initialized before proceeding.

2. Create conda environment and install packages

conda create -n dgg python=3.10 -y
conda activate dgg
pip install --upgrade pip  # enable PEP 660 support
pip install -e .

3. Install custom transformers and DeepSpeed

cd third_party/DGG-transformers && pip install -e . && cd ../..
cd third_party/DGG-DeepSpeed && pip install -e . && cd ../..

4. Install additional packages for training

pip install -e ".[train]"
pip install flash-attn==2.3.4 --no-build-isolation

Dataset

All data is organized under the ./playground directory, which contains two subdirectories:

data/ — stores datasets
model/ — stores model checkpoints (e.g., LLaVA-v1.5-7b, CLIP vision tower, etc.)

VQAv2 Dataset

The VQAv2 dataset directory structure under ./playground/data_VQAv2/:

VQAv2/
├── image/             # VQAv2 images
│   ├── COCO2014/
├── text/              # VQAv2 annotations / questions
│   ├── train/
│   ├── test/
└── replay_data/       # replay annotation data for continual training

Download VQAv2 Data

1. Download COCO2014 images (used as VQAv2 image source):

mkdir -p ./playground/data_VQAv2/image/COCO2014
cd ./playground/data_VQAv2/image/COCO2014
wget http://images.cocodataset.org/zips/train2014.zip
wget http://images.cocodataset.org/zips/val2014.zip
unzip train2014.zip && unzip val2014.zip && rm *.zip

2. Download CL4VQA instructions (VQAv2 question-answer annotations):

Download from CL4VQA Google Drive, then extract the train/ and test/ folders into ./playground/data_VQAv2/text/.

UCIT Dataset

The UCIT dataset directory structure under ./playground/data_UCIT/:

UCIT/
├── ArxivQA/
│   ├── images/
│   ├── test_3000.json
│   └── train_4w.json
├── CLEVR/
│   ├── images/
│   │   ├── train/
│   │   ├── test/
│   │   └── val/
│   ├── test_3000.json
│   └── train_4w.json
├── Flickr30k/
│   ├── train/
│   ├── val/
│   ├── test_3000.json
│   ├── train_brief_4w.json
│   └── val_coco_type_3000.json
├── IconQA/
│   ├── iconqa_data/
│   │   └── iconqa/
│   ├── test_3000.json
│   └── train.json
├── ImageNet-R/
│   ├── train/
│   ├── test/
│   ├── test_3000.json
│   └── train.json
├── VizWiz/
│   ├── train/
│   ├── test/
│   ├── val/
│   ├── test_3000.json
│   ├── train.json
│   └── val_coco_type_3000.json
└── replay_data/       # replay annotation data for continual training

Download UCIT Data

Please refer to HiDe-LLaVA for detailed instructions on downloading and preparing the UCIT dataset (ArxivQA, CLEVR, Flickr30k, IconQA, ImageNet-R, VizWiz).

Prepare Replay Data

To construct instruction tuning data with replay (memory buffer) for continual training, use the scripts under scripts/merge_replay_data/:

Automated (recommended):

# For VQAv2
bash scripts/merge_replay_data/random_sample_vqav2.sh && bash scripts/merge_replay_data/merge_all_vqav2.sh

# For UCIT
bash scripts/merge_replay_data/random_sample_ucit.sh && bash scripts/merge_replay_data/merge_all_ucit.sh

These will automatically sample replay data from previous tasks and merge them with the current task data into replay_data/.

Manual (step-by-step):

Randomly sample replay data from each previous task:

python scripts/merge_replay_data/random_sample_replay_data.py <input_json> <output_json> <sample_size>

Merge current task data with sampled replay data:

python scripts/merge_replay_data/merge_replay_data.py \
    current_task.json repeat_count \
    replay_task1.json repeat_count1 \
    replay_task2.json repeat_count2 \
    ... \
    output_merged.json

Model Checkpoints

Download the required model weights to ./playground/model/:

clip-vit-large-patch14-336 — vision tower (shared by both benchmarks)
LLaVA-v1.5-7b — LLaVA-1.5 base model and pretrained projector (used for UCIT dataset)
llava-v1.5-mlp2x-336px-pretrain-vicuna-7b-v1.5 — Vicuna base model (used for VQAv2 dataset)
vicuna-7b-v1.5 — Vicuna pretrained projector (used for VQAv2 dataset)

Note on base models: We adopt different base models for UCIT and VQAv2 to ensure fair comparison with respective baselines. For UCIT, we follow HiDe-LLaVA which uses LLaVA-v1.5-7b as the base. For VQAv2, we follow CL-MoE which uses Vicuna-7b-v1.5 as the base. This allows direct comparison with reported numbers from both methods.

Train

Note: The current codebase supports single-GPU training only. We recommend running on GPUs with ≥80GB VRAM, such as H800 (80GB) or RTX 6000 PRO (96GB).

We provide training scripts for both benchmarks. All scripts support LoRA-based fine-tuning with our Dynamic Gradient Guidance method.

VQAv2 (10 tasks)

bash scripts/VQAv2/train/train_all.sh

This runs continual instruction tuning across all 10 VQAv2 sub-tasks: recognition, location, judge, commonsense, count, action, color, type, subcategory, causal.

Individual task training scripts are also available in scripts/VQAv2/train/.

UCIT (6 tasks)

bash scripts/UCIT/train/train_all.sh

This runs continual instruction tuning across 6 vision-language tasks: ImageNet-R, ArxivQA, VizWiz, IconQA, CLEVR, Flickr30k.

Individual task training scripts are also available in scripts/UCIT/train/.

Key Training Arguments in Scripts

Argument	Description
`--data_path`	Path to the task's JSON annotation file
`--output_dir`	Directory to save checkpoints
`--history_model_dir`	Path to the previous task's checkpoint (for continual learning)
`--bernolli_coef`	Coefficient for Bernoulli sampling in DGG

Evaluation

VQAv2 Evaluation

bash scripts/VQAv2/eval/Eval_all.sh

UCIT Evaluation

bash scripts/UCIT/eval/Eval_all.sh

Citation

If you find this work useful for your research, please consider citing:

@article{dgg2025,
  title={Multimodal Continual Instruction Tuning with Dynamic Gradient Guidance},
  author={Songze Li, Mingyu Gao, Tonghua Su, Xu-Yao Zhang, Zhongjie Wang},
  journal={arXiv preprint arXiv:2511.15164},
  year={2025}
}