---

Overview

Why This Problem Is Different

Existing continual image-to-text protocols often build tasks from disjoint object categories or scene labels. This gives clear task boundaries, but it also removes many natural images that contain multiple topics. As a result, the task sequence becomes less similar to the visual streams that models face in deployment. ECA uses main topic splits. Each image is assigned to a task by its dominant semantic topic, while other visible topics remain in the sample. The tasks still shift over time, but shared concepts can appear across different tasks. This setting makes continual learning more realistic. The model must learn the new dominant topic while preserving alignment for earlier concepts that may reappear later.

Main-topic split with realistic cross-topic overlap.

Method

❓ Which VLM architectures does ECA support

ECA is instantiated on BLIP-2-style VLMs and further extended to projector-based MLLMs such as LLaVA-v0. In both cases, ECA updates the alignment module while keeping the major pretrained modules frozen.

❓ What does ECA comprise

❓ How is ECA instantiated

BLIP-2-style VLMs

For BLIP-2-style VLMs, a frozen visual encoder produces patch embeddings. MoQ aggregates task-specific queries with pretrained queries, FeDEx equips the Q-Former with expandable parallel adapters, and DR replays the embedding dictionary to retain former alignment.

Projector-based MLLMs

For projector-based MLLMs, a frozen visual encoder with a pretrained projector produces visual tokens. MoQ generates soft prompt tokens, FeDEx adapts the top L LLM layers, and DR replays dictionary atoms as visual tokens to preserve past visual semantics.

Benchmarks

Each card reports the source dataset, the OpenITG task category, the evaluation metric, and the finalized ToS annotation files. Following the paper, the benchmarks fall into two OpenITG task categories, Image Captioning (IC) and Visual Question Answering (VQA).

Click the card panel to open the Hugging Face annotation repository. Individual annotation files are linked below.

Benchmark	Annotation files
ToS-COCO Caption	train · val · test · topics
ToS-VQAv2	train · val-eval · topics
ToS-TextCaps	train · val · val-eval · topics
ToS-TextVQA	train · val · val-eval · topics

Main-topic distributions

The plots show the topic composition inside each main-topic task. A task is defined by its dominant topic, but other visible topics remain in the samples.

MSCOCO-derived splits for ToS-COCO Caption and ToS-VQAv2.

TextCaps-derived splits for ToS-TextCaps and ToS-TextVQA.

Results Snapshot

The main results below are reported with a pretrained BLIP-2 backbone. The visual encoder and LLM are frozen, and ECA is instantiated on the Q-Former alignment module. The parameter column reports trainable parameters only and is checked against Table 1 and Table 2 in the paper.

Benchmark	Metric	Trainable Params	Avg ↑	BWT ↑	FWT ↑
ToS-COCO Caption	BLEU-4	12.29M	43.42	-0.64	7.39
ToS-COCO Caption	CIDEr	12.29M	125.56	-1.86	20.58
ToS-COCO Caption	SPICE	12.29M	23.80	-0.35	3.00
ToS-VQAv2	VQA Acc	21.74M	68.05	1.81	16.38
ToS-TextCaps	BLEU-4	21.74M	30.05	-0.18	12.13
ToS-TextCaps	CIDEr	21.74M	103.03	1.94	39.22
ToS-TextCaps	SPICE	21.74M	16.86	0.14	4.39
ToS-TextVQA	VQA Acc	21.74M	38.13	2.36	19.30

---

Backbone Support

Methods Reproduced

We include the adapted baseline implementations used in the paper so the continual OpenITG comparisons can be reproduced under the same data split, backbone, and evaluation protocol.

• LwF [paper] · code · config BLIP-2 ☑️ · LLaVA-v0 ☑️ · InternVL-2.5 ⬜
• EWC [paper] · code · config BLIP-2 ☑️ · LLaVA-v0 ☑️ · InternVL-2.5 ☑️
• Dual-Prompt [paper] · code · config BLIP-2 ☑️ · LLaVA-v0 ⬜ · InternVL-2.5 ⬜
• CODA-Prompt [paper] · code · config BLIP-2 ☑️ · LLaVA-v0 ⬜ · InternVL-2.5 ⬜
• MoE-LoRA [paper] · code · config BLIP-2 ☑️ · LLaVA-v0 ☑️ · InternVL-2.5 ⬜
• ModalPrompt [paper] · code · config BLIP-2 ☑️ · LLaVA-v0 ☑️ · InternVL-2.5 ☑️

Quick Start

🧰 Environment

ECA registers BLIP-2, LLaVA-v0, and InternVL modules at startup.

conda create -n eca python=3.8.19 -y
conda activate eca

# Use the same PyTorch and MKL stack as the local ECA environment.
conda install pytorch==2.0.1 torchvision==0.15.2 torchaudio==2.0.2 pytorch-cuda=11.7 numpy==1.24.3 "mkl=2023.1.*" "intel-openmp=2023.1.*" -c pytorch -c nvidia -c defaults -y

# Caption metrics use pycocoevalcap from LAVIS requirements.
# SPICE and METEOR require Java.
conda install -c conda-forge openjdk=8 -y

# Optional: uv only accelerates Python package installation.
pip install uv

# Install dependency packages explicitly. Use pip if uv is unavailable.
uv pip install -r third_party/LAVIS/requirements.txt
uv pip install -r requirements/eca_extra.txt

# Re-apply the tested Transformer stack after dependency resolution.
uv pip install transformers==4.37.2 tokenizers==0.15.1 huggingface-hub==0.23.4

# Install local source packages in editable mode without re-solving dependencies.
uv pip install --no-deps -e third_party/LAVIS
uv pip install --no-deps -e third_party/adapters-0.2.2
uv pip install --no-deps -e third_party/LLaVA
uv pip install --no-deps -e third_party/InternVL/internvl_chat

👉 uv is optional. It only speeds up Python package installation. If you do not use it, replace each uv pip install command with pip install.

👉 Caption and VQA evaluation use COCO metric packages, including pycocoevalcap and pycocotools. SPICE and METEOR also need Java at runtime. The official pycocoevalcap (click link for details) README lists Java 1.8.0, while a system Java installation also works if java is available on PATH.

👉 third_party/LAVIS/ and third_party/adapters-0.2.2/ are patched third-party packages. Use the copies in this repository so the path handling, BLIP-Diffusion gate, and adapter fusion changes are available. The patch files are in third_party/patches/.

👉 third_party/LLaVA/ and third_party/InternVL/ are compatible LLaVA-v0 and InternVL source snapshots for reproducibility. You can replace them with official LLaVA and InternVL installations if import llava and import internvl expose the same APIs. The default checkpoint roots are now checkpoints/LLaVA/ and checkpoints/InternVL/, which keeps source code and model weights separated.

👉 ECA does not use BLIP-Diffusion, so the bash scripts disable its model registration by default with export ECA_ENABLE_BLIP_DIFFUSION=${ECA_ENABLE_BLIP_DIFFUSION:-0}. Set ECA_ENABLE_BLIP_DIFFUSION=1 only if you want to restore the original LAVIS BLIP-Diffusion registration and have installed its optional dependencies.

🗂️ ToS Annotations

The ToS benchmark annotations are hosted on 🤗 Hugging Face at Snowball0823/ECA-ToS-Benchmarks. Download them into the runtime cache/ layout before training or evaluation.

huggingface-cli download Snowball0823/ECA-ToS-Benchmarks \
  --repo-type dataset \
  --local-dir ECA-ToS-Benchmarks \
  --local-dir-use-symlinks False

mkdir -p cache/coco/annotations cache/TextCaps
cp ECA-ToS-Benchmarks/annotations/coco/*.json cache/coco/annotations/
cp ECA-ToS-Benchmarks/annotations/text/*.json cache/TextCaps/

Images are not redistributed with the ToS annotations. Download the original MSCOCO, TextCaps, and TextVQA images from their official sources and place them under cache/coco/images/ and cache/TextCaps/images/.

🧩 Pretrained Checkpoints

🎯 BLIP-2

BLIP-2 weights are managed by LAVIS. On the first run, LAVIS downloads the BLIP-2 OPT checkpoint and facebook/opt-2.7b automatically, so no manual BLIP-2 folder is needed under checkpoints/.

🎯 InternVL

InternVL weights are loaded from local paths in the configs. Download the model sizes you need into checkpoints/InternVL/:

mkdir -p checkpoints/InternVL

huggingface-cli download OpenGVLab/InternVL2_5-1B \
  --local-dir checkpoints/InternVL/InternVL2_5-1B \
  --local-dir-use-symlinks False

huggingface-cli download OpenGVLab/InternVL2_5-4B \
  --local-dir checkpoints/InternVL/InternVL2_5-4B \
  --local-dir-use-symlinks False

huggingface-cli download OpenGVLab/InternVL2_5-8B \
  --local-dir checkpoints/InternVL/InternVL2_5-8B \
  --local-dir-use-symlinks False

🎯 LLaVA-v0

LLaVA-v0 requires the full merged checkpoint. Follow the official LLaVA weight preparation instructions and place the final checkpoint at checkpoints/LLaVA/llava-7b-v0. If you use a config with a separate base model path, place Vicuna at checkpoints/LLaVA/vicuna-7b-v0.

🎯 Checkpoint links

👉 🤗 InternVL2.5-1B link

👉 🤗 InternVL2.5-4B link

👉 🤗 InternVL2.5-8B link

👉 LLaVA official weight instructions link

🚀 Train

Example ECA training command:

bash train.sh configs/train/ecaq_cl_train_cap.yaml

For multi-GPU training, set NPROC_PER_NODE before launching:

NPROC_PER_NODE=4 bash train.sh configs/train/ecaq_cl_train_cap.yaml

Before launching a fresh run, check whether the selected config contains stale resume_ckpt_path, output_dir, or local checkpoint paths from a previous machine.

Config Map

Paper-main BLIP-2/Q-Former training configs:

Benchmark	Train config
ToS-COCO Caption	configs/train/ecaq_cl_train_cap.yaml
ToS-VQAv2	configs/train/ecaq_cl_train_vqa.yaml
ToS-TextCaps	configs/train/ecaq_cl_train_textcaps.yaml
ToS-TextVQA	configs/train/ecaq_cl_train_textvqa.yaml

Backbone extension examples:

Backbone	Example train config
LLaVA-v0	configs/train/eca_llava_cl_train_cap.yaml
InternVL	configs/train/eca_internvl_cl_train_textvqa.yaml

Citation

If this repository is useful for your research, please cite:

@inproceedings{kong2026eca,
  title = {ECA: Efficient Continual Alignment for Open-Ended Image-to-Text Generation},
  author = {Kong, Jiangtao and Zhao, Peijun and Chen, Chun-Fu and Do, Youngwook and Hu, Shaohan and Zhou, Tianyi and Shao, Huajie},
  booktitle = {Proceedings of the 43rd International Conference on Machine Learning (ICML)},
  year = {2026},
  note = {To appear}
}

Acknowledgements

This codebase builds on the following third-party projects and research code. Please also follow their licenses and usage requirements.

• BLIP-2 and LAVIS
• LLaVA-v0 and InternVL
• AdapterHub adapters
• DualPrompt, CODA-Prompt, ModalPrompt, and CoIN
• pycocoevalcap and pycocotools
• Source datasets including COCO, VQAv2, TextVQA, and TextCaps