iDPA

iDPA — Instance Decoupled Prompt Attention for Incremental Medical Object Detection — is a framework designed for incremental learning in medical object detection. It builds on a pre-trained Vision-Language Object Detection (VLOD) model and enables the system to learn new medical object categories over time (organs, lesions, structures, etc.) while preserving performance on previously learned categories.

Key Concepts

• Instance-level Prompt Generation (IPG) — For each object instance (defined by a bounding box) in a medical image, iDPA extracts region-level (instance) features and generates a dedicated prompt. This allows the prompt to focus on fine-grained, instance-specific visual-semantic information rather than global or background context.
• Decoupled Prompt Attention (DPA) — Rather than prepending prompts to the global image/text token sequence (which may dilute prompt influence), iDPA decouples the prompt attention mechanism: prompt-to-vision, prompt-to-text, and vision–text mutual attention are handled separately via residual connections. This enables more direct and efficient injection of prompt-derived knowledge while reducing memory use and mitigating catastrophic forgetting.

Repository Structure

Below is a high-level view of the typical layout:

configs/                 # configuration files: dataset paths, class definitions, prompt & training / inference settings  
maskrcnn_benchmark/      # baseline detection / model backbone (VLOD-based)  
script/                  # scripts for training, prompt generation, inference, evaluation, incremental steps  
tools/                   # utilities: dataset processing, annotation conversion, few-shot sampling, evaluation, data preparation etc.  
environment.yaml         # (or requirements.txt) — dependencies specification  
setup.py                 # optional installation setup  
README_GLIP.md           # (optional) instructions for using a specific baseline VLOD model (e.g. GLIP)  

• configs/ defines datasets, class names, prompt settings (length, initialization), training/inference hyperparameters.
• maskrcnn_benchmark/ (or similar) provides the VLOD backbone + detection head to build upon.
• script/ contains user-facing scripts: training, incremental learning, prompt generation, inference, evaluation, etc.
• tools/ hosts helper utilities: dataset conversion, few-shot sampling, evaluation metrics, data loaders, etc.
• Environment files specify dependencies (Python, PyTorch, CUDA, etc.).

Installation

git clone https://github.com/HarveyYi/iDPA.git
cd iDPA
# if using conda
conda env create -f environment.yaml
conda activate <env_name>

# or using pip
pip install -r requirements.txt

Quick Start

# prepare
bash script/idpa_f/infer_T_zero_feature.sh

# Run training
bash script/idpa_f/train.sh
bash script/idpa_f/save.sh


# Run evaluation
bash script/idpa_f/eval.sh

Citation

If you use iDPA in your research or application, please cite:

@article{Yi2025iDPA,
  title = {iDPA: Instance Decoupled Prompt Attention for Incremental Medical Object Detection},
  author = {Huahui Yi and Wei Xu and Ziyuan Qin and Xi Chen and Xiaohu Wu and Kang Li and Qicheng Lao},
  year = {2025},
  note = {Accepted by ICML 2025},
  url = {https://github.com/HarveyYi/iDPA},
}