BriMA is a multi-modal continual AQA framework designed to handle non-stationary modality imbalance, where different modalities (video, audio, text) appear or disappear across tasks. Instead of relying on noisy imputation or data-heavy generative synthesis, BriMA reconstructs missing modalities through a bridged space that leverages exemplar retrieval and residual correction, ensuring score-preserving feature completion. A modality-aware replay mechanism further stabilizes learning by prioritizing samples with high modality distortion or score drift. Together, these two components enable BriMA to maintain robust, drift-resistant scoring under evolving modality availability in continual learning.

Requirements

• torch==2.0.1+cu118
• torch-geometric==2.3.0
• torchaudio==2.0.1+cu118
• torchvision==0.15.2+cu118
• triton==2.0.0

Usage

Install dependencies:

conda env create -f environment.yml

Preparing Datasets:

Refer to the MLAVL repo instructions to install all the required datasets: RG, Fis-V, and FS1000.

Training from Scratch:

You can train the model in both distributed and dataparallel modes. Below are sample commands for each training setup:

Joint Training Model:

CUDA_VISIBLE_DEVICES=0 python main.py \
    --config configs/sample_config.yaml \
    --dataset sample-dataset --action_type SampleAction \
    --model joint \
    --n_epochs 100 --batch_size 8

3. Train the continual training model:

CUDA_VISIBLE_DEVICES=0 python main.py \
    --config configs/sample_config.yaml \
    --dataset sample-dataset --action_type SampleAction \
    --model continual --n_tasks 10 \
    --n_epochs 50 --batch_size 8 \
    --modality_missing_type random \
    --modality_missing_rate 0.5/0.25/0.1 

Choose the appropriate command based on your training setup and adjust the configurations as needed.

Evaluation:

If you want to perform evaluation using the same configurations as training but with the addition of the --phase test option,

Acknowledgements

If you have any specific questions or if there's anything else you'd like assistance with regarding the code, feel free to let me know.