MER-DG: Modality-Entropy Regularization for Multimodal Domain Generalization

Yavuz Yarici  Ghassan AlRegib

Georgia Institute of Technology (OLIVES Lab)

Published at ICML 2026

[Read the Paper Here]

---

Abstract

Deploying multimodal models in real-world scenarios requires generalization to new environments where recording conditions differ from training, a challenge known as multimodal domain generalization (MMDG). Standard architectures employ separate encoders for each modality and a fusion module, training the system end-to-end by optimizing on the fused features. In this paper, we identify that such joint optimization causes encoders to exploit cross-modal co-occurrences, statistical relationships between modalities that arise from source-specific recording conditions, rather than learning domain-invariant features. We term this failure mode Fusion Overfitting. To address this, we propose Modality-Entropy Regularization for Domain Generalization (MER-DG), which maximizes the entropy of each encoder's feature distribution to preserve feature diversity. MER-DG is architecture-agnostic and integrates into existing multimodal frameworks as an additive loss term. Extensive experiments on EPIC-Kitchens and HAC benchmarks demonstrate average improvements of approximately 5% over standard fusion and approximately 2% over state-of-the-art methods.

Code

The code was tested using Python 3.10.4, torch 1.11.0+cu113.

Environments:

mmcv-full 1.2.7
mmaction2 0.13.0

EPIC-Kitchens & HAC Datasets Preparation

Download Pretrained Weights

1. Download Audio model link, rename it as vggsound_avgpool.pth.tar and place under the EPIC-rgb-flow-audio/pretrained_models and HAC-rgb-flow-audio/pretrained_models directories.

2. Download SlowFast model for RGB modality link and place under the pretrained_models directories.

3. Download SlowOnly model for Flow modality link and place under the pretrained_models directories.

Download Datasets

• EPIC-Kitchens: Download Audio files EPIC-KITCHENS-audio.zip. Follow the original EPIC-Kitchens extraction format.
• HAC: Download at link.

(See the original SimMMDG repository for the exact desired directory tree structures for the datasets).

Running the Code (Experiments)

We provide clean compilation scripts for both datasets to run our MER-DG approach alongside the standard Baseline Fusion and the state-of-the-art SimMMDG framework.

Each directory contains a unified run_experiments.sh script that organizes configuring and training the models. Before running:

• Edit EPIC-rgb-flow-audio/run_experiments.sh or HAC-rgb-flow-audio/run_experiments.sh
• Point DATAPATH= to where you stored the datasets locally.

EPIC-Kitchens

cd EPIC-rgb-flow-audio
bash run_experiments.sh

HAC Dataset

cd HAC-rgb-flow-audio
bash run_experiments.sh

By default, the scripts execute the following experiments sequentially:

1. Baseline Fusion
2. Baseline Fusion + MER-DG
3. SimMMDG Baseline
4. SimMMDG + MER-DG

Modify the script to isolate specific experiments. Ensure wandb is configured for metric logging.

Citation

@inproceedings{yarici2026merdg,
    title={MER-DG: Modality-Entropy Regularization for Multimodal Domain Generalization},
    author={Yarici, Yavuz and AlRegib, Ghassan},
    booktitle={2026 International Conference on Machine Learning (ICML)},
    note={Accepted on April 30, 2026},
    year={2026}
}

Acknowledgement

This codebase is adapted from the SimMMDG framework. We sincerely thank the authors for open-sourcing their code.