Chinmay Savadikar¹, Michelle Dai², Tianfu Wu^1
1North Carolina State University, ²Johns Hopkins University
CVPR 2026
[CVPR Proceedings] | [ArXiv]

Data Preparation

1. Download the ImageNet dataset and place it in data/imagenet/.
2. Download the individual datasets from the MTIL benchmark and place them in data/MTIL/. The expected MTIL dataset directories are configured in conf/benchmark/mtil.yaml.

Training and Evaluation

ImageNet Mean calculation

Before running MTIL experiments, compute the ImageNet feature means used to initialize the component:

bash scripts/imagenet/calculate_imagenet_mean.sh <GPU> <SEED> [vit_base|deit_tiny]

The model variant defaults to vit_base. For example, to compute ViT-Base means on GPU 0 with seed 42:

bash scripts/imagenet/calculate_imagenet_mean.sh 0 42

To compute DeiT-Tiny means:

bash scripts/imagenet/calculate_imagenet_mean.sh 0 42 deit_tiny

The script expects ImageNet at data/imagenet/ and writes the mean checkpoint to:

artifacts/<MODEL>/mlp_down-imagenet-mean/means.pth

where <MODEL> is vit_base or diet_tiny.

Training and evaluating on MTIL

The MTIL benchmark is trained incrementally from the ImageNet initialization task through the remaining 11 downstream tasks:

imagenet -> aircraft -> caltech-101 -> cifar100 -> dtd -> eurosat -> vgg-flowers -> food-101 -> mnist -> oxford-pets -> stanfordcars -> sun397

Before running MTIL, make sure:

• MTIL datasets are available under data/MTIL/.
• ImageNet is available at data/imagenet.
• The MLP ImageNet mean checkpoint exists at artifacts/<MODEL>/mlp_down-imagenet-mean/means.pth, where <MODEL> is vit_base or diet_tiny.
• WANDB_USERNAME in scripts/mtil/run_mlp.sh is updated if LOG_WANDB=True.

Run the MLP experiment with:

bash scripts/mtil/run_mlp.sh <GPU> <SEED> [vit_base|deit_tiny]

The model variant defaults to vit_base. For example, to run ViT-Base on GPU 0 with seed 42:

bash scripts/mtil/run_mlp.sh 0 42

To run DeiT-Tiny:

bash scripts/mtil/run_mlp.sh 0 42 deit_tiny

The script runs search.py, finetune.py, and evaluate.py for each downstream MTIL task. Outputs are written to:

artifacts/<MODEL>/mtil/ee-<SEED><SUFFIX>/

where <MODEL> is vit_base or diet_tiny, and <SUFFIX> is the model-specific suffix defined in the script. Each task's output includes: