AMU-Tuning: Effective Logit Bias for CLIP-based Few-shot Learning

April 19, 2024 · View on GitHub

This is an code implementation base on Mindspore2.2 and pytorch 1.7.1 of CVPR 2024 paper AMU-Tuning: Effective Logit Bias for CLIP-based Few-shot Learning

Introduction

This paper proposes a novel AMU-Tuning method to learn effective logit bias for CLIP-based few shot classification. Specifically, our AMU-Tuning predicts logit bias by exploiting the appropriate Auxiliary features, which are fed into an efficient feature-initialized linear classifier with Multi-branch training. Finally, an Uncertainty based fusion is developed to incorporate logit bias into CLIP for few-shot classification. The experiments are conducted on several widely used benchmarks, and the results show AMU-Tuning clearly outperforms its counterparts while achieving state-of-the-art performance of CLIP based few-shot learning without bells and whistles.

Usage

Environments

●OS：16.04
●CUDA：11.6
●Toolkit：MindSpore2.2 & PyTorch 1.7.1
●GPU:GTX 3090

Requirements

Install

create virtual enviroment and install dependencies:

git clone https://github.com/TJU-sjyj/MindSpore-AMU
conda create -n AMU python=3.7
conda activate AMU
# Install the according versions of torch and torchvision
conda install pytorch torchvision cudatoolkit

CUDA 10.1

conda install mindspore-gpu cudatoolkit=10.1 -c mindspore -c conda-forge

CUDA 11.1

conda install mindspore-gpu cudatoolkit=11.1 -c mindspore -c conda-forge

validataion

python -c "import mindspore;mindspore.run_check()"

Dataset

Our dataset setup is primarily based on Tip-Adapter. Please follow DATASET.md to download official ImageNet and other 10 datasets.

Foundation Models

  • The pre-tained weights of CLIP will be automatically downloaded by running.
  • The pre-tained weights of MoCo-v3 can be download at MoCo v3.

Get Started

One-Line Command by Using run.sh

We provide run.sh with which you can complete the pre-training + fine-tuning experiment cycle in an one-line command.

Arguments

  • clip_backbone is the name of the backbone network of CLIP visual coders that will be used (e.g. RN50, RN101, ViT-B/16).
  • lr learning rate for adapter training.
  • shots number of samples per class used for training.
  • alpha is used to control the effect of logit bias.
  • lambda_merge is a hyper-parameter in Multi-branch Training

More Arguments can be referenced in parse_args.py

Training Example

You can use this command to train a AMU adapter with ViT-B-16 as CLIP's image encoder by 16-shot setting for 50 epochs.

CUDA_VISIBLE_DEVICES=0 python train.py\
    --rand_seed 2 \
    --torch_rand_seed 1\
    --exp_name test_16_shot  \
    --clip_backbone "ViT-B-16" \
    --augment_epoch 1 \
    --init_alpha 0.5\
    --lambda_merge 0.35\
    --train_epoch 50\
    --lr 1e-3\
    --batch_size 8\
    --shots 16\
    --root_path 'your root path' \

Test Pretrained Model

You can use the test scripts test.sh to test the pretrained model. More Arguments can be referenced in parse_args.py

Main Results

MethodAcc-MindSporeAcc-PyTorchCheckpoint(PyTorch)Checkpoint(MindSpore)
MoCov3-ResNet50-16shot-lmageNet1k69.9870.02DownloadDownload
MoCov3-ResNet50-8shot-lmageNet1k68.2168.25DownloadDownload
MoCov3-ResNet50-4shot-lmageNet1k65.7965.92DownloadDownload
MoCov3-ResNet50-2shot-lmageNet1k64.1964.25DownloadDownload
MoCov3-ResNet50-1shot-lmageNet1k62.5762.60DownloadDownload

Acknowledgement

This repo benefits from Tip and CaFo. Thanks for their works.

Contact

If you have any questions or suggestions, please feel free to contact us: tangyuwei@tju.edu.cn and linzhenyi@tju.edu.cn.