AVDD

April 14, 2025 · View on GitHub

This repository includes code for : Audio Visual Dataset Distillation (TMLR 2024).

Create env

conda create -n avdd python=3.9 -y
conda activate avdd
pip install torch==2.1.1 torchvision==0.16.1 torchaudio==2.1.1 --index-url https://download.pytorch.org/whl/cu118

git clone https://github.com/sakshamsingh1/AVDD
cd AVDD
pip install -r requirements.txt

Inference code

Download and unzip VGG-10k distilled and real data.

bash scripts/download_vgg_subset.sh

Evaluate VGG-10k data. The code will evaluate 3 distilled data for 5 times.

bash scripts/vgg10k_evaluate.sh

Note: Please uncomment different commands in the file to test different IPC settings
Similar scripts are present for other datasets.

Visualization
You can visualize the distilled data using the visualize_data.ipynb notebook.

More
  • Synthetic data can be learned independently for each class.
  • Training all classes together with high class counts and images-per-class (IPC) may lead to out-of-memory (OOM) issues.
  • This approach allows parallel training for each class.

Download classwise training data

# Open the script and uncomment the datasets you wish to download 
bash scripts/download_classwise_train_data.sh

Alternatively, you can generate class-wise data from the full training set using: python preprocess/create_classwise_data.py

Pre-saving real mean embeddings

# Basically we can pre-save real mean embeddings with ipc=1
bash scripts/create_classwise_buffer.sh

Alternatively, you can download pre-extracted embeddings from huggingface

Learn synthetic data classwise

Now you can learn the synthetic data classwise with different IPCs

bash scripts/train_classwise.sh

Evaluate classwise trained data

bash scripts/classwise_evaluate.sh

Training

Prepare data

Download preprocessed training data from huggingface

# Open the script and uncomment the datasets you wish to download
bash scripts/download_train_data.sh
Preparing preprocessed training/testing data (For eg. AVE)
  • Download AVE dataset
  • Extract audio/frames preprocess/extract_audio_and_frames.py
  • Create training/testing dataset (.pt) file preprocess/AVE_input_data.py ( for VGG-subset see: preprocess/VGG_subset_input_data.py)
  • These scripts can be modified to support other datasets.
  • We also provide metadata in preprocess/meta_data .

Herding

We provide the precomputed herding indices in data/herding_data

More
  • The synthetic data is initialised with herding selected method.
  • To compute herding data, we follow the pseudocode mentioned here .

Launch training

Download preprocessed training data from huggingface

# Please open the script and refer to some examples here.
bash scripts/train_avdd.sh

TODO

  • Inference code
  • Training code
  • Parallelized training code
  • Retrieval code

🤗 Citation

@article{kushwahaaudio,
  title={Audio-Visual Dataset Distillation},
  author={Kushwaha, Saksham Singh and Vasireddy, Siva Sai Nagender and Wang, Kai and Tian, Yapeng},
  journal={Transactions on Machine Learning Research}
}

The code is based on Distribution Matching, AV-robustness