VIP-ANT: VIsually-Pivoted Audio and(N) Text
May 16, 2022 ยท View on GitHub
Code for the paper Connecting the Dots between Audio and Text without Parallel Data through Visual Knowledge Transfer @ NAACL 2022.
Data
AudioSet can be downloaded and preprocessed via this tool.
AudioSet Data
See AudioSet. It elaborates on our customized index files for pre-training on AudioSet.
Curated Audio-Text Data
See AudioTxt. It elaborates on our curation methods and customized index files for audio-text fine-tuning.
Vision-Audio (VA) Pre-training
Check out the running script bash/run_bimodal_va.sh.
Audio-Text (AT) Fine-tuning
Check out the running script bash/run_bimodal_at.sh. Fine-tuning starts with a VA pre-trained audio encoder.
Pre-trained Models
We provide a checkpoint that performs best for each task.
Pre-trained Models for Audio-Text Retrieval
| Model | AudioCaps | Clotho (18s) | Clotho (10s) |
|---|---|---|---|
| VIP-ANT | 00051623 | 00043681 | 00043681 |
| +AT w/ GC | 00006210 | 00006900 | 00004140 |
Pre-trained Models for Zero-shot Audio Classification
| Model | ESC50 (w/ prompt) | US8K (w/ prompt) |
|---|---|---|
| VIP-ANT | 00083391 | 00079420 |
| +AT w/ GC | 00004140 | 00004140 |
Dependencies
Dockerfile defines minimum dependencies of the repo.
License
MIT