Omni2Sound — Your Multimodal Audio Generation Codebase 🎵

Official repository for "Omni2Sound: Towards Unified Video-Text-to-Audio Generation"

🏆 Accepted at CVPR 2026 (Highlight)

Omni2Sound is a unified framework for generating temporally aligned and semantically faithful audio from video, text, or both, handling three tasks — VT2A (video + text → audio), V2A (video → audio), and T2A (text → audio) — within a single model. Our design principle is to keep the model simple and push performance through data and training: instead of chasing bespoke architectures, Omni2Sound is built on a plain, off-the-shelf DiT backbone, and all of our gains come from a high-quality dataset (SoundAtlas) and a three-stage progressive multitask training schedule. With nothing fancy in the model, this deliberately minimal design still delivers state-of-the-art performance across all three tasks and remains robust under challenging scenarios such as off-screen audio synthesis and incomplete text inputs.

✨ Highlights

• 🏆 Omni2Sound — unified state-of-the-art across VT2A, V2A, and T2A on the VGGSound-Omni benchmark
• 📀 SoundAtlas — large-scale, high-quality A-V-T aligned audio captions, surpassing even human-expert annotation quality
• 🧪 VGGSound-Omni — unified VTA evaluation benchmark with robustness off-screen tracks
• 🔓 Open-source friendly — a multimodal extension of stable-audio-tools, with both inference and finetune code released

🎬 Out-of-Distribution demos

Omni2Sound generalizes to stylised / fictional videos it never saw during training while preserving tight audio-visual synchronization.

👉 For the full set of demos (VT2A, V2A, caption-quality comparisons), see the 🎬 project page.

🛠️ Quick Start

1. Environment

Tested on Python 3.10 + CUDA 12.1. Clone the repo, install the torch wheel matching your CUDA first, then install the remaining requirements:

git clone https://github.com/omni2sound/Omni2Sound.git
cd Omni2Sound

# 1) Install torch first (pick the CUDA wheel matching your driver)
pip install torch==2.1.0 torchaudio==2.1.0 torchvision==0.16.0 \
  --index-url https://download.pytorch.org/whl/cu121

# 2) Install the rest
pip install -r requirements.txt

2. Download model weights

Download three model folders into weights/:

One-liner with huggingface-cli:

huggingface-cli download Dalision/Omni2Sound        --local-dir weights/omni2sound
huggingface-cli download apple/DFN5B-CLIP-ViT-H-14-384 --local-dir weights/DFN5B-CLIP-ViT-H-14-384
huggingface-cli download google/flan-t5-base        --local-dir weights/flan-t5-base

Expected layout:

weights/
├── omni2sound/
│   ├── oob_vae_16k_224410.ckpt
│   ├── synchformer_state_dict.pth
│   └── vt2a-24-v55vt35-oa15-mq-td15/
│       ├── args.yaml
│       ├── data_config.yaml
│       ├── model_config.json
│       └── checkpoints/model.ckpt
├── DFN5B-CLIP-ViT-H-14-384/
└── flan-t5-base/

🎧 Inference

Omni2Sound accepts any of { video, text, video + text } and produces temporally and semantically aligned audio.

Online — from raw mp4

CLIP / Synchformer features are extracted on the fly. The input jsonl does not need any feature field.

bash scripts/infer_online.sh

Offline — with pre-extracted features

If you have pre-computed CLIP and Synchformer features per clip, use the faster offline path:

bash scripts/infer.sh

Both scripts default to the mock dataset under data/mock_dataset/. To run on your own data, edit dataset_config= at the top of the script and point it to your jsonl.

🧬 Finetune on your own data

Finetuning is organised in scripts/train.sh as two sequential stages:

bash scripts/train.sh 0   # Stage 0: joint VT2A + V2A + T2A finetuning
bash scripts/train.sh 1   # Stage 1: resume stage 0 with data augmentation
                          #          (off-screen synthesis + text dropout)

<conda_root>/envs/<your_env>/lib/python3.10/site-packages/pytorch_lightning/utilities/data.py

batch_sampler = getattr(dataloader, "batch_sampler")
batch_sampler_cls = type(batch_sampler)

# --- patch: skip Lightning's batch-sampler re-wrapping ---
return {
    "sampler": None,
    "shuffle": False,
    "batch_sampler": batch_sampler,
    "batch_size": 1,
    "drop_last": False,
}

if batch_sampler is not None and (batch_sampler_cls is not BatchSampler or is_predicting):

• Stage 0 — Multi-task Interleaved Finetuning. Jointly optimises VT2A, V2A, and T2A on paired (V, T, A) triplets plus T2A / V2A data, using a shared DiT backbone.
• Stage 1 — Decoupled Robustness Finetuning. Continues from the Stage-0 checkpoint with two push–pull synergistic augmentations: off-screen synthesis counteracts video bias, while text dropout counteracts text bias, keeping cross-modal reliance balanced under asymmetric input scenarios.

Batch feature extraction (CLIP and Synchformer) follows the pipeline of MMAudio — please refer to that repo for the extractor setup.

🧪 Benchmark

Two benchmarks are released at 🤗 Dalision/Omni2Sound_Benchmark:

• SoundAtlas — a large-scale A-V-T triple dataset augmenting AudioSet and VGGSound, produced by an agentic annotation pipeline that combines Vision-to-Language Compression (to mitigate visual hallucinations) with a Junior–Senior Agent Handoff (for 5× cost reduction), delivering captions that surpass even human-expert quality.

The SoundAtlas agentic annotation pipeline.

• VGGSound-Omni — unified VT2A / V2A / T2A evaluation, including off-screen audio and incomplete-text robustness tracks.

📝 Citation

If you find this work useful, please cite:

@article{dai2026omni2sound,
  title   = {Omni2Sound: Towards Unified Video-Text-to-Audio Generation},
  author  = {Dai, Yusheng and Chen, Zehua and Jiang, Yuxuan and Gao, Baolong and
             Ke, Qiuhong, Cai, Jianfei and Zhu, Jun},
  journal = {arXiv preprint arXiv:2601.02731},
  year    = {2026}
}

📧 Contact

If you have any comments or questions, feel free to contact: yusheng.dai@monash.edu

🙏 Acknowledgements

• stable-audio-tools — codebase, DiT backbone and OOB VAE
• FreeAudio — T2A pretrain + Wav VAE
• MMAudio — feature extraction pipeline
• Synchformer — synchronisation features
• AudioX — prior work on unified multimodal audio generation

📄 License

Both the code and the model weights are released under CC BY-NC 4.0 (non-commercial use only).