How Would It Sound? Material-Controlled Multimodal Acoustic Profile Generation for Indoor Scenes

This repository contains the implementation and dataset for this paper, accepted in ICCV 2025

How Would It Sound? Material-Controlled Multimodal Acoustic Profile Generation for Indoor Scenes
Mahnoor Fatima Saad, Ziad Al-Halah
University of Utah

Project Page: https://mahnoor-fatima-saad.github.io/m-capa.html Paper: ICCV 2025

Abstract

How would the sound in a studio change with a carpeted floor and acoustic tiles on the walls? We introduce the task of material-controlled acoustic profile generation, where, given an indoor scene with specific audio-visual characteristics, the goal is to generate a target acoustic profile based on a user-defined material configuration at inference time. We address this task with a novel encoder-decoder approach that encodes the scene’s key properties from an audio-visual observation and generates the target Room Impulse Response (RIR) conditioned on the material specifications provided by the user. Our model enables the generation of diverse RIRs based on various material configurations defined dynamically at inference time. To support this task, we create a new benchmark, the Acoustic Wonderland Dataset, designed for developing and evaluating material-aware RIR prediction methods under diverse and challenging settings. Our results demonstrate that the proposed model effectively encodes material information and generates high-fidelity RIRs, outperforming several baselines and state-of-the-art methods.

Installation and Dependencies

This code and environment was created and tested with python 3.9.19 and pytorch 2.3.1 with cuda 12.1

To create the environment and install all dependencies run:

conda env create -f environment.yml
conda activate MASynth

Dataset

Download the Acoustic Wonderland Dataset using this link. The downloaded file should be a single .tar file of size 115GB. Untar this file. Within this directory there will be 84 individual tar files that will also need to be untared. Each directory corresponds to a scene in the Acoustic Wonderland Dataset and should have:

1. RGB: 200 .png files with the following naming convention: {scene_name}rgb{location_idx}.png
2. Category: 200 .npz segmentation maps corresponding to MP3D semantic segmentation with the following naming convention: {scene_name}category{location_idx}.npz
3. Material Config: This is folder containing 2,673 subfolder, each represending a material configuration from the Acoustic Wonderland dataset. Each folder contains:
   1. Material: .npz material segmentation map where each index corresponds to a material from the Acoustic Wonderland Dataset (more details about ids, names and acoustic coeffiecients can be found in dataset/material_configs). Each file follows: {scene_name}_{material_config}material{location_idx}.npz
   2. Waveform: .wav audio file with 0.5s at 16kHz sampling rate and corresponds to a matching material file. Each file follows: {scene_name}_{material_config}ir{location_idx}.wav

Store this dataset at any location and point to this location during training/testing Information about the testing splits, seen/unseen material configurations and acoustic properties can be found in the dataset folder.

Code

Batch scripts for training and testing are available in the /code/batch_scripts/ folder

Model checkpoints can be downloaded from this link..

Training

1. Please set all relevant information, paths and variables in the /code/config_training

sbatch run_training.sh

Testing

1. Download model checkpoints. Place model checkpoints in the root of y our parent directory (model checkpoints available upon request, will be adding links soon!)
2. For each variation of our model you will find best_checkpoint.pt, and config.json. Please update all paths and information inside each config to point to the relevant storage locations

sbatch run_evaluation.sh

Please reach out in case of any issues!

Citation

@InProceedings{Saad_2025_ICCV,
    author    = {Saad, Mahnoor Fatima and Al-Halah, Ziad},
    title     = {How Would It Sound? Material-Controlled Multimodal Acoustic Profile Generation for Indoor Scenes},
    booktitle = {Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)},
    month     = {October},
    year      = {2025},
    pages     = {12232-12241}
}