Look, Listen, and Answer: Overcoming Biases for Audio-Visual Question Answering

This repository contains the MUSIC-AVQA-R dataset , as proposed in Look, Listen, and Answer: Overcoming Biases for Audio-Visual Question Answering, as well as the code implementation of the Multifaceted Cycle Collaborative Debiasing (MCCD) method.

Figure 1: Robust AVQA architecture to overcome bias learning. Our MCCD strategy is plug-and-play,allowing seamless integration with other AVQA methods.

Dataset

We introduce the first dataset, MUSIC-AVQA-R, to evaluate the robustness of AVQA models. The construction of this dataset involves two key processes: rephrasing and splitting. The former involves the rephrasing of questions in the test split of MUSIC-AVQA, and the latter is dedicated to the categorization of questions into frequent (head) and rare (tail) subset.

We followed the previous work in partitioning the dataset into "head" and "tail" categories. Based on the number of answers in the dataset, answers with a count greater than $1.2$times the mean, denoted as$\mu(a), were categorized as "head" while those with counts less than \1.2\mu(a)$ were categorized as "tail"

The sizes of our MUSIC-AVQA-R dataset are presented in Table 1 and Table 2. We present it based on different question scenarios and question types.

Table 1. Test split comparison between MUSIC-AVQA and MUSIC-AVQA-R. EXIST, LOC, CNT, COMP, and TEMP, which are question types, denote “Existential”, “Location”, “Counting”, “Comparative”, and “Temporal”, respectively.

Table 2. The sizes of head and tail subsets for different question types in MUSIC-AVQA-R, which are divided based on their corresponding μ(a).

• example

  

Figure 2: Answer distributions of the "location" type of questions in the AVQA task. μ(a) is the average number of answers in the group

In this scenario, answers such as "yes," "no," "left," and others, whose total count exceeds $1.2\mu(a)$, are divided as "head." The remaining answers such as "accordion", "tuba" and so forth, are divided as "tail."

Training Set and Validation Set

Download here.

The dataset modifications exclusively pertain to the test split of MUSIC-AVQA, thus leaving the training set and validation set unchanged.

Test Set

Download here.

We utilized the QuillBot to rephrase each question in the MUSIC-AVQA dataset 25 times. Subsequently, three annotators independently voted on whether the rephrased questions retained semantic equivalence with the original questions. Questions receiving two or more votes were retained. The Fleiss Kappa value, indicating the consistency of this voting process, was 0.849. Ultimately, 92.4% of the questions were preserved. In MUSIC-AVQA-R's test set, compared to MUSIC-AVQA, the number of questions expanded from 9,129 to 211,572, and the vocabulary of questions increased from 93 to 465.

Figure 2 displays the first three words in MUSIC-AVQA-R and MUSIC-AVQA. Intuitively, we observe that questions in MUSIC-AVQA-R are more diverse than those in MUSIC-AVQA in terms of their expression.

Figure 3: Distribution visualization of questions based on the first three words.

• Demo

  

Figure 4. Demo in MUISC-AVQA-R

In the test set demo, we include audio, video, and corresponding questions. In this example, the question template is "Is the <Object> in the video always playing?" After rephrasing with the tool, the question becomes "Does the cello in the video play all the time," which maintains the same meaning as the original question but with a different expression.

Usage

Getting Started

$ conda create --name <env> --file requirements.txt

Download Features of Videos and Audios

Download here.

We directly utilized the audio and video features provided in MUSIC-AVQA. We extend our gratitude to the authors for open-sourcing them!

Training

1. set hyper parameters in option.yaml, configure the run_mode parameter as "train", and specify the parameters under the path section.

2.  python main.py
   

Evaluation

1. set hyper parameters in option.yaml, configure the run_mode parameter as "test", and specify the parameters under the path section.

2. python main.py
   

Case Study

Figure 5. Attention weight visualization on the uniformly sampled audio and video frames. We can see that our method can focus on the key audio and video frames for QA simultaneously in both in- and out-of-distribution settings

Citation

If you find our work useful in your research, please consider citing:

@article{ma2024look,
  title={Look, Listen, and Answer: Overcoming Biases for Audio-Visual Question Answering},
  author={Ma, Jie and Hu, Min and Wang, Pinghui and Sun, Wangchun and Song, Lingyun and Pei, Hongbin and Liu, Jun and Du, Youtian},
  journal={arXiv preprint arXiv:2404.12020},
  year={2024}
}

Acknowledgement

The dataset presented in this paper is built upon the MUSIC-AVQA. We extend our gratitude to the authors for open-sourcing their work.