Save single prediction file

May 12, 2026 Β· View on GitHub

MECAT: A Multi-Experts Constructed Benchmark for Fine-Grained Audio Understanding Tasks

πŸ“– Paper | 🎧 Demo | πŸ”Š MECAT-Caption (HF) | πŸ”Š MECAT-QA (HF)

MECAT Logo

Table of Contents

1. Introduction

MECAT is a comprehensive benchmark constructed on large-scale data to evaluate machine understanding of audio content through two core tasks:

  • Audio Captioning: Generating textual descriptions for given audio
  • Audio Question Answering: Answering questions about given audio

image

2. Features

  • Data Source：Diverse-scenario coverage via the part of ACAV100M dataset
  • Processing Pipeline:
    • MetaInfo: Source video metadata extraction (titles/descriptions)
    • Content-Specific: Content-specific feature extraction using 10-20 dedicated models (speech/music/general audio)
    • Content-Unrelated: Non-content audio analysis: quality metrics, loudness measurements, reverberation assessment
  • Understanding & Genration: LLM-powered comprehension & generation with Chain-of-Thought
  • Quality Control： Multi-stage verification framework
  • Evluation System: Multi-perspective assessment with progressive difficulty levels

3. Data Distribution

Data Code Description Audio Caption Audio Question Answering
# Pairs (Train) # Pairs (Test) # Pairs (Train) # Pairs (Test)
000 silence 173 179 865 895
00A general sound excluding speech and music 837 848 4185 4240
0M0 music 2593 2593 12965 12965
0MA music and general sound 206 199 1030 995
S00 speech 7839 7839 39195 39195
S0A speech and general sound 2424 2439 12120 12195
SM0 speech and music 5312 5312 26560 26560
SMA speech, music and general sound 668 643 3340 3215

4. Tasks

4.1 Audio-Captioning

Type Subtask Category Level Descrption Evaluated Data Abbreviation Sample
Systemtic Short πŸ”΅ Specialized Simplified caption over the whole audio within 15 words 000, 00A, 0M0, 0MA
S00, S0A, SM0, SMA		 20052
Long πŸ”΅ Specialized Caption over the whole audio using 1-2 sentences 000, 00A, 0M0, 0MA
S00, S0A, SM0, SMA		 20052
Content-Specific Speech Clean 🟒 Basic Caption over clean speech   S00 7839
Mixed πŸ”΄ Complex Caption over speech with music/sound interference 0MA, S0A, SM0, SMA 8593
Music Clean 🟒 Basic Caption over clean Music      0M0 2593
Mixed πŸ”΄ Complex Caption over music with speech/sound interference 0MA, S0A, SM0, SMA 8593
Sound Clear 🟒 Basic Caption over general sound excluding speech and music 00A 848
Mixed πŸ”΄ Complex Caption over sound with speech/music interference 0MA, S0A, SM0, SMA 8593
Content-Unrelated Environment πŸ”΅ Specialized Caption over acoustic characteristic and environment 000, 00A, 0M0, 0MA
S00, S0A, SM0, SMA		 20052

4.2 Audio-Question-Answering

Description

Type Subtask Level Description Data Abbreviation Sample
Perception Direct_Perception 🟒🟑 Perceive sound types 000, 00A, 0M0, 0MA, S00, S0A, SM0, SMA 20624
Analysis Sound_Characteristics πŸŸ’πŸŸ‘πŸŸ πŸ”΄ Analyze sound characteristics 000, 00A, 0M0, 0MA, S00, S0A, SM0, SMA 19767
Quality_Assessment πŸŸ’πŸŸ‘πŸŸ πŸ”΄ Analyze sound quality 000, 00A, 0M0, 0MA, S00, S0A, SM0, SMA 18942
Reasoning Environment_Reasoning πŸŸ’πŸŸ‘πŸŸ πŸ”΄ Reasoning acoustic environment 000, 00A, 0M0, 0MA, S00, S0A, SM0, SMA 18300
Inference_Judgment πŸŸ’πŸŸ‘πŸŸ πŸ”΄ Cross-modal reasoning 000, 00A, 0M0, 0MA, S00, S0A, SM0, SMA 19756
Application_Context πŸŸ’πŸŸ‘πŸŸ πŸ”΄ Semantic understanding 000, 00A, 0M0, 0MA, S00, S0A, SM0, SMA 2871

Difficulty Distribution

Difficulty Symbol Ratio (%) Description
Basic 🟒 25 Direct descriptive questions
Intermediate 🟑 35 Analytical questions   
Advanced 🟠 25 Inferential questions  
Complex πŸ”΄ 15 Comprehensive judgment questions

5. Example Data

The following examples are from the SMA (Speech, Music and General Sound) domain. You can listen to the audio and browse more samples across all 8 domains on the Demo Page.

5.1 Audio Captioning Example

Each audio clip has 6 caption types (short, long, speech, music, sound, environment), each with 3 paraphrased variants:

{
  "tFg4IwL09Rk_111_64000000000001_121_64": {
    "short": [
      "Electronic music with water splashing and brief speech amid static interference",
      "Dark electronic instrumental with aquatic sounds and momentary vocal utterance",
      "Water splashing noises under brooding synth music featuring short spoken phrase"
    ],
    "long": [
      "A dark electronic instrumental track with water splashing sounds and a male voice briefly speaking, accompanied by persistent background static and mid-frequency noise artifacts.",
      "Moody electronic composition blending synthesized tones with environmental water sounds and fragmented speech, degraded by audio interference",
      "Sustained static underlies a brief male utterance and aquatic noises within a dramatic electronic musical arrangement"
    ],
    "speech": [
      "Male voice briefly uttering 'We over' with cheerful vocal characteristics",
      "Short spoken phrase 'We over' delivered in an upbeat tone",
      "Fragmentary speech segment showing positive vocal affect"
    ],
    "music": [
      "Dark electronic music featuring synthesizer and guitar elements with moderate tempo",
      "Brooding instrumental track combining synthetic textures and guitar tones",
      "Dramatic electronic composition with industrial-inspired rhythmic patterns"
    ],
    "sound": [
      "Prominent water splashing sounds with intermittent static bursts",
      "Aquatic environment noises dominated by liquid movement",
      "Liquid splashing effects accompanied by electrical interference"
    ],
    "environment": [
      "Low-quality audio environment with background static and mid-range distortion",
      "Noisy recording space exhibiting persistent electrical interference",
      "Degraded acoustic conditions with broadband static artifacts"
    ],
    "domain": "SMA"
  }
}

5.2 Audio Question Answering Example

Each audio clip has 5 QA pairs spanning different cognitive categories and difficulty levels:

{
  "tFg4IwL09Rk_..._b4dfbcf5": {
    "category": "direct_perception",
    "difficulty": "basic",
    "question": "Is speech present in the audio?",
    "answer": "Yes, a male voice briefly says 'We over'",
    "domain": "SMA"
  }
}

6. Evaluation Metrics

MECAT supports multiple evaluation metrics for comprehensive assessment:

  • Traditional Metrics: BLEU
  • FENSE: Fluency Error-based Sentence-bert Evaluation for audio captioning
  • DATE: Discriminability based Audio Task Evaluation - DATE is particularly effective for audio captioning and question-answering tasks as it considers both the quality of generated text and the model's discriminative capabilities.

7. Usage

7.1 Installation

python3 -m pip install mecat
# Or the development 
# pip install git+https://github.com/xiaomi-research/mecat.git

7.2 Quick Start with Qwen2-Audio Example

This section provides a complete walkthrough of evaluating audio models using MECAT, using Qwen2-Audio as a practical example. The same approach can be adapted for other audio understanding models.

7.2.1 Preliminary Steps: Environment Setup and Model Loading

import torch
from tqdm import tqdm
from transformers import AutoProcessor, Qwen2AudioForConditionalGeneration

# Device setup
device = "cuda" if torch.cuda.is_available() else "cpu"
print(f"Using device: {device}")

# Load Qwen2-Audio model and processor
model = Qwen2AudioForConditionalGeneration.from_pretrained(
    "Qwen/Qwen2-Audio-7B", 
    trust_remote_code=True,
    device_map="auto"
)
processor = AutoProcessor.from_pretrained(
    "Qwen/Qwen2-Audio-7B", 
    trust_remote_code=True
)

7.2.2 Audio Caption Evaluation

Step 1: Load MECAT-Caption Dataset
from datasets import load_dataset
data = load_dataset(
    'mispeech/MECAT-Caption', 
    split='test', 
)
print(f"Loaded {len(data)} samples from datasets")
Step 2: Generate and Evaluate Captions

Method 1: Single Dictionary Approach (for non-instruction-following models)

Generation:

from mecat import evaluate
# Generate general predictions using a single prompt
predictions = {}

for item in tqdm(data, desc="Generating general captions"):
    key = item['__key__']
    audio = item['flac']['array']
    sampling_rate = item['flac']['sampling_rate']
    # Note: the sampling rate of audio provided by MECAT is 16kHz 
    
    # Create general prompt for caption generation
    prompt = "<|audio_bos|><|AUDIO|><|audio_eos|>Generate the caption in English:"

    # Process inputs
    inputs = processor(
        text=prompt, 
        audio=audio, 
        sampling_rate=sampling_rate,
        return_tensors="pt"
    ).to(device)
    
    # Generate response
    with torch.no_grad():
        generated_ids = model.generate(
            **inputs, 
            max_length=512,
            do_sample=False,
            temperature=0.1
        )
    
    # Decode response
    generated_ids = generated_ids[:, inputs.input_ids.size(1):]
    response = processor.batch_decode(
        generated_ids, 
        skip_special_tokens=True, 
        clean_up_tokenization_spaces=False
    )[0]
    predictions[key] = response.strip()

print(f"Generated {len(predictions)} general captions")

# Save single prediction file
import csv
with open('caption_predictions.csv', 'w', encoding='utf-8') as f:
    writer = csv.writer(f, quoting=csv.QUOTE_ALL)
    for key, value in predictions.items():
        writer.writerow([key, value])

Evaluation:

# Evaluate general predictions across all subtasks
results = evaluate(
    predicted_data=predictions,
    task='caption', 
    metrics=['fense', 'date']
)

print("\nSingle Dictionary Evaluation Results:")
print(results)

Method 2: Multi-Dictionary Approach (recommended for instruction-following models)

Generation:

# Generate task-specific predictions using different prompts
task_prompts = {
    'long': "<|audio_bos|><|AUDIO|><|audio_eos|>Listen to this audio and describe it in 1-2 sentences:",
    'short': "<|audio_bos|><|AUDIO|><|audio_eos|>Listen to the audio and provide a caption for this audio within 15 words:",
    'speech': "<|audio_bos|><|AUDIO|><|audio_eos|>Listen to the audio and provide a caption describing the speech content in this audio:",
    'music': "<|audio_bos|><|AUDIO|><|audio_eos|>Listen to the audio and provide a caption for the music content in this audio:",
    'sound': "<|audio_bos|><|AUDIO|><|audio_eos|>Listen to the audio and provide a general sound excluding speech and music:",
    'environment': "<|audio_bos|><|AUDIO|><|audio_eos|>Listen to the audio and provide a caption for quality or acoustic environment for this audio:"
}

# Generate predictions for each subtask
subtask_predictions = {}

for subtask, prompt_template in task_prompts.items():
    print(f"\nGenerating {subtask} captions...")
    subtask_predictions[subtask] = {}
    
    for item in tqdm(data, desc=f"Generating {subtask} captions"):
        key = item['__key__']
        audio = item['flac']['array']
        sampling_rate = item['flac']['sampling_rate']
        
        # Process inputs with task-specific prompt
        inputs = processor(
            text=prompt_template, 
            audio=audio, 
            sampling_rate=sampling_rate,
            return_tensors="pt"
        ).to(device)
        
        # Generate response
        with torch.no_grad():
            generated_ids = model.generate(
                **inputs, 
                max_length=512,
                do_sample=False,
                temperature=0.1
            )
        
        # Decode response
        generated_ids = generated_ids[:, inputs.input_ids.size(1):]
        response = processor.batch_decode(
            generated_ids, 
            skip_special_tokens=True, 
            clean_up_tokenization_spaces=False
        )[0]
        subtask_predictions[subtask][key] = response.strip()

# Save separate prediction files for each subtask
for subtask, preds in subtask_predictions.items():
    filename = f'{subtask}_caption.csv'
    with open(filename, 'w', encoding='utf-8') as f:
        writer = csv.writer(f, quoting=csv.QUOTE_ALL)
        for key, value in preds.items():
            writer.writerow([key, value])
    print(f"Saved {len(preds)} {subtask} predictions to {filename}")

Evaluation:

# Evaluate task-specific predictions for optimal performance
results_multisubtask = evaluate(
    predicted_data=subtask_predictions,
    task='caption', 
    metrics=['fense', 'date']
)

print("\nMulti-Dictionary Evaluation Results:")
print(results_multisubtask)
Step 3: Expected Results

Expected Caption Evaluation Output: This result does not represent the actual performance of Qwen2-Audio-7B

   subtask     num_samples  fense  date
 content_long        20052   47.3 40.5
content_short        20052   45.8 41.0 
  pure_speech         7839   30.9 28.5 
 mixed_speech         8593   31.7 27.1 
   pure_music         2593   42.1 50.7
  mixed_music         8593   28.3 33.1
   pure_sound          848   41.2 46.6 
  mixed_sound         8593   16.2 34.1 
  environment        20052   45.4 47.8
score_caption         <NA>   35.2 39.3

Note: The formulae of score_caption:

Scaption=0.4Γ—(0.8Slong+0.2Sshort)+0.4Γ—(0.6Sspeech+0.3Smusic+0.1Ssound)+0.2Γ—SenvironmentS_{\rm caption} = 0.4\times({0.8S_{\rm long} + 0.2S_{\rm short}}) + 0.4\times(0.6S_{\rm speech} + 0.3S_{\rm music} + 0.1S_{\rm sound}) + 0.2\times S_{\rm environment}

where Sspeech,SmusicS_{\rm speech}, S_{\rm music} and SsoundS_{\rm sound} were the average score of pure data and mixed data, e.g., Sspeech=Sspeech,pure+Sspeech,mixed2S_{\rm speech} = \frac{S_{\rm speech,pure}+S_{\rm speech,mixed}}{2}

7.2.3 Audio Question Answering Evaluation

Step 1: Load MECAT-QA Dataset
# Load MECAT-QA test data 
qa_data = load_dataset(
    'mispeech/MECAT-QA', 
    split='test', 
)
print(f"Loaded {len(qa_data)} QA samples from datasets")
Step 2: Generate and Evaluate Answers

Generation:

# Generate predictions for each question-audio pair
qa_predictions = {}

for item in tqdm(qa_data, desc="Generating answers"):
    key = item['__key__']
    audio = item['flac']['array']
    sampling_rate = item['flac']['sampling_rate']
    question = item['json']['question']
    
    
    # Create prompt for QA
    prompt = f"<|audio_bos|><|AUDIO|><|audio_eos|>{question}"
    
    # Process inputs
    inputs = processor(
        text=prompt, 
        audio=audio, 
        sampling_rate=sampling_rate,
        return_tensors="pt"
    ).to(device)
    
    # Generate response
    with torch.no_grad():
        generated_ids = model.generate(
            **inputs, 
            max_length=512,
            do_sample=False,
            temperature=0.1
        )
    
    # Decode response
    generated_ids = generated_ids[:, inputs.input_ids.size(1):]
    response = processor.batch_decode(
        generated_ids, 
        skip_special_tokens=True, 
        clean_up_tokenization_spaces=False
    )[0]
    
    qa_predictions[key] = response.strip()

print(f"Generated {len(qa_predictions)} answers")

# Output the results to csv files
import csv
with open('qa_predictions.csv', 'w', encoding='utf-8') as f:
    writer = csv.writer(f, quoting=csv.QUOTE_ALL)
    for key, value in qa_predictions.items():
        writer.writerow([key, value])

Evaluation:

# Evaluate using MECAT metrics
qa_results = evaluate(
    predicted_data=qa_predictions, 
    task='qa', 
    metrics=['fense', 'date']
)

print("\nQA Evaluation Results:")
print(qa_results)
Step 3: Expected Results

Expected QA Evaluation Output: This result does not represent the actual performance of Qwen2-Audio-7B

           subtask     num_samples fense date
    direct_perception        20624 44.0 54.0
sound_characteristics        19767 39.0 53.1 
   quality_assessment        18942 18.0 17.8
environment_reasoning        18300 42.0 35.5 
  inference_judgement        19756 51.0 42.0
  application_context         2871 40.0 49.9
             score_qa        <NA>  39.0 42.1

Note: the final score is the average scores of all six subtasks

7.3 Command Line Evaluation

You can also use the command line interface for evaluation:

7.3.1 Single File Evaluation

# Caption evaluation for different audio types (using single dictionary predictions)
python -m mecat.evaluate --prediction caption_predictions.csv --task caption --subtask long   --metrics fense date
python -m mecat.evaluate --prediction caption_predictions.csv --task caption --subtask short  --metrics fense date
python -m mecat.evaluate --prediction caption_predictions.csv --task caption --subtask music  --metrics fense date
python -m mecat.evaluate --prediction caption_predictions.csv --task caption --subtask speech --metrics fense date
python -m mecat.evaluate --prediction caption_predictions.csv --task caption --subtask sound  --metrics fense date
python -m mecat.evaluate --prediction caption_predictions.csv --task caption --subtask environment --metrics fense date

# Batch evaluation across all subsets (using single dictionary predictions)
python -m mecat.evaluate --prediction caption_predictions.csv --task caption --metrics fense date

For instruction-following models that can generate task-specific captions, you can provide multiple prediction files at once to get comprehensive evaluation results across all caption subtasks:

# Evaluate multiple caption prediction files in order: long, short, speech, music, sound, environment
python -m mecat.evaluate --prediction \
    long_caption.csv \
    short_caption.csv \
    speech_caption.csv \
    music_caption.csv \
    sound_caption.csv \
    environment_caption.csv \
    --task caption --metrics fense date

# Evaluate with fewer files (will evaluate only available subtasks with warning)
python -m mecat.evaluate --prediction \
    long_caption.csv \
    short_caption.csv \
    --task caption --metrics fense date

Benefits of Multi-File Evaluation:

  • βœ… Complete Coverage: Evaluates all caption subtasks with task-specific predictions
  • βœ… Better Performance: Each prediction file contains responses optimized for specific caption types
  • βœ… Comprehensive Results: Provides the full evaluation matrix including overall scores
  • ⚠️ File Order Matters: Files are mapped to subtasks in order: long β†’ short β†’ speech β†’ music β†’ sound β†’ environment

7.3.3 QA Task Evaluation

# QA evaluation for different question types
python -m mecat.evaluate --prediction qa_predictions.csv --task qa --subtask direct_perception     --metrics fense date
python -m mecat.evaluate --prediction qa_predictions.csv --task qa --subtask sound_characteristics --metrics fense date
python -m mecat.evaluate --prediction qa_predictions.csv --task qa --subtask quality_assessment    --metrics fense date
python -m mecat.evaluate --prediction qa_predictions.csv --task qa --subtask environment_reasoning --metrics fense date
python -m mecat.evaluate --prediction qa_predictions.csv --task qa --subtask inference_judgement   --metrics fense date
python -m mecat.evaluate --prediction qa_predictions.csv --task qa --subtask application_context   --metrics fense date

# Batch evaluation across all subsets (recommended)
python -m mecat.evaluate --prediction qa_predictions.csv --task qa --metrics fense date

Prediction File Format:

# csv File
"audio_key_1", "Generated caption or answer text"
"audio_key_2", "Another generated response"
"audio_key_3", "More predictions..."

Important Notes:

  • Audio Captioning Task
    • For instruction-following models (Recommended):
      • Generate 6 different prediction files using task-specific prompts (one per sub-task). Requires 6 inference passes.
      • Prompts example:
        • long: "Listen to this audio and describe it in 1-2 sentences"
        • short: "Listen to the audio and provide a caption for this audio within 15 words"
        • speech: "Listen to the audio and provide a caption describing the speech content in this audio"
        • music: "Listen to the audio and provide a caption for the music content in this audio"
        • sound: "Listen to the audio and provide a general sound excluding speech and music"
        • environment: "Listen to the audio and provide a caption for quality or acoustic environment for this audio"
    • For non-instruction-following models:
      • Evaluate using a single prediction file (single inference pass).
      • The same predictions will be evaluated across all subtasks.
  • Audio Question Answering Task:
    • Evaluate all sub-tasks in a single inference pass using the standard method.
    • Single prediction file is sufficient as questions are task-specific.

7.4 Direct Data Evaluation

If you have complete predicted_data and reference_data, you can directly use them for evaluation without loading from files or datasets:

Audio Captioning Task Example:

from mecat import evaluate

predicted_data = [['silence']]

reference_data = [['Extended silence with severe audio distortion and background noise.', 'Persistent quiet period containing heavy signal interference.', 'Continuous silence disrupted by pronounced technical artifacts.']]

results = evaluate(
    predicted_data=predicted_data, 
    reference_data=reference_data, 
    task='caption',
    metrics='bleu',
)

print(results)

Audio Question Answering Task Example:

Similarly, for QA tasks, you can also provide predicted_data and reference_data directly:

from mecat import evaluate

predicted_data = [['A woman speaking expressively and dog barks.']]

reference_data = [['A woman speaking expressively and dog barks.']]

results = evaluate(
    predicted_data=predicted_data, 
    reference_data=reference_data, 
    task='qa',
    metrics='bleu',
    subtask='direct_perception',
)

print(results)

This approach is useful when:

  • You already have the predictions and references in memory
  • You want to evaluate custom data that is not part of the MECAT dataset
  • You need to quickly test evaluation metrics on specific examples

Note: Both predicted_data and reference_data should be lists of lists, where each inner list contains the predictions or references for a single sample. For reference data, multiple references per sample are supported (as shown in the caption example above).

Important: If you want to obtain results for different tasks (e.g., evaluating multiple caption subtasks or QA subtasks), it is recommended to use the methods described in sections 7.1-7.3, as they automatically select the appropriate keys for different tasks and provide comprehensive evaluation results across all subtasks.

8. Results

8.1 Audio-Captioning Task (DATE %)

Bold indicates the best performance. Models above the line are open-weight; below are proprietary/API-based.

TypeModel SystemicContent-Specific Content
Unrelated		Score
SpeechMusicSound
LongShortPureMixedPureMixedPureMixedEnv
Caption
-Only		Pengi43.546.827.229.529.313.142.814.67.129.4
EnClap48.653.130.231.817.915.948.815.26.831.9
LALMPhi-4-Multimodal42.444.026.931.314.924.028.518.113.130.0
Kimi-Audio-7B49.554.230.031.327.716.943.116.27.032.8
Baichuan-Audio42.636.546.040.421.320.744.817.715.133.7
Baichuan-Omni47.050.943.541.735.213.734.319.711.335.6
Mimo-Audio56.556.945.844.935.819.446.821.09.840.1
Audio Flamingo 352.551.549.348.840.424.850.621.911.540.4
Qwen3-Omni47.943.750.248.751.226.849.019.518.240.4
Step-Audio-2-mini55.658.744.243.635.332.042.818.916.141.5
Qwen2.5-Omni 3B56.455.242.541.346.629.752.923.919.442.5
Qwen2.5-Omni 7B61.156.539.940.932.130.950.723.817.942.6
Qwen3-Omni-Flash-120165.762.559.259.957.432.555.831.627.152.9
Gemini-2.5-Flash65.663.957.557.552.941.052.228.322.151.6
Gemini-2.5-Pro62.362.456.657.553.638.753.429.924.050.6
Gemini-3-Flash63.661.959.460.843.132.951.129.725.751.1
Gemini-3-Pro64.965.860.562.449.839.855.129.926.153.1

8.2 Audio-Question-Answering (DATE %)

Model PerceptionAnalysisReasoning Score
Direct
Perception		Sound
Characteristics		Quality
Assessment		 Environment
Reasoning		Inference &
Judgment		Application
Context
Kimi-Audio-7B45.639.218.734.648.941.238.0
Baichuan-Audio40.745.231.035.149.046.941.3
Baichuan-Omni43.644.733.739.949.349.143.4
Phi-4-Multimodal48.446.334.740.249.348.744.6
Mimo-Audio59.349.324.939.152.746.245.2
Step-Audio-2-mini57.754.337.239.248.948.047.6
Audio Flamingo 353.850.236.043.054.549.647.8
Qwen2.5-Omni 3B55.753.238.641.151.850.848.5
Qwen2.5-Omni 7B57.852.939.144.053.250.849.6
Qwen3-Omni61.754.639.345.056.956.152.3
Qwen3-Omni-Flash-120148.045.929.545.656.754.846.7
Gemini-2.5-Flash56.355.337.746.858.658.052.1
Gemini-2.5-Pro55.554.437.747.657.356.651.5
Gemini-3-Flash54.351.134.147.257.257.051.0
Gemini-3-Pro55.545.525.844.053.252.046.0

9. Acknowledgement

We have referred to the implementation of FENSE for the evaluation

10. Contributing

Yadong Niu* Β· Tianzi Wang* Β· Heinrich Dinkel Β· Xingwei Sun Β· Jiahao Zhou Β· Gang Li Β· Jiahao Mei Β· Jizhong Liu Β· Xunying Liu Β· Junbo Zhang Β· Jian Luan

*: Equal Contribution

11. Citation

@article{mecat2025,
  title={MECAT: A Multi-Experts Constructed Benchmark for Fine-Grained Audio Understanding Tasks},
  author={Niu, Yadong and Wang, Tianzi and Dinkel, Heinrich and Sun, Xingwei and Zhou, Jiahao and Li, Gang and Liu, Jizhong and Liu, Xunying and Zhang, Junbo and Luan, Jian},
  journal={arXiv preprint arXiv:2507.23511},
  year={2025}
}

12. License

The dataset of the project is from the part of ACAV100M undert the Creative Commons Attribution License 3.0 (CC BY-3.0) license.

The code of the project is under Apache License 2.0 license.