MTRAG: Multi-Turn RAG Benchmark

July 14, 2026 ยท View on GitHub

Papers | Corpora | MTRAG - Human Data | MTRAG - Synthetic Data | MTRAG-UN | MTRAGEval | Getting Started | Contact

We present MTRAG, a comprehensive and diverse human-generated multi-turn RAG dataset, accompanied by four document corpora. To the best of our knowledge, MTRAG is the first end-to-end human-generated multi-turn RAG benchmark that reflects real-world properties of multi-turn conversations.

Papers

The papers describing the benchmarks and experiments are available on Arxiv:

MTRAG: A Multi-Turn Conversational Benchmark for Evaluating Retrieval-Augmented Generation Systems
Yannis Katsis, Sara Rosenthal, Kshitij Fadnis, Chulaka Gunasekara, Young-Suk Lee, Lucian Popa, Vraj Shah, Huaiyu Zhu, Danish Contractor, Marina Danilevsky
Transactions of the Association for Computational Linguistics, 2025

MTRAG-UN: A Benchmark for Open Challenges in Multi-Turn RAG Conversations
Sara Rosenthal, Yannis Katsis, Vraj Shah, Lihong He, Lucian Popa, Marina Danilevsky ACL Findings, 2026

If you use MTRAG, please cite the paper as follows:

@article{katsis2025mtrag,
      title={MTRAG: A Multi-Turn Conversational Benchmark for Evaluating Retrieval-Augmented Generation Systems},
      author={Yannis Katsis and Sara Rosenthal and Kshitij Fadnis and Chulaka Gunasekara and Young-Suk Lee and Lucian Popa and Vraj Shah and Huaiyu Zhu and Danish Contractor and Marina Danilevsky},
      journal={Transactions of the Association for Computational Linguistics},
      volume={13},
      pages={784--808},
      year={2025},
      doi={10.1162/TACL.a.19},
      url={https://doi.org/10.1162/TACL.a.19},
}

If you use MTRAG-UN, please cite the paper as follows:

@inproceedings{rosenthal-etal-2026-mtrag,
    title = "{MTRAG}-{UN}: A Benchmark for Open Challenges in Multi-Turn {RAG} Conversations",
    author = "Rosenthal, Sara  and
      Katsis, Yannis  and
      Shah, Vraj  and
      He, Lihong  and
      Popa, Lucian  and
      Danilevsky, Marina",
    booktitle = "Findings of the {A}ssociation for {C}omputational {L}inguistics: {ACL} 2026",
    month = jul,
    year = "2026",
    address = "San Diego, California, United States",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2026.findings-acl.503/",
    doi = "10.18653/v1/2026.findings-acl.503",
    pages = "10363--10369",
    ISBN = "979-8-89176-395-1",
}

Our SemEval task, MTRAGEval, can be cited as follows:

@inproceedings{rosenthal-etal-2026-semeval,
    title = "{S}em{E}val-2026 Task 8: {MTRAGE}val: Evaluating Multi-Turn {RAG} Conversations",
    author = "Rosenthal, Sara  and
      Shah, Vraj  and
      Katsis, Yannis  and
      Danilevsky, Marina",
    booktitle = "Proceedings of the 20th {I}nternational {W}orkshop on {S}emantic {E}valuation (2026)",
    month = jul,
    year = "2026",
    address = "San Diego, California, USA",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2026.semeval-1.447/",
    doi = "10.18653/v1/2026.semeval-1.447",
    pages = "3673--3690",
    ISBN = "979-8-89176-414-9",
}

Corpora

Our benchmark is built on document corpora from 4 domains: ClapNQ, Cloud, FiQA and Govt. ClapNQ and FiQA are existing corpora from QA/IR datasets, while Govt and Cloud are new corpora assembled specifically for this benchmark.

Important

Download and uncompress the files to use the corpora.

CorpusDomainData# Documents# Passages
ClapNQ [1]WikipediaCorpus4,293183,408
CloudTechnical DocumentationCorpus57,63861,022
FiQA [2]FinanceCorpus7,66149,607
GovtGovernmentCorpus8,57872,422

Note

Please see the corpora README regarding using the corpus at passage level (preferred) vs document level.

MTRAG - Human Data

MTRAG consists of 110 multi-turn conversations that are converted to 842 evaluation tasks, spanning conversations, retrieval tasks, and generation tasks across four domains.

See the mtrag-human README for full details.

MTRAG - Synthetic Data

We provide 200 synthetically generated conversations and generation tasks that follow the properties of the human data.

See the mtrag-synthetic README for full details.

MTRAG-UN

MTRAG-UN is a benchmark for exploring open challenges in multi-turn RAG, focusing on UNanswerable, UNderspecified, and NONstandalone questions and UNclear responses. It consists of 666 tasks across 6 domains, including two new enterprise corpora (Banking and Telco).

See the mtragun-human README for full details.

MTRAGEval

MTRAGEval is a task for Evaluating Multi-Turn RAG Conversations at SemEval 2026. MTRAG is the training data and MTRAG-UN is the evaluation benchmark.

Sample data from MTRAG in the format used by the evaluation scripts is available at scripts/evaluation/sample_data.

Getting Started

Running Retrieval

Retrieval experiments can be run using the BEIR codebase as described in the retrieval README. The corpus will need to be ingested to run experiments.

Running Generation

Generation experiments can be run using any desired models (e.g. available on HuggingFace) and settings as described in the generation README.

Evaluating Retrieval and Generation

Retrieval and Generation experiments can be evaluated using our evaluation scripts as described in the evaluation README.

Viewing Evaluations

We provide analytics files in InspectorRAGet format, which can be used to inspect the evaluation results and perform further analysis. Load any of the analytics files in InspectorRAGet by clicking "Visualize" and follow the instructions shown on the screen.

Acknowledgements

  • We'd like to thank our internal annotators for their considerable effort in creating these conversations: Mohamed Nasr, Joekie Gurski, Tamara Henderson, Hee Dong Lee, Roxana Passaro, Chie Ugumori, Marina Variano, Eva-Maria Wolfe
  • We'd like to thank Krishnateja Killamsetty for question classification
  • We'd like to thank Lihong He for corpus ingestion
  • We'd like to thank Aditya Gaur for deployment help

Contributors

Sara Rosenthal, Yannis Katsis, Vraj Shah, Kshitij Fadnis, Chulaka Gunasekara, Young-Suk Lee, Lucian Popa, Huaiyu Zhu, Lihong He, Danish Contractor, Marina Danilevsky

Contact