Benchmark.md
September 4, 2024 ยท View on GitHub
Benchmarks
| Benchmark | Year | Language | Domain Type | Turn Type | Collection | Paper Link | Download Link |
|---|---|---|---|---|---|---|---|
| ATIS | 1994 | English | Single-domain | Single | Hand-crafted | Paper | Download |
| GeoQuery | 1996 | English | Single-domain | Single | Hand-crafted | Paper | Download |
| Restaurants | 2003 | English | Single-domain | Single | Hand-crafted | Paper | Download |
| Academic | 2014 | English | Single-domain | Single | Hand-crafted | Paper | Download |
| IMDb | 2017 | English | Single-domain | Single | Hand-crafted | Paper | Download |
| Yelp | 2017 | English | Single-domain | Single | Hand-crafted | Paper | Download |
| Scholar | 2017 | English | Single-domain | Single | Hand-crafted | Paper | Download |
| WikiSQL | 2017 | English | Cross-domain | Single | Hand-crafted | Paper | Download |
| Advising | 2018 | English | Single-domain | Single | Hand-crafted | Paper | Download |
| Spider | 2018 | English | Cross-domain | Single | Hand-crafted | Paper | Download |
| SParC | 2019 | English | Cross-domain | Multiple | Hand-crafted | Paper | Download |
| CoSQL | 2019 | English | Cross-domain | Multiple | Hand-crafted | Paper | Download |
| CSpider | 2019 | Chinese | Cross-domain | Single | Hand-crafted | Paper | Download |
| MIMICSQL | 2020 | English | Single-domain | Single | Auto-generated + Mannual | Paper | Download |
| SQUALL | 2020 | English | Cross-domain | Single | Hand-crafted | Paper | Download |
| FIBEN | 2020 | English | Single-domain | Single | Hand-crafted | Paper | Download |
| ViText2SQL | 2020 | Vietnamese | Cross-domain | Single | Hand-crafted | Paper | Download |
| DuSQL | 2020 | Chinese | Cross-domain | Single | Auto-generated + Mannual | Paper | Download |
| PortugueseSpider | 2021 | Portuguese | Cross-domain | Single | Auto-generated + Mannual | Paper | - |
| CHASE | 2021 | Chinese | Cross-domain | Multiple | Hand-crafted | Paper | Download |
| Spider-Syn | 2021 | English | Cross-domain | Single | Hand-crafted | Paper | Download |
| Spider-DK | 2021 | English | Cross-domain | Single | Hand-crafted | Paper | Download |
| Spider-Realistic | 2021 | English | Cross-domain | Single | Hand-crafted | Paper | Download |
| KaggleDBQA | 2021 | English | Cross-domain | Single | Hand-crafted | Paper | Download |
| SEDE | 2021 | English | Single-domain | Single | Hand-crafted | Paper | Download |
| MT-TEQL | 2021 | English | Cross-domain | Single | Auto-generated | Paper | Download |
| PAUQ | 2022 | Russian | Cross-domain | Single | Hand-crafted | Paper | Download |
| knowSQL | 2022 | Chinese | Cross-domain | Single | Hand-crafted | Paper | - |
| Dr.Spider | 2023 | English | Cross-domain | Single | Auto-generated + Mannual | Paper | Download |
| BIRD | 2023 | English | Cross-domain | Single | Hand-crafted | Paper | Download |
| AmbiQT | 2023 | English | Cross-domain | Single | ChatGPT-aided + Mannual | Paper | Download |
| ScienceBenchmark | 2024 | English | Single-domain | Single | Auto-generated + Mannual | Paper | Download |
| BULL | 2024 | English/Chinese | Single-domain | Single | Hand-crafted | Paper | Download |
| BookSQL | 2024 | English | Single-domain | Single | Hand-crafted | Paper | Download |
| Archer | 2024 | English/Chinese | Cross-domain | Single | Hand-crafted | Paper | Download |