LLMRouterBench

A Massive Benchmark and Unified Framework for LLM Routing

English | 中文

News · Overview · Results · Installation · Quick Start · Datasets · Model Pools · Citation

News

• April 2026: 🎉🎉🎉 This paper has been accepted to Findings of ACL 2026!

Overview

LLMRouterBench is a large-scale benchmark and unified framework for LLM routing. It consolidates standardized outputs across 21 datasets and 33 models, supports both performance-only and performance-cost routing, and provides adapters for 10 representative routing baselines.

What's Included

• Two Routing Paradigms: Performance-oriented | Performance-Cost tradeoff
• State-of-the-Art Model Pools: 20 lightweight ~7B LLMs (Qwen3-8B, DS-Qwen3, NVIDIA-Nemo, etc.) + 13 flagship LLMs from 8 providers (GPT-5, Gemini-2.5-Pro, Claude-4, DeepSeek-V3.1, etc.)
• Challenging & Diverse Datasets: Math (AIME, LiveMathBench), Code (LiveCodeBench, SWE-Bench), Logic (BBH, KORBench), Knowledge (HLE, SimpleQA), Affective (EmoryNLP, MELD), Instruction Following (ArenaHard), Tool Use (τ²-Bench)
• Representative Routing Baselines: RouterDC (NeurIPS'24), EmbedLLM (ICLR'25), MODEL-SAT (AAAI'25), Avengers (AAAI'26), HybridLLM (ICLR'24), FrugalGPT (TMLR'24), RouteLLM (ICLR'25), GraphRouter (ICLR'25), Avengers-Pro (DAI'25 Best Paper), OpenRouter
• Substantial Data Collection Cost: ~1K GPU hours + $3000 API spend
• Standardized Data Fields (per instance): origin_query, prompt, prediction(raw output), ground_truth, score, prompt_tokens, completion_tokens, cost
• Modular Architecture: Collector (unified LLM API) → Evaluator (dataset-specific scoring) → Adaptor (algorithm-specific formatting)

Key Findings

Performance-Oriented Setting

No single model rules every domain; models exhibit complementary strengths. As shown below, mathematics benchmarks are led by models like Intern-S1-mini or Qwen3-8B, code benchmarks by Qwen-Coder or Fin-R1, confirming the central premise of LLM routing.

Top routing methods achieve comparable performance, but a significant gap to Oracle remains. We compare routing methods against three baselines: Random (randomly selects a model), Best Single (single model with highest average accuracy), and Oracle (always selects the best model per query—theoretical upper bound). Key metrics include:

• AvgAcc: Average accuracy across all datasets
• Gain@R / Gain@B: Relative improvement over Random / Best Single
• Gap@O: Gap to Oracle (lower is better)

Despite methodological differences, leading routers (EmbedLLM, GraphRouter, MODEL-SAT, Avengers) yield similar results. Notably, Avengers achieves this without neural-network training. The proximity to Dataset Oracle (which selects the best model per dataset, shown as hatched bars) suggests that routing gains largely stem from capturing coarse-grained domain structure. However, a significant Gap@O remains due to model-recall failures—when only a few models answer correctly, routers often fail to select them.

Performance-Cost Setting

Effective routing achieves significant gains, but not all routers succeed. We measure performance-cost tradeoffs using:

• PerfGain: Best achievable performance improvement over Best Single (at highest-accuracy configuration)
• CostSave: Maximal cost reduction while maintaining Best Single's accuracy
• Pareto frontier: The set of optimal configurations where no method is simultaneously cheaper and more accurate
• ParetoDist: Average distance to Pareto frontier (smaller is better)

Top methods achieve up to 4% PerfGain and 31.7% CostSave. However, some routers (including commercial ones like OpenRouter) fail to outperform the Best Single. Avengers-Pro dominates the Pareto frontier with near-zero ParetoDist.

PerfGain and CostSave vs. GPT-5.

Accuracy vs. cost with Pareto frontier.

For more findings (embedding ablations, model pool scaling, latency analysis, etc.), please refer to our paper.

Original Data Tables

Installation

git clone https://github.com/ynulihao/LLMRouterBench.git
cd LLMRouterBench
pip install -r requirements.txt

Quick Start

python -m data_collector.cli run \
    config/data_collector_small_model_config.yaml

from baselines import BaselineDataLoader, BaselineAggregator

loader = BaselineDataLoader("config/baseline_config.yaml")
records = loader.load_all_records()
agg = BaselineAggregator(records, data_loader=loader)
agg.print_summary_tables(
    score_as_percent=True,
    test_mode=False
)

python -m baselines.adaptors.avengerspro_adaptor \
    --config config/baseline_config.yaml \
    --seed 42 \
    --split-ratio 0.7 \
    --output-dir baselines/AvengersPro/data/small_models_seed_42

python -m baselines.AvengersPro.simple_cluster_router \
    --config baselines/AvengersPro/config/simple_config_small_models_42.json \
    --output baselines/AvengersPro/logs/simple_config_small_models_42.json

Configurations

LLMRouterBench supports two routing paradigms with corresponding configuration files:

Core Components

LLMRouterBench provides a modular three-component architecture:

Customization

Datasets

LLMRouterBench includes 21 unique datasets across two settings.

Model Pools

Data

LLMRouterBench stores standardized JSON records under results/bench/ and provides download bundles for pre-collected results.

Data Download

Download pre-collected benchmark results:

# Extract to results directory
tar xzf bench-release.tar.gz

Directory structure after extraction:

results/
└── bench/
    ├── aime/
    ├── bbh/
    ├── humaneval/
    ├── mmlu_pro/
    └── ...

See results/download.md for details.

{
  "performance": 0.85,
  "time_taken": 120.5,
  "prompt_tokens": 50000,
  "completion_tokens": 20000,
  "cost": 0.15,
  "counts": 100,
  "records": [
    {
      "index": 1,
      "origin_query": "What is the sum of 2+2?",
      "prompt": "Question: What is the sum of 2+2?\nAnswer:",
      "prediction": "4",
      "ground_truth": "4",
      "score": 1.0,
      "prompt_tokens": 15,
      "completion_tokens": 5,
      "cost": 0.0001,
      "raw_output": "The sum of 2+2 is 4."
    }
  ]
}

from baselines import BaselineDataLoader, BaselineAggregator

loader = BaselineDataLoader('config/baseline_config.yaml')
records = loader.load_all_records()

test_mode = False
score_as_percent = True
train_ratio = 0.7
random_seed = 3407

agg = BaselineAggregator(records, data_loader=loader)
agg.print_summary_tables(
    score_as_percent=score_as_percent,
    test_mode=test_mode,
    random_seed=random_seed,
    train_ratio=train_ratio,
)

agg.save_summary_tables_to_excel(
    output_file='small_models_total.xlsx',
    score_as_percent=score_as_percent,
    test_mode=test_mode,
    train_ratio=train_ratio,
    random_seed=random_seed
)

Project Structure

LLMRouterBench/
├── data_collector/     # Collector module
├── evaluation/         # Evaluator (21 datasets)
├── baselines/          # Adaptor & routing algorithms
├── generators/         # Model API interface
├── common/cache/       # Caching system
├── external_bench/     # Third-party integration
├── config/             # Configuration files
└── results/            # Benchmark results

🗓️ Roadmap

Recent Updates

• ✅ Integrated three challenging benchmarks (FrontierScience, SGIBench, SFE)
• ✅ Extended support for multimodal routing evaluation

Long-term Goals

• Broader model coverage
• Expanded benchmark suite
• Additional baseline methods
• Extended routing paradigms

Related Work

Comparison with Existing Routing Benchmarks

Existing routing benchmarks face several limitations:

• RouterBench: Restricted to early-generation models and 8 relatively simple datasets.
• EmbedLLM & RouterEval: Focus on open-source models without inference cost information.
• FusionFactory: Benchmarks open-source models with estimated costs.
• RouterArena: Uses inconsistent model pools across routers, undermining fair comparison and lacking per-prompt, per-model data.

Citation

If you find LLMRouterBench useful, consider citing our paper:

@article{li2026llmrouterbench,
  title   = {LLMRouterBench: A Massive Benchmark and Unified Framework for LLM Routing},
  author  = {Li, Hao and Zhang, Yiqun and Guo, Zhaoyan and Wang, Chenxu and Tang, Shengji and Zhang, Qiaosheng and Chen, Yang and Qi, Biqing and Ye, Peng and Bai, Lei and Wang, Zhen and Hu, Shuyue},
  journal = {arXiv preprint arXiv:2601.07206},
  year    = {2026}
}

This work is part of our series of studies on LLM routing; if you're interested, please refer to and cite:

@inproceedings{zhang2025avengers,
  title        = {The Avengers: A Simple Recipe for Uniting Smaller Language Models to Challenge Proprietary Giants},
  author       = {Zhang, Yiqun and Li, Hao and Wang, Chenxu and Chen, Linyao and Zhang, Qiaosheng and Ye, Peng and Feng, Shi and Wang, Daling and Wang, Zhen and Wang, Xinrun and others},
  booktitle    = {Proceedings of the AAAI Conference on Artificial Intelligence (AAAI)},
  year         = {2025},
  note         = {Oral presentation},
  url          = {https://arxiv.org/abs/2505.19797}
}
@inproceedings{zhang2025beyond,
  title        = {Beyond gpt-5: Making llms cheaper and better via performance-efficiency optimized routing},
  author       = {Zhang, Yiqun and Li, Hao and Chen, Jianhao and Zhang, Hangfan and Ye, Peng and Bai, Lei and Hu, Shuyue},
  booktitle    = {Distributed AI (DAI) conference},
  year         = {2025},
  note         = {Best Paper Award},
  url          = {https://arxiv.org/abs/2508.12631}
}
@inproceedings{wang2025icl,
  title        = {ICL-Router: In-Context Learned Model Representations for LLM Routing},
  author       = {Wang, Chenxu and Li, Hao and Zhang, Yiqun and Chen, Linyao and Chen, Jianhao and Jian, Ping and Ye, Peng and Zhang, Qiaosheng and Hu, Shuyue},
  booktitle    = {Proceedings of the AAAI Conference on Artificial Intelligence (AAAI)},
  year         = {2025},
  note         = {Poster},
  url          = {https://arxiv.org/abs/2510.09719}
}
@article{chen2025learning,
  title        = {Learning Compact Representations of LLM Abilities via Item Response Theory},
  author       = {Chen, Jianhao and Wang, Chenxu and Zhang, Gengrui and Ye, Peng and Bai, Lei and Hu, Wei and Qu, Yuzhong and Hu, Shuyue},
  journal      = {arXiv preprint arXiv:2510.00844},
  year         = {2025},
  url          = {https://arxiv.org/abs/2510.00844v1}
}