Model Router: Improving LLM Response Quality While Optimizing Cost

Introduction

• KT Agentic AI Lab's A-Pattern Team researches and develops the Model Router.
• This article explains the concept of the Model Router, the problem it is intended to solve, and the design and architecture proposed by KT.

Why We Need a Model Router

• A wide range of LLMs is now available, including KT's Mi:dm K 2.0, SOTA K, and Llama K, as well as GPT, Gemini, and Claude. However, when building LLM services, agent services, and ultimately MAS (Multi-Agent Service), it is important to determine whether these models are being used effectively.
• When developing an LLM-powered application, teams typically integrate, compare, and evaluate multiple LLMs before selecting the model best suited to the service. Suitability must account not only for response quality but also for inference cost. For commercial services in particular, inference cost is a key decision factor.
• Comparing multiple LLMs can improve efficiency to some extent, but processing every user request with a single LLM is unlikely to provide the optimal balance between quality and cost.
• LLMs generally follow scaling laws: performance improves according to a power law as the number of parameters, the amount of training data, and compute increase [1, 2]. At the same time, inference cost also rises as model size grows. This creates a positive relationship between quality and cost among viable models.
• Some models may be excluded because another candidate delivers higher quality at a lower cost. Among the remaining candidates, users must ordinarily choose a single model according to the desired quality-cost balance.
• Dynamically selecting an LLM according to the characteristics of each request can achieve a combination of quality and cost that is difficult to reach with any single model. The purpose of a Model Router is therefore to combine multiple LLMs dynamically and deliver both better quality and greater cost efficiency than a single-LLM approach [3, 4].

Representative Model-Routing Approaches

• A Model Router can select an LLM dynamically for each user request through two major approaches.

i. User-Preference-Based Routing

• User-preference-based routing trains a routing model through supervised learning on data composed of <user request, preference by LLM> pairs.
• This approach is intuitive to design and supports fine-grained, request-level routing without category constraints.
• However, the router must be retrained whenever the target LLMs change. It is also difficult to obtain high-quality preference data at scale.
• Representative preference-based Model Routers include the approaches described in [5, 6].

ii. Category-Based Routing

• Category-based routing first classifies a user request into a predefined category and then maps that category to an LLM.
• Implementing this approach requires:
  1. Defining the category system.
  2. Developing a Category Classifier.
  3. Establishing a category-to-LLM mapping policy.
• Overall performance depends on classification quality and the mapping policy. However, because the Category Classifier is loosely coupled to the target LLMs, the approach can respond flexibly when models are replaced in a production environment.
• <User request, category> data is also easier to obtain than preference data.
• Representative category-based Model Routers include the approaches described in [7, 8].
• KT implemented its Model Router using category-based routing because this approach is suitable for commercial services.

Model Router Category System

• Implementing a category-based Model Router begins with defining a category system. Common classification frameworks include:
  • Dewey Decimal Classification (DDC) [9]: A knowledge-classification system used by libraries. It divides all fields of knowledge into ten main classes and organizes them systematically through numerical subdivisions.
  • Bloom's Taxonomy [10, 11]: An educational framework that classifies the cognitive difficulty of learning objectives. It organizes levels of thinking progressively, such as Remember, Understand, Apply, Analyze, and Create.
• The Model Router defines categories across three dimensions—Task, Level, and Domain—by combining Domain and Level criteria with the Task criteria commonly used in LLM evaluation benchmarks.
• Each dimension contains detailed subcategories to support more precise request classification [12].
• Before assigning labels, the classifier identifies the user's core intent from the full conversation, including relevant system instructions and earlier messages. Labels reflect the underlying purpose and cognitive demand of the request rather than isolated keywords or surface wording.
• Task, Domain, and Level are classified independently. Multiple Task or Domain labels are assigned only when the request genuinely requires more than one distinct task type or area of knowledge; Level always receives exactly one label.

Task

• Task describes what the user is asking the system to do:
  • T1 — Translation: Translate between natural languages.
  • T2 — Summarization: Condense provided content while preserving its key information.
  • T3 — Math Problem Solving: Perform calculations, apply formulas, solve equations, or use mathematical reasoning.
  • T4 — Code Tasks: Write, modify, explain, review, or debug source code.
  • T5 — Text Editing: Revise existing text without changing its meaning, including grammar correction, style refinement, or politeness adjustment.
  • T6 — Image Generation: Generate an image from a request.
  • T7 — Creative Writing: Produce creative or emotion-focused text, such as advertising copy, poetry, or fiction, or substantially re-express content in a new style or emotional tone.
  • T8 — Planning/Reasoning: Provide strategies, recommendations, judgments, decisions, opinions, advice, comparisons, causal analysis, or reasoning over hypothetical and conditional cases.
  • T9 — Classification: Assign a given input to one or more categories.
  • T10 — Factoid QA: Answer a factual or definitional question directly without substantial reasoning.
  • T11 — Chat: Engage in empathy, comfort, casual conversation, fortune-telling, or other emotion-centered interaction.
  • T12 — Unknown: Handle an ambiguous, uninterpretable, or otherwise unclassifiable request.
• Multiple Task labels capture distinct operations when both are necessary. For example, translating a sentence and making it more polite is labeled as both Translation and Text Editing.

Domain

• Domain describes which body of knowledge is required to answer the request accurately:
  • D1 — Mathematics/Statistics: Mathematics and statistics.
  • D2 — Finance/Economics: Finance, investment, and economics.
  • D3 — Computing/Programming: Computer science, programming, software and system development, and artificial intelligence technologies.
  • D4 — Health/Medicine: Health, diseases, treatment, and medical information.
  • D5 — Science/Technology: Science, engineering, and general technology outside the more specific computing domain.
  • D6 — Law: Laws, legal matters, and contracts.
  • D7 — Politics: Political systems, government policy, elections, political parties and ideologies, diplomacy, and public governance.
  • D8 — Humanities/Society/Culture: Humanities, sociology, history, philosophy, arts, and culture.
  • D9 — Daily Life: Everyday activities, personal expenses, hobbies, scheduling, and general office work.
  • D10 — Uncategorized: Non-informational or uninterpretable input, or a request whose domain cannot be determined.
• Multiple Domain labels are used only when an accurate response genuinely requires knowledge from more than one area. For example, a question about insurance reimbursement and medical tax deductions may require both Finance/Economics and Health/Medicine.

Level

• Level describes how difficult the request is, based on both lexical complexity and reasoning depth. Reasoning demand takes precedence over superficially simple wording, and exactly one Level label is selected.
• The Level dimension supports two alternative schemes:
  • Type 1: Three discrete levels:
    • L1 — Easy: Everyday vocabulary; direct factoid or simple definition questions; no reasoning required.
    • L2 — Intermediate: Vocabulary at or below the high-school curriculum level; simple conceptual comparisons that do not require judgment; or straightforward requests with incomplete context.
    • L3 — Complex: Undergraduate-major-level terminology or requests requiring strategy, selection, judgment, decision-making, opinions, advice, comparative analysis, causal or relational reasoning, or hypothetical and conditional branching. A request is classified as L3 when it requires deep reasoning even if it is phrased simply.
  • Type 2: A single composite score calculated as a weighted sum of the following subcategory values:
    • Reasoning Depth & Steps
    • Constraint/Requirement Density
    • Accuracy/Verification Demand
    • Domain Specificity

Model Router Architecture

• The Model Router applies category-based routing. It classifies each user request and routes it to one of several LLMs according to a predefined routing policy.
• A routing policy is a {category → LLM} mapping that determines which LLM processes requests belonging to each category.
• The Model Router consists of the following major components:
  • TLD Classifier: Accepts a user request and assigns a category composed of Task, Level, and Domain.
    • During routing-policy generation, the category is passed to the Quality Evaluator and used to select a task-specific LLM-as-a-Judge [13].
    • During inference, the category is used to route the request to an LLM according to the policy produced by the Routing Policy Optimizer.
  • Quality Evaluator: Generates LLM-response-quality data used to identify the optimal LLM.
  • Routing Policy Optimizer: Generates candidate routing policies, identifies the Pareto-optimal policies in terms of quality and cost, and selects the optimal routing policy.

1. Routing-Policy Generation

• Routing-policy generation consists of two processes:
  1. Obtaining quality and cost information for each LLM in relation to user requests.
  2. Identifying the optimal quality-cost policy among the many possible routing-policy candidates.

1-1. Obtaining Quality and Cost Information

• Building a routing policy requires quality and cost information for each LLM's response to a user request.
• The information is collected as follows:
  • Step 0: Collect user queries for defining the routing policy.
  • Step 1: Send the user queries to the target LLMs and collect their responses. The response data includes cost information.
  • Step 2: Classify each user query so that a category-specific LLM-as-a-Judge can be used.
  • Step 3: Use the category-specific LLM-as-a-Judge to evaluate the quality of each <user query, LLM response> pair.
  • Result: Obtain per-LLM quality and cost information for each user query.
• The collected information provides the quality-cost data needed to compare the candidate LLMs and optimize a routing policy.

1-2. Identifying Pareto-Optimal Routing Policies

• Selecting different LLMs according to the characteristics of each request can produce a better quality-cost combination than applying one model uniformly.
• A lightweight model can process relatively simple requests at lower cost, while a high-performance model handles complex or difficult requests. This can maintain or improve average system-wide quality while reducing total inference cost.
• This is a multi-objective optimization problem involving both cost and quality. The solutions for which one objective cannot be improved without worsening the other form the Pareto-optimal set from a quality-cost perspective [4, 15].
• Pareto-optimal routing policies are identified as follows:
  • Step 1: Obtain quality and cost information through the process described in Section 1-1.
  • Step 2: Generate routing-policy candidates.
    • Because a routing policy maps each category to one LLM, a system with four LLMs and 360 categories ($10 \times 12 \times 3$) can have as many as 4^360 possible routing policies.
  • Step 3: Retain only the routing policies that are Pareto-optimal in terms of cost and quality.
    • A Pareto-optimal routing policy is not outperformed by another policy in quality and/or cost without requiring a trade-off.
  • Step 4: Select one of the Pareto-optimal routing policies according to service requirements.
    • The selected policy can achieve a high-quality, high-efficiency operating point that would be difficult to reach with a single LLM.

2. Routing

• When the Model Router receives a user query, it analyzes the query in real time, classifies it by Task, Level, and Domain, and selects the appropriate LLM according to the predefined routing policy [14].
• The TLD Classifier must minimize routing overhead while maintaining accurate and consistent classification performance.
• Routing-policy generation performs offline optimization, while routing is the online inference process that executes the selected policy efficiently in real time.
• The Model Router operates a mixture of encoder-based models, such as KoBigBird [16, 17], and decoder-based models, such as Qwen-3-4B [18], according to the length and complexity of the user query.
• Encoder models require relatively few resources and support fast inference, but their input context length is limited. Even large BERT-family models, for example, typically support a context length of approximately 8K.
• Decoder models can process longer input contexts, but require more computation and therefore introduce greater inference latency and overhead.
• The TLD Classifier is trained through supervised learning. Its training data is constructed through knowledge distillation using a teacher model [19].

Preliminary Evaluation

• As a preliminary evaluation of the Model Router, we conducted an experiment using three GPT-family models.

Dataset

• Sources: NIA-Helpfulness, KoAlpaca, Won-Instruct, HAERAE, OIG-small-chip2-ko, KoChatGPT, HRC, and others.
• Size: 1,500 calibration samples and 1,500 test samples.
• Sampling: Partial stratified sampling based on TLD categories.

Target LLMs

• GPT-4.1
• GPT-4.1-mini
• GPT-OSS-120B

Evaluation Procedure and Results

• A routing policy was first generated using the calibration data. The resulting policy was then applied to the test data to measure generalization performance [3, 7].
• During calibration, the quality-cost points of the three GPT models were evaluated together with the Pareto-optimal routing policies. One policy was selected and applied to the test data, and its performance was compared with the individual models.
• The Model Router maintained slightly higher quality than GPT-4.1—101% of its score—while reducing cost to approximately 56% of GPT-4.1's cost. This corresponds to a cost reduction of approximately 44%.
• These preliminary results suggest that dynamic routing based on request characteristics can improve the quality-cost balance compared with uniformly applying a single high-performance model.

Table 1. Model Router Results

Model	Quality	Cost
GPT-4.1	1,289	5.45
GPT-4.1-mini	1,253	0.978
GPT-OSS-120B	1,258	1.73
Model Router	1,296 (101% of GPT-4.1)	3.04 (56% of GPT-4.1)

RouterArena

• Beyond the preliminary evaluation, we submitted the Model Router to the RouterArena leaderboard [8], an open benchmark for comparing LLM routers on accuracy and cost.

Dataset

• Source: RouterArena's full benchmark query set.
• Size: 8,400 test queries, with a 1,976-query subset, stratified by TLD category, held out for calibration (routing-policy generation).

Model Pool

• deepseek/deepseek-v4-flash
• deepseek/deepseek-v4-pro
• gemma-4-31B-it (reference model)
• google/gemini-3-flash-preview
• qwen/qwen3-235b-a22b-2507

TLD Classifier

• mmBERT-based TLD classifier. The resulting policy and its TLD → model mapping are included with this PR: routing_policy_for_router_arena.yaml and routing-heatmap.html.

Evaluation Procedure and Results

• The routing policy was generated on the calibration set through confidence-filtered Pareto selection, then applied unchanged to the full RouterArena test set.
• On the test set, the Model Router reached an arena score of 0.764 (78.2% accuracy) at an average cost of $0.270 per 1,000 queries.
• Compared with the best single model in the pool (google/gemini-3-flash-preview, no routing: arena score 0.761, 78.9% accuracy, $0.484 per 1,000 queries), the Model Router matches quality — 100.4% of its arena score — at about 56% of the cost, a roughly 44% cost reduction.
• On the public RouterArena leaderboard, this places the Model Router 2nd among submitted routers, behind Paix2 and ahead of Sqwish Router, Divyam, Cross-Router, and vLLM-SR — and ahead of simply using the best single model in the pool.

Table 2. RouterArena Leaderboard (submitted routers, ranked by arena score)

Rank	Router	Arena Score	Accuracy
1	Paix2	0.776	79.7%
2	Model Router (ours)	0.764	78.2%
3	Sqwish Router	0.762	79.8%
4	Divyam	0.759	78.6%
5	Cross-Router	0.758	78.1%
6	vLLM-SR	0.749	77.2%

Reference points, not ranked as submitted routers: query-level oracle — 0.857 arena score / 87.1% accuracy (per-query upper bound); TLD-cell ceiling — 0.772 / 79.3% (best achievable ceiling under this TLD taxonomy); best single model, no routing — 0.761 / 78.9%.

References

1. Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., & Amodei, D. (2020). Scaling laws for neural language models. arXiv.
2. Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., … Sifre, L. (2022). Training compute-optimal large language models. arXiv.
3. Chen, L., Zaharia, M., & Zou, J. (2023). FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance. arXiv.
4. Šakota, M., Peyrard, M., & West, R. (2024). Fly-Swat or Cannon? Cost-Effective Language Model Choice via Meta-Modeling. In Proceedings of the 17th ACM International Conference on Web Search and Data Mining (WSDM 2024) (pp. 606–615). Association for Computing Machinery.
5. Ong, I., Almahairi, A., Wu, V., Chiang, W.-L., Wu, T., Gonzalez, J. E., Kadous, M. W., & Stoica, I. (2024). RouteLLM: Learning to Route LLMs with Preference Data. arXiv.
6. Lu, K., Yuan, H., Lin, R., Lin, J., Yuan, Z., Zhou, C., & Zhou, J. (2023). Routing to the expert: Efficient reward-guided ensemble of large language models. arXiv.
7. Hu, Q. J., Bieker, J., Li, X., Jiang, N., Keigwin, B., Ranganath, G., Keutzer, K., & Upadhyay, S. K. (2024). RouterBench: A benchmark for multi-LLM routing system. arXiv.
8. Lu, Y., Liu, R., Yuan, J., Cui, X., Zhang, S., Liu, H., & Xing, J. (2025). RouterArena: An open platform for comprehensive comparison of LLM routers. arXiv.
9. Dewey, M. (1876). A classification and subject index for cataloguing and arranging the books and pamphlets of a library. (Centennial facsimile text). Forest Press Division, Lake Placid Educational Foundation.
10. Bloom, B. S., Engelhart, M. D., Furst, E. J., Hill, W. H., & Krathwohl, D. R. (1956). Taxonomy of educational objectives: The classification of educational goals. Handbook I: Cognitive domain. David McKay.
11. Anderson, L. W., & Krathwohl, D. R. (Eds.). (2001). A taxonomy for learning, teaching, and assessing: A revision of Bloom's taxonomy of educational objectives. Longman.
12. Lu, Y., Liu, R., Yuan, J., Cui, X., Zhang, S., Liu, H., & Xing, J. (2025). RouterArena: An open platform for comprehensive comparison of LLM routers. arXiv.
13. Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., … Stoica, I. (2023). Judging LLM-as-a-judge with MT-Bench and Chatbot Arena. arXiv.
14. Hari, S. N., & Thomson, M. (2023). Tryage: Real-time, intelligent routing of user prompts to large language models. arXiv.
15. Liu, Y., Zhang, H., Miao, Y., Le, V.-H., & Li, Z. (2024). OptLLM: Optimal assignment of queries to large language models. In Proceedings of the IEEE International Conference on Web Services (ICWS 2024) (pp. 788–798). IEEE.
16. Yang, K., Jang, Y., Lee, T., Seong, J., Lee, H., Jang, H., & Lim, H. (2023). KoBigBird-large: Transformation of transformer for Korean language understanding. In Proceedings of the 13th International Joint Conference on Natural Language Processing and the 3rd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (Volume 1: Long Papers) (pp. 1058–1066). Association for Computational Linguistics.
17. Warner, B., et al. (2024, December 19). Finally, a replacement for BERT: Introducing ModernBERT. Hugging Face Blog.
18. Qwen Team. (2024). Qwen3-7B Technical Report. Alibaba Cloud / Qwen.
19. Hinton, G., Vinyals, O., & Dean, J. (2015). Distilling the knowledge in a neural network. arXiv.