Model Routing: When and How to Send Different Queries to Different Models

1. Definition

Model routing is the architectural pattern of selecting which model should answer a given query. In its simplest form, a router chooses between a cheaper “small” model and a more capable “large” model. In production systems, the router may choose among model families, providers, regions, inference endpoints, tool-capable variants, context-window sizes, and reasoning-effort settings.

The core decision is:

For this request, under this cost, latency, reliability, policy, and quality target, which model invocation is expected to produce the best outcome?

NotDiamond defines model routing as analyzing an input and predicting which LLM can produce the highest-quality response at the lowest cost; its product exposes modes such as quality, cost, and latency optimization. ([NotDiamond, “What is Model Routing?”] Not Diamond) OpenRouter’s Auto Router similarly describes a prompt-analysis step, model selection, request forwarding, and response tracking, with the selected model returned in the API response. ([OpenRouter Auto Router documentation] OpenRouter) RouteLLM formalizes the problem as choosing between a stronger and weaker model using a routing threshold that controls the cost-quality tradeoff. ([RouteLLM paper] arXiv)

A model router is not only a “smart if-statement.” It is a control-plane component that depends on LLM Evaluation, Inference Cost Accounting, Latency Budgets, Fallbacks and Circuit Breakers, and Production Observability. Without those pieces, routing can quietly degrade quality while making costs appear better.

2. Why model routing exists

The motivating observation is that no single model is uniformly optimal across all tasks, prices, latencies, and deployment constraints. Martian’s RouterBench writeup frames routing around the fact that model performance and cost vary by task, so a system can exploit the full landscape of LLMs instead of treating one model as globally best. ([Martian RouterBench] Martian) RouteLLM makes the same point empirically: it trains routers over human preference data so that simple requests can go to cheaper models while harder requests go to stronger models. ([RouteLLM LMSYS blog] LMSYS Org)

Routing appears when at least one of these is true:

Condition	Why routing helps	Example
Task difficulty varies	Easy prompts can be answered by cheaper models; hard prompts need stronger reasoning.	FAQ answers vs. multi-hop technical debugging.
Task type varies	Different models specialize in code, math, summarization, extraction, long context, multilingual work, or tool use.	Use a code-specialized model for repository edits and a general model for prose.
Cost spread is large	The same application can spend orders of magnitude more or less depending on model choice.	FrugalGPT exploited price differences between LLM APIs and learned cascades to reduce cost while maintaining quality. ([FrugalGPT paper] arXiv)
Provider reliability varies	Outages, rate limits, moderation flags, and regional availability require fallbacks.	OpenRouter’s fallback mechanism retries alternative models when a first model errors, including context-limit, moderation, rate-limit, or downtime failures. ([OpenRouter fallback documentation] OpenRouter)
Compliance constraints vary	Some requests require zero-data-retention, regional routing, allowlisted providers, or no external calls.	OpenRouter exposes provider-routing controls for provider order, data collection settings, zero-data-retention requirements, price, latency, and throughput preferences. ([OpenRouter provider routing documentation] OpenRouter)
User tiers vary	Free, pro, enterprise, and internal users may have different budgets and latency expectations.	A SaaS assistant may route premium users to stronger models and anonymous traffic to cheaper models unless escalation is needed.

The economic case is strongest when the router can reliably identify prompts that do not need the most expensive model. The quality case is strongest when different models are complementary rather than merely ranked versions of one another. The reliability case is strongest when the router is also a LLM Gateway that can retry, fail over, or enforce provider policy.

3. The architectural pattern

A production router usually has five layers:

1. Request analysis. Extract features from the prompt: task type, length, language, domain, user tier, tool needs, safety class, context-window requirements, latency budget, and expected output length.

2. Candidate filtering. Remove models that cannot satisfy hard constraints: unavailable providers, insufficient context window, unsupported tool-calling, disallowed data policy, missing JSON mode, excessive price, or regional restrictions. OpenRouter’s provider routing docs expose exactly this kind of filtering and sorting through provider order, allowed providers, ignored providers, quantization filters, price limits, throughput limits, latency limits, and data-collection constraints. ([OpenRouter provider routing documentation] OpenRouter)

3. Scoring. Estimate expected quality, expected cost, expected latency, and risk for each candidate. Learned routers often predict whether a weaker model will be “good enough” compared with a stronger model; RouteLLM, for example, trains routers on preference data and uses a threshold to trade off quality against cost. ([RouteLLM paper] arXiv)

4. Dispatch. Invoke the selected model or provider endpoint. Some systems return the selected model to the application, as NotDiamond’s select_model API does, leaving the application to call that model through its preferred SDK. ([NotDiamond chat router documentation] Not Diamond) Others act as a gateway and forward the request themselves, as OpenRouter’s Auto Router describes. ([OpenRouter Auto Router documentation] OpenRouter)

5. Telemetry and learning. Log the selected model, candidate set, router score, latency, token usage, cost, fallback behavior, user feedback, eval score, and downstream business outcome. OpenRouter returns the actual model used, and NotDiamond returns a session identifier and provider/model selection; those fields are the minimum needed for later evaluation and debugging. ([OpenRouter Auto Router documentation] OpenRouter) ([NotDiamond chat router documentation] Not Diamond)

A canonical request path looks like this:

User request  ↓Policy filter  - data retention  - region  - user tier  - tool support  - context length  ↓Router  - rules  - classifier  - embedding retrieval  - cost-quality threshold  - latency/availability signals  ↓Selected model or cascade  ↓LLM response  ↓Evaluation + telemetry  - cost  - latency  - quality  - fallback  - user feedback  - regression checks

The router is therefore a Control Plane decision, not just an inference optimization. It can become the place where product policy, infrastructure policy, and evaluation policy meet.

4. Canonical implementations

4.1 Martian

Martian presented model routing as a way to choose the best LLM for each query rather than committing to one global model. Its launch post claimed that cheap models could be dramatically less expensive than frontier models and that a router could route individual queries in real time to improve performance and reduce cost. Martian reported, as a vendor benchmark, outperforming GPT-4 on 91.8% of OpenAI eval tasks with an average 20% cost reduction when optimizing for performance, and matching performance with a 97% cost reduction on some tasks. ([Martian, “Introducing Martian”] Martian)

Martian’s RouterBench contribution is more generally useful than any single product claim. RouterBench distinguishes predictive routers, which choose a model before generation, from non-predictive routers, which generate one or more outputs and then decide. It includes KNN and MLP predictive routers, plus non-predictive cascades and overgenerate-and-rerank baselines. ([Martian RouterBench] Martian) It also defines an “Oracle Router” as the upper bound and a “Zero Router” baseline, making clear that a router that cannot beat the best single default model is adding complexity without value. ([Martian RouterBench] Martian)

4.2 NotDiamond

NotDiamond is a commercial model-routing API that exposes pretrained routers for chat and code, plus custom routers trained on user evaluation data. Its documentation describes routing as selecting the best model from a candidate set to improve accuracy and reduce cost, with optimization modes for quality, cost, and latency. ([NotDiamond, “What is Model Routing?”] Not Diamond)

The implementation pattern is explicit: call select_model, receive a chosen provider/model and session ID, then call the selected model using the application’s usual SDK. ([NotDiamond chat router documentation] Not Diamond) For custom routers, NotDiamond requires inputs, candidate model responses, and evaluation scores, then trains a routing algorithm over the candidate set. Its docs state a minimum of 15 samples and caps such as 10,000 samples or 5 MB for training data. ([NotDiamond custom router documentation] Not Diamond)

NotDiamond’s code router documentation claims that its coding-agent router accounts for provider cache pricing and can reduce coding-agent cost by more than 30% without affecting code quality. That should be treated as a vendor claim unless reproduced on the reader’s own coding workload. ([NotDiamond code router documentation] Not Diamond)

4.3 OpenRouter

OpenRouter is best understood as a model and provider gateway with routing features at multiple levels. Its quickstart describes a unified API across hundreds of models and says it handles fallbacks and cost-effective selection. ([OpenRouter quickstart] OpenRouter) Its Auto Router uses openrouter/auto, powered by NotDiamond, to analyze a prompt, select a model, forward the request, and track the response. ([OpenRouter Auto Router documentation] OpenRouter)

OpenRouter also exposes provider routing, which is different from task-level model routing but operationally important. Provider routing can sort endpoints by price, throughput, or latency; set provider order; restrict providers; require zero-data-retention; enforce maximum price; and use latency/throughput percentiles over recent windows. ([OpenRouter provider routing documentation] OpenRouter) ([OpenRouter provider latency documentation] OpenRouter) This makes OpenRouter a useful example of routing as infrastructure rather than only as an ML classifier.

OpenRouter’s fallback docs illustrate the reliability side of routing: an application can specify a priority-ordered list of models, and OpenRouter will try alternatives when a model errors because of context length, moderation, rate limits, or downtime. ([OpenRouter fallback documentation] OpenRouter) Its reasoning-token docs also show how gateways can normalize model-specific controls such as reasoning effort across different providers. ([OpenRouter reasoning-token documentation] OpenRouter)

4.4 Custom internal routers

Many teams build internal routers because their real objective is not “pick the best public model” but “pick the cheapest safe-enough path for this product.” A customer-support bot may route password-reset requests to a small model, billing disputes to a stronger model, and regulated medical or legal requests to a human escalation queue. A coding agent may route file search and simple edits to a cheap model, architectural planning to a stronger reasoning model, and tests to tools.

Internal routers often start as rules:

if requires_tool_calling and model_does_not_support_tools: excludeif prompt_tokens > context_window: excludeif user_tier == enterprise and provider lacks ZDR: excludeif task == "simple_extraction": prefer cheap structured-output modelif task == "architecture_review": prefer strongest reasoning model

Then they become learned routers once enough evaluation data exists. Anyscale’s engineering writeup on building an LLM router describes the production workflow as data labeling, fine-tuning a router, and offline evaluation, which is the usual transition from rules to model-based routing. ([Anyscale engineering blog] Anyscale)

AWS Bedrock’s Intelligent Prompt Routing is a managed version of the same idea inside one cloud platform. AWS describes it as a single serverless endpoint that analyzes prompts, predicts response quality, and routes requests between models in the same family to optimize quality and cost; AWS also markets cost reductions of up to 30% without compromising accuracy. ([AWS Bedrock Intelligent Prompt Routing documentation] docs.aws.amazon.com) ([AWS Bedrock overview] Amazon Web Services, Inc.)

4.5 Anthropic auto-routing in Claude apps

Anthropic’s first-party routing examples are narrower than multi-vendor routers, but they are important because they show how routing can disappear into product UX. Claude Code documents aliases such as sonnet, opus, haiku, best, and opusplan; opusplan uses Opus for planning and then switches to Sonnet for execution. ([Claude Code model configuration documentation] Claude) Claude Code’s docs also state that it may automatically fall back to Sonnet after an Opus usage threshold, and that opusplan uses Opus in plan mode for complex reasoning and architecture while execution mode switches to Sonnet for code generation and implementation. ([Claude Code model configuration documentation] Claude API Docs)

Anthropic’s “Claude Code auto mode” engineering post is not model routing in the usual “choose a model for the user query” sense. It describes routing permission decisions through model-based classifiers, including a fast filter and a deeper classifier when needed. That is still relevant: in mature AI systems, routing often applies to sub-decisions inside the product, not only to the final response model. ([Anthropic Engineering, Claude Code auto mode] Anthropic)

5. Implementation comparison

Implementation	Routing boundary	Candidate set	Optimization knobs	What it teaches
Martian	Model-level, per query	Multiple LLMs	Quality, cost, benchmark-specific performance	Routing can outperform a fixed model only if the router beats the baseline and candidate models are complementary. Vendor benchmark claims need workload replication. ([Martian, “Introducing Martian”] Martian)
NotDiamond	Model recommendation API	User-specified providers/models; pretrained or custom routers	Quality, cost, latency	Routing can be externalized as a control-plane API; custom routers require evaluation data. ([NotDiamond custom router documentation] Not Diamond)
OpenRouter Auto	Gateway-level model selection	Curated model pool; optional allowed models	Prompt complexity, task, capability, model pool constraints	Routing can be hidden behind a unified API; the actual selected model should be logged and returned. ([OpenRouter Auto Router documentation] OpenRouter)
OpenRouter provider routing	Provider endpoint selection	Providers serving a given model, or multiple acceptable models	Price, throughput, latency, availability, ZDR, provider order	Provider routing is as important as model routing when uptime and latency dominate. ([OpenRouter provider routing documentation] OpenRouter)
AWS Bedrock Intelligent Prompt Routing	Managed cloud endpoint	Models within a model family	Quality and cost	Cloud vendors can package routing as an endpoint-level feature rather than an application library. ([AWS Bedrock documentation] docs.aws.amazon.com)
Claude Code opusplan / fallback	Product-mode routing	Anthropic model modes	Planning vs execution, usage threshold, effort level	First-party apps may replace explicit model selection with mode-specific routing and effort controls. ([Claude Code model configuration documentation] Claude API Docs)

6. Algorithmic approaches

6.1 Rules-based routing

Rules-based routing is the baseline every production system should understand before deploying learned routers. It encodes hard constraints and obvious preferences: context length, tool availability, structured-output support, data-retention policy, provider allowlist, user tier, task type, and fallback priority.

Rules are appropriate when:

Rule type	Example
Capability rule	Do not route tool-calling prompts to a model that lacks tool support.
Context rule	Exclude models whose context window is too small.
Policy rule	Enterprise customer requests must use zero-data-retention providers.
Cost rule	Use a cheap model for deterministic extraction unless confidence is low.
Latency rule	For interactive UI completions, exclude endpoints above a p90 latency threshold.
Availability rule	Fail over when a provider returns rate-limit or downtime errors.

OpenRouter’s provider-routing interface is effectively a rich rules and preferences layer: it can set provider order, allow fallbacks, require parameters, restrict data collection, sort by price or latency, and apply maximum-price constraints. ([OpenRouter provider routing documentation] OpenRouter) This is not “AI” in the glamorous sense, but it is often where most production value appears.

Rules fail when the distinction is semantic: “Is this math problem hard enough for the expensive model?” “Will the cheaper model miss the subtle instruction?” “Is this legal question a simple summary or actual legal reasoning?” Those cases motivate learned routers.

6.2 Learned classifiers

A learned classifier predicts which model should answer a prompt. The most common academic setup is binary: route between a weaker, cheaper model and a stronger, more expensive model. RouteLLM formulates this as a win-prediction problem over preference data: given a prompt, estimate whether the strong model is likely to beat the weak model, then route to the strong model only when the predicted advantage crosses a threshold. ([RouteLLM paper] arXiv)

This approach has a clean Cost-Quality Pareto Frontier. Lower thresholds send more traffic to the strong model and improve quality at higher cost; higher thresholds send more traffic to the weak model and reduce cost at higher quality risk. RouteLLM evaluates multiple router types, including similarity-weighted ranking, matrix factorization, BERT-based classifiers, and causal-LLM classifiers. ([RouteLLM paper] arXiv)

Learned classifiers work best when the training data matches production traffic. RouteLLM reports that routers trained only on Chatbot Arena data can perform poorly out of distribution, and that data augmentation helps on benchmarks such as MMLU. ([RouteLLM paper] arXiv) That is the central practical lesson: a routing classifier is an evaluation model, and evaluation models can overfit just like generation models.

6.3 Embedding-similarity and k-nearest-neighbor routers

Embedding-based routers retrieve similar past prompts and infer which model should handle the new prompt from historical outcomes. RouteLLM includes a similarity-weighted ranking method that embeds the incoming prompt, computes cosine similarity against prior prompts, and uses local preference data to decide which model should answer. ([RouteLLM paper] arXiv)

The appeal is sample efficiency and interpretability. If similar previous prompts were reliably answered by the cheap model, the router can choose the cheap model. If similar prompts required the expensive model, it can escalate. Recent routing studies have found that well-tuned KNN-style methods can match or outperform more complex learned routers across diverse tasks, which is a warning against assuming that deeper router architectures automatically win. ([Rethinking Predictive Modeling for LLM Routing] arXiv)

Embedding routers fail when “semantic similarity” does not align with “model difficulty.” Two prompts can look close in embedding space while differing sharply in required precision, hidden constraints, or adversarial structure. Conversely, two prompts can look different but require the same cheap deterministic extraction strategy.

6.4 Cost-quality Pareto routing

The Cost-Quality Pareto Frontier is the right abstraction for most routing decisions. A router is useful if it produces a better cost-quality point than any single model baseline. RouterBench explicitly uses routing curves and compares routers against Zero Router and Oracle Router baselines; the Zero Router asks whether routing beats the best fixed default, while the Oracle Router estimates the unattainable upper bound if the system knew the best model for each query. ([Martian RouterBench] Martian) ([Martian RouterBench] Martian)

Cost-quality routing should not optimize average quality alone. A product may care about tail failures, hallucination rate, escalation cost, user trust, p95 latency, or “catastrophic wrong answer” frequency. A router that saves 60% on average cost but occasionally sends high-risk legal, medical, or security prompts to an underpowered model may be unacceptable.

A practical scoring function often looks like this:

score(model, request) =    expected_quality(model, request)  - λ_cost * expected_cost(model, request)  - λ_latency * expected_latency(model, request)  - λ_risk * expected_policy_risk(model, request)

The coefficients are product decisions, not purely technical constants. The same organization may use different coefficients for internal summarization, customer-facing support, coding agents, regulated workflows, and creative drafting.

6.5 Cascading

Model Cascading is routing with sequential escalation. Instead of predicting the model before generation, the system asks a cheap model first, evaluates the result, and escalates only if confidence or quality is insufficient. FrugalGPT popularized this framing for LLM APIs, combining prompt adaptation, approximation, and cascades to reduce costs; it reported matching GPT-4-level performance with up to 98% cost reduction or improving accuracy by 4% at the same cost on some benchmark settings. ([FrugalGPT paper] arXiv)

AutoMix is another cascade-style approach: it obtains an answer from a smaller model, performs self-verification or confidence assessment, and routes to a larger model when needed. ([AutoMix paper] arXiv) A unified routing-and-cascading paper distinguishes routing, where one model is selected initially, from cascading, where increasingly capable models are tried until the output is satisfactory; it argues that combining both can outperform either alone when quality estimators are strong. ([Unified routing and cascading paper] arXiv)

Cascades trade lower expected cost for higher worst-case latency. If the cheap model is often good enough, cost drops. If many requests escalate, the user pays for multiple calls and waits for multiple generations. RouteLLM emphasizes that predictive routing avoids the extra LLM calls used by response-based cascades, which can reduce latency relative to cascade systems. ([RouteLLM paper] arXiv)

6.6 Overgenerate-and-rerank

Overgenerate-and-rerank asks multiple models to answer, then uses a judge or ranker to pick the best response. RouterBench treats this as a non-predictive routing method and notes that it can serve as a high-cost upper-bound style baseline. ([Martian RouterBench] Martian)

This method can improve quality when the judge is reliable and the application can afford the cost and latency. It is usually inappropriate for high-volume interactive systems unless used sparingly, because it multiplies inference cost by the number of candidate models.

6.7 Latency-aware and provider-aware routing

In real production systems, “which model?” is often less important than “which endpoint for this model right now?” Provider-aware routing selects among providers based on recent latency, throughput, availability, price, and policy. OpenRouter’s provider-routing docs describe sorting by price, throughput, or latency and tracking provider latency and throughput percentiles over recent windows. ([OpenRouter provider routing documentation] OpenRouter)

Latency-aware routing research is also emerging. The IPR paper reports an industrial prompt router deployed on a major cloud platform, with user-controlled quality-cost tradeoffs, 1.5 million prompts across 11 LLM candidates, 43.9% cost reduction at quality parity with the strongest Claude-family model, and sub-150 ms router latency. ([IPR paper] arXiv) SCORE frames routing as maximizing generation quality under both cost and latency constraints, including prediction of model quality and response length. ([SCORE paper] Minlan Yu)

The practical lesson is that model routing and load balancing converge. A router that ignores live endpoint behavior may choose the “right” model but send the request to a slow, overloaded, or failing provider.

7. Empirical evidence: when routing improves cost-quality

The strongest evidence for model routing comes from benchmark settings where three conditions hold: candidate models have different strengths, the router has useful historical/evaluation data, and the workload contains a mix of easy and hard prompts.

RouteLLM reports that its routers can reduce cost while retaining a large fraction of GPT-4 quality. The LMSYS blog summarizes results as cost reductions above 85% on MT-Bench, 45% on MMLU, and 35% on GSM8K while achieving 95% of GPT-4 performance. ([RouteLLM LMSYS blog] LMSYS Org) The paper’s cost table reports savings such as 3.66× on MT-Bench at 95% GPT-4 quality, with smaller savings on MMLU and GSM8K, and finds that routing overhead is small relative to expensive generation. ([RouteLLM paper] arXiv)

FrugalGPT provides an earlier cascade-based result: by learning when to use cheaper or more expensive APIs, it reports large cost reductions or accuracy improvements at fixed cost. ([FrugalGPT paper] arXiv) Martian’s RouterBench argues that routers should be evaluated as routing curves rather than single numbers, because the relevant question is how quality changes as cost changes. ([Martian RouterBench] Martian)

Recent benchmark studies complicate the optimism. LLMRouterBench, a 2026 benchmark with more than 400,000 instances from 21 datasets and 33 models, reports strong model complementarity but also finds that many routing methods perform similarly, that some recent approaches including commercial routers fail to reliably beat simple baselines, and that the gap to Oracle routing is largely driven by model-recall failures. ([LLMRouterBench paper] arXiv) A separate study on past-performance retrieval finds that routing performance-cost curves often lie above the Pareto front of individual LLMs, but also that simple KNN methods are hard to beat in many benchmark settings. ([Generalising LLM Routing Using Past Performance Retrieval] ACL Anthology)

So the honest synthesis is:

Evidence claim	Status
Routing can reduce cost at near-constant benchmark quality.	Strong evidence in RouteLLM, FrugalGPT, RouterBench-style evaluations, and industrial reports.
Routing beats the best single model in all domains.	False. The router must beat fixed-model baselines, and many routers do not.
More complex routers reliably beat simple KNN or rules.	Contested. Recent studies find simple methods competitive.
Commercial routing claims generalize to arbitrary workloads.	Thin. Vendor claims should be reproduced on the actual traffic distribution.
Routing overhead is always negligible.	False. Predictive routers can be cheap, but cascades, overgenerate-and-rerank, and fallback retries can add visible latency.
Routing works best with in-domain eval data.	Strong. Out-of-distribution routing is a central failure mode.

8. When routing adds latency or degrades quality

Routing can fail in at least six ways.

8.1 The router is worse than the baseline

The simplest failure is that the router sends too many hard prompts to weak models or too many easy prompts to expensive models. RouterBench’s Zero Router baseline is useful because it asks whether the routing system actually improves on a fixed choice. ([Martian RouterBench] Martian) LLMRouterBench’s finding that some commercial and recent methods fail to reliably beat simple baselines is the strongest warning here. ([LLMRouterBench paper] arXiv)

8.2 The router lacks model recall

A router can only choose among models it recognizes as good candidates. LLMRouterBench attributes much of the gap to Oracle routing to model-recall failures: the router does not reliably include or select the model that would have performed best. ([LLMRouterBench paper] arXiv) This matters in rapidly changing model ecosystems, where new model releases can invalidate old routing preferences.

8.3 The workload distribution shifts

A router trained on chatbot preference data may not work for enterprise legal extraction, code review, spreadsheet analysis, or long-context retrieval. RouteLLM explicitly lists real-world distribution mismatch as a limitation and shows that training data choice affects benchmark transfer. ([RouteLLM paper] arXiv)

8.4 The system uses a cascade when a predictive router would suffice

Cascades can be efficient when most cheap outputs pass verification. They are inefficient when most requests escalate, because the system pays for the cheap attempt and the expensive attempt. RouteLLM’s critique of response-based cascades is exactly that multiple LLM queries increase latency, while predictive routing can decide before generation. ([RouteLLM paper] arXiv)

8.5 Fallbacks hide failures while increasing tail latency

Fallbacks are necessary for reliability, but they are not free. OpenRouter’s latency documentation states that fallback requests can add latency when a completion fails, even though the system tracks provider failures to route around unavailable providers. ([OpenRouter latency documentation] OpenRouter) In user-facing systems, fallback latency can matter more than mean cost savings.

8.6 The router optimizes the wrong metric

A router optimized for average benchmark score may underweight safety, refusal quality, privacy, tool reliability, long-tail correctness, or user trust. Routing is especially risky when the application has asymmetric error costs: a wrong medical triage answer, security recommendation, or financial instruction is not equivalent to a slightly worse creative-writing answer.

9. Relationship to model cascading

Model routing and model cascading are adjacent but not identical.

Pattern	Decision timing	Calls made	Main benefit	Main failure mode
Predictive model routing	Before generation	One model call	Low latency, clean cost control	Router may misclassify difficulty.
Model cascading	After cheap model response or confidence estimate	One or more model calls	Cheap path for easy cases; escalation for hard cases	Escalated cases pay extra latency and cost.
Overgenerate-and-rerank	After multiple model responses	Many model calls	High quality if judge is reliable	Very expensive; judge may be wrong.
Fallback routing	After provider/model failure	One or more model calls	Reliability	Tail latency; inconsistent outputs.

FrugalGPT is the canonical cascade reference for LLM APIs, while RouteLLM is the canonical predictive routing reference. ([FrugalGPT paper] arXiv) ([RouteLLM paper] arXiv) The “Unified Approach to Routing and Cascading” paper argues that routing and cascading are best seen as related policies over model calls, not separate religions; the right design depends on quality estimators, latency constraints, and cost constraints. ([Unified routing and cascading paper] arXiv)

A good production system often uses both:

1. Predictively route obvious easy requests to cheap model.2. Predictively route obvious hard/high-risk requests to strong model.3. For ambiguous mid-band requests, use cheap model plus verifier.4. Escalate only when verifier confidence is low.5. Log every decision for offline replay.

The key is to know which layer is making which promise. The router predicts difficulty; the cascade evaluates an actual output; the fallback system handles infrastructure failure.

10. Relationship to mixture-of-experts

Mixture of Experts is routing inside the model rather than routing among external models. In a sparse MoE layer, a learned gate sends an input representation, often at token or sequence level, to a small subset of expert subnetworks. The original Sparsely-Gated MoE work describes a trainable gating network that determines a sparse combination of experts, allowing very large model capacity with relatively modest computation per token. ([Sparsely-Gated Mixture-of-Experts paper] arXiv) Switch Transformer simplified this idea by routing tokens to experts and reported major pretraining speedups relative to dense baselines. ([Switch Transformer paper] jmlr.org)

The analogy is useful but limited:

Dimension	Model-level routing	Mixture-of-experts routing
Routing unit	Request, conversation, tool call, or subtask	Token, hidden state, sequence, or layer activation
Router location	Application/gateway/control plane	Inside model architecture
Candidate set	Distinct LLMs or provider endpoints	Expert subnetworks
Observability	Application can log selected model	Internal routing often opaque to application
Optimization target	Product cost, latency, quality, reliability, policy	Training/inference efficiency and model capacity
Failure mode	Wrong model selection	Expert imbalance, undertrained experts, routing collapse

Seen abstractly, both are routing problems: allocate computation to the expert most likely to help. But application-level routing is constrained by API prices, data policy, provider availability, and user experience. MoE routing is constrained by training dynamics, hardware efficiency, and load balancing.

The open philosophical point is that model routing may be a coarse-grained version of a more general Conditional Computation principle: spend more computation only where it is expected to buy more value.

11. Routing and effort dials

A major alternative to external model routing is a unified model with an effort or reasoning dial. Instead of choosing between Model A and Model B, the application calls one model and sets something like reasoning.effort = low | medium | high.

OpenAI’s reasoning-model documentation describes internal reasoning tokens, reasoning.effort levels from minimal through high and xhigh, and a cost/quality/latency tradeoff where lower effort can be faster and cheaper while higher effort can improve complex reasoning. ([OpenAI reasoning-model documentation] OpenAI Developers) OpenAI’s GPT-5 system card describes GPT-5 as a unified system with a fast model, a deeper reasoning model, and a real-time router that decides based on conversation type, complexity, tool needs, and user intent. ([OpenAI GPT-5 system card] OpenAI) Anthropic’s Claude Code docs similarly document effort levels for adaptive reasoning and mode-level routing such as opusplan. ([Claude Code model configuration documentation] Claude API Docs)

This changes the routing question. The application may no longer ask, “Which vendor model should I use?” It may ask:

Should this request use:  - low effort?  - medium effort?  - high effort?  - a specialized code mode?  - a fallback provider?  - an external model anyway?

Effort dials do not eliminate routing; they move routing inside the vendor’s product boundary. The user sees fewer explicit model choices, while the vendor’s hidden router decides how much computation to spend. That may simplify consumer UX but reduce application-level control and auditability.

12. Observability requirements

A model router without observability is dangerous because it can silently shift quality, cost, and policy behavior. The minimum production log for a routed request should include:

Field	Why it matters
request_id / session_id	Needed for replay and attribution.
User tier / tenant	Routing decisions may differ by contract or budget.
Prompt features	Task type, length, language, tool needs, risk class.
Candidate set	Shows what the router could have selected.
Excluded models and reasons	Reveals policy, capability, context, or provider constraints.
Router score per candidate	Enables debugging and threshold tuning.
Selected model/provider	Essential for cost, quality, and incident analysis.
Fallback attempts	Shows whether reliability path changed the answer.
Latency and token usage	Needed for cost-quality-latency curves.
Evaluation label or user feedback	Needed to improve the router.
Router version	Required for regression analysis.

OpenRouter’s docs explicitly return the routed model in the response for Auto Router, and its fallback docs state that the response is priced using the model ultimately used. ([OpenRouter Auto Router documentation] OpenRouter) ([OpenRouter fallback documentation] OpenRouter) NotDiamond returns a selected provider/model and session ID, which serves a similar audit function. ([NotDiamond chat router documentation] Not Diamond)

The serious version of observability is offline replay. Save a stratified sample of production prompts, run them across candidate models, judge outputs with human labels or reliable evals, and compare the router against fixed baselines. Without replay, teams tend to mistake lower invoice totals for successful routing.

13. Active critiques

13.1 Routing-quality bottleneck

The router can become the weakest model in the system. If it mispredicts task difficulty, the final answer quality is bounded by that mistake. LLMRouterBench’s finding that many routers fail to outperform simple baselines, and that model-recall failures drive much of the gap to Oracle routing, is the clearest empirical version of this critique. ([LLMRouterBench paper] arXiv)

The bottleneck is sharper in high-stakes domains. A router that sends “apparently simple” legal, medical, or security questions to a weak model may produce plausible but dangerous answers. The correct route may be a stronger model, a restricted workflow, or human review, not simply another LLM.

13.2 Vendor lock-in

Routers can reduce lock-in by abstracting over providers, but they can also create a new lock-in layer. A gateway provider may maintain the routing policy, performance metrics, provider integrations, fallback logic, and billing abstraction. Once the application depends on those routing decisions, switching away can be as hard as switching model vendors.

OpenRouter and NotDiamond both expose mechanisms that mitigate this: OpenRouter allows provider restrictions, provider ordering, maximum price, and data-policy controls, while NotDiamond’s docs emphasize choosing a model and letting the application call that provider through its own SDK. ([OpenRouter provider routing documentation] OpenRouter) ([NotDiamond chat router documentation] Not Diamond) But the strategic risk remains: if the router is a black box, the application may not know why quality changed.

13.3 Observability and auditability

Model routing can make incident analysis harder. When a bad answer appears, the team must ask not only “Why did the model answer this way?” but also “Why did the router choose that model?” and “Would another model have behaved differently?” This requires model provenance, router scores, fallback logs, and evaluation replay.

OpenRouter’s provider latency docs show how dynamic endpoint routing can depend on recent provider performance. ([OpenRouter latency documentation] OpenRouter) That is useful, but it also means a request at 10:00 may route differently from a similar request at 10:05 because live infrastructure conditions changed. Dynamic routing needs versioned decision records.

13.4 Benchmark overfitting

Routing benchmarks are vulnerable to the same problem as model benchmarks: routers can look strong on public tasks while failing on product-specific traffic. RouteLLM’s limitations section explicitly notes real-world distribution differences, and RouterBench-style evaluation distinguishes Oracle, Zero Router, and learned routers precisely because single aggregate scores can mislead. ([RouteLLM paper] arXiv) ([Martian RouterBench] Martian)

A robust router should be evaluated on the application’s actual prompt distribution, not only on MT-Bench, MMLU, GSM8K, or Arena-style preference data.

13.5 Security and adversarial routing

A router creates a new attack surface. If an adversary can phrase a request so that it is routed to a cheaper, weaker, less safety-aligned, or less tool-aware model, the router becomes part of the exploit path. Recent work on “rerouting LLM routers” studies adversarial robustness of LLM routers as control-plane components that balance quality and cost. ([Rerouting LLM Routers paper] arXiv)

The safe default is to treat routing decisions as policy decisions. Sensitive domains should be excluded from cheap routes unless there is strong evidence that the cheaper route satisfies the same safety and correctness requirements.

14. Design guidance

14.1 Use routing when the traffic has separable difficulty bands

Routing is most useful when you can draw meaningful boundaries:

Traffic band	Typical route
Deterministic extraction	Cheap structured-output model or tool.
Simple summarization	Cheap or mid-tier model.
Ambiguous user intent	Mid-tier model or classifier plus clarifying step.
Complex reasoning	Strong reasoning model or high effort.
Code architecture	Strong code/reasoning model.
Routine code edits	Mid-tier code model.
High-risk regulated query	Strong model plus policy guardrail, or human review.
Provider failure	Fallback model/provider.

If every request is equally hard, route less. If every request is equally easy, route less. Routing helps when the distribution is mixed.

14.2 Start with rules and baselines

Before training a learned router, build a fixed baseline and rules baseline:

Baseline A: Always use cheapest acceptable model.Baseline B: Always use strongest model.Baseline C: Rules-based router.Baseline D: Learned router.Baseline E: Oracle estimate from offline labels.

The learned router must beat the rules baseline, not just the expensive-model baseline. RouterBench’s Zero Router and Oracle Router framing is useful because it prevents teams from celebrating a router that is worse than a simple fixed policy. ([Martian RouterBench] Martian)

14.3 Build the evaluation set before the router

A custom router needs labeled examples: prompts, model outputs, and quality judgments. NotDiamond’s custom-router docs make this explicit by requiring inputs, responses, and evaluation scores. ([NotDiamond custom router documentation] Not Diamond)

A good routing eval set should include:

Slice	Why
Easy, medium, hard prompts	Tests whether the router finds difficulty bands.
Long-context prompts	Tests context-window filtering.
Tool-use prompts	Tests capability filtering.
Safety-sensitive prompts	Tests policy routing.
Multilingual prompts	Tests language-specific model strengths.
Adversarial prompts	Tests whether users can force weak routes.
Recent production prompts	Tests distribution drift.
Human-labeled failures	Tests whether the router learns from real incidents.

The eval should report not only mean quality and cost, but also tail failure rate, escalation rate, p95 latency, and cost per successful task.

14.4 Use thresholds, not vibes

A binary strong/weak router needs a threshold. RouteLLM’s formalism uses a cost threshold to decide when the strong model’s expected advantage justifies its cost. ([RouteLLM paper] arXiv) Product teams should set thresholds by policy:

Route to strong model if:  P(strong beats weak | prompt) * value_of_quality_gain  >  incremental_cost + latency_penalty + policy_risk

In low-risk internal tools, the threshold may favor savings. In customer-facing or high-risk settings, it should favor quality and safety.

14.5 Version the router like a model

A router is itself a model or policy artifact. It needs versioning, regression tests, rollback, and change management. This is especially important when candidate models change. A new cheap model can invalidate old routing thresholds; a price drop can change the Pareto frontier; a provider outage can make yesterday’s best route unavailable.

14.6 Separate hard constraints from soft preferences

Do not let a learned router override hard constraints. Data-retention policy, context length, tool availability, jurisdiction, and safety restrictions should filter the candidate set before scoring. Learned scoring should operate only over valid candidates.

A robust selection pipeline is:

def route(request):    features = analyze(request)    candidates = all_models()    candidates = [        m for m in candidates        if m.context_window >= features.required_context        and m.supports_required_tools(features.tools)        and m.allowed_for_tenant(request.tenant)        and m.allowed_for_data_policy(request.data_policy)        and m.price <= request.max_price    ]    if not candidates:        return escalate_or_fail_closed(request)    scores = router.predict(request, candidates)    selected = argmax(        scores[m].quality        - cost_weight(request) * scores[m].cost        - latency_weight(request) * scores[m].latency        - risk_weight(request) * scores[m].risk        for m in candidates    )    response = call_with_fallbacks(selected, request)    log_route_decision(        request=request,        features=features,        candidates=candidates,        scores=scores,        selected=selected,        response=response,        router_version=router.version,    )    return response

The important detail is fail_closed: if no model satisfies hard constraints, the system should not silently route to an invalid model.

15. The open infrastructure question

The unsettled question is whether external model-level routing becomes standard infrastructure or recedes as model vendors ship unified systems with internal routers and effort dials.

Case for model routing becoming standard infrastructure

Model routing is likely to persist wherever applications need multi-provider resilience, cost control, compliance policy, model specialization, and independent evaluation. OpenRouter’s gateway model, NotDiamond’s router API, AWS Bedrock’s managed prompt routing, and Martian’s RouterBench framing all point toward routing as a normal part of the LLM stack. ([OpenRouter quickstart] OpenRouter) ([NotDiamond, “What is Model Routing?”] Not Diamond) ([AWS Bedrock documentation] docs.aws.amazon.com) ([Martian RouterBench] Martian)

The infrastructure argument is strongest for enterprises. Enterprises care about provider redundancy, data policy, audit logs, jurisdiction, procurement, cost ceilings, and differentiated workloads. Those needs are not solved by one model becoming better.

Case for model routing receding

External routing may recede in consumer products if frontier vendors hide the decision behind unified model systems. OpenAI’s GPT-5 system card describes a unified system with a real-time router choosing between fast and deeper reasoning paths, and OpenAI’s docs expose effort controls for reasoning models. ([OpenAI GPT-5 system card] OpenAI) ([OpenAI reasoning-model documentation] OpenAI Developers) Anthropic’s Claude Code similarly exposes mode aliases, adaptive effort, automatic fallback, and opusplan rather than asking every user to manually select a model for every step. ([Claude Code model configuration documentation] Claude API Docs)

If one vendor’s unified model family provides adequate cost, latency, quality, and policy controls, many application developers will prefer a simpler API. The router still exists, but it is vendor-internal rather than application-level.

Most likely outcome

The most likely outcome is bifurcation:

Context	Likely routing future
Consumer chat apps	Hidden internal routing and effort dials.
Enterprise AI platforms	Explicit model/provider routing with audit logs.
Coding agents	Hybrid routing: planning model, execution model, tool-specific models, fallbacks.
Regulated workflows	Conservative routing with hard policy filters and human escalation.
Research/eval platforms	Open routing benchmarks and model-selection experiments.
Low-cost commodity apps	Simple rules or cheapest-acceptable model.

Model routing will not disappear. It will become less visible in some products and more formalized in others. The durable abstraction is not “choose between model names”; it is Adaptive Computation Allocation: spend the right amount of model capability, from the right provider, under the right constraints, for this request.

16. Practical summary

A model router is worth building when:

Use routing when…	Avoid or delay routing when…
Traffic contains easy and hard prompts.	Workload is narrow and uniform.
Candidate models are complementary.	Candidate models are near-equivalent.
Model prices differ materially.	Cost difference is small relative to engineering complexity.
You have evaluation data.	You cannot measure quality.
Latency budget allows routing overhead.	Every millisecond matters and no predictive router is available.
Compliance and provider policy vary by request.	One provider/model satisfies all policy constraints.
You can log and replay decisions.	You cannot audit which model answered.
Fallbacks are needed for uptime.	Reliability is already handled elsewhere.

The disciplined implementation path is:

1. Define hard constraints.

2. Build fixed-model baselines.

3. Build a rules router.

4. Collect in-domain eval data.

5. Train or tune a predictive router.

6. Compare against Zero Router, rules baseline, and Oracle estimate.

7. Deploy with conservative thresholds.

8. Log every decision.

9. Replay production traffic.

10. Update thresholds as model prices, capabilities, and workloads change.

The undisciplined path is to buy or train a router, trust its average benchmark score, and stop looking. That path usually produces hidden quality regressions.

Companion entries

Core theory: LLM Control Plane, Adaptive Computation Allocation, Conditional Computation, Cost-Quality Pareto Frontier, Model Cascading, Mixture of Experts, Effort Dials, Oracle Router, Zero Router

Implementation patterns: LLM Gateway, Provider Routing, Fallbacks and Circuit Breakers, Rules-Based Routing, Embedding Similarity Routing, Learned Router Classifiers, Overgenerate and Rerank, Latency-Aware Inference

Evaluation and operations: LLM Evaluation, Inference Cost Accounting, Observability for AI Systems, Offline Replay Evaluation, Router Versioning, Distribution Shift in LLM Systems, Adversarial Prompting

Canonical systems: Martian, NotDiamond, OpenRouter, AWS Bedrock Intelligent Prompt Routing, Claude Code, RouteLLM, FrugalGPT, RouterBench, LLMRouterBench

Counterarguments and open questions: Unified Reasoning Models, Vendor Lock-In, Router Robustness, Benchmark Overfitting, Goodhart’s Law in AI Evaluation, Single Model vs Model Portfolio