What Is LLM Routing? Rules, Weights, and Cost-Aware Policies Explained
LLM routing is the process of choosing the model, provider path, or fallback path that should handle a particular AI request. The decision can be fixed in application code, made by explicit rules, calculated from weighted criteria, or delegated to a learned classifier. The point is not to make every request take a different path. It is to make each path intentional, testable, and safe.
If you are asking what is LLM routing, the practical answer is: it is the policy layer between an application request and a model call. A router first rules out options that cannot satisfy the request, then selects an eligible route using the priorities your team set—such as capability, quality, latency, cost, availability, and data policy.
That distinction matters. A cheap model is not automatically a safe default for a customer-facing workflow. A highly capable model is not automatically the right choice for a simple classification job. And a fallback is useful only when it preserves the output, tool, and data contracts your application expects.
Short version: good LLM routing starts with hard constraints, ranks only compatible candidates, records the reason for the choice, and falls back only when the backup is contract-safe.
That is the standard a production answer to what is LLM routing should meet: a decision that an engineer can inspect, test, and change without guessing which hidden rule moved traffic.
What is LLM routing, exactly?
What is LLM routing? It is a decision process that maps an incoming request to the best approved execution path for that request. The path may identify a specific model, a provider, a region, a model group, or a tested fallback.
The router may live in your application, an API gateway, or a managed model-routing service. For example, Amazon Bedrock intelligent prompt routing selects among eligible models within a family based on predicted response quality and cost. Microsoft Foundry’s model router exposes balanced, quality, and cost-oriented routing modes. These are implementations of the same underlying idea: select deliberately rather than treating every prompt as identical.
LLM routing is not the same thing as:
- A load balancer. A load balancer distributes requests among similar servers. An LLM router also considers model capabilities and task outcomes.
- A retry loop. Retrying the same route can resolve a transient error; it does not decide whether another model is valid for the task.
- An unreviewed model swap. Moving a request to a backup that changes JSON shape, tool behavior, privacy boundaries, or answer quality can create a more serious failure than an explicit error.
- Model training. Routing selects a serving path at request time. It does not change model weights.
For a broader look at the boundary that hosts these decisions, see our guide to what an LLM gateway is and when it becomes useful.
How an LLM router makes a decision
The most reliable design separates eligibility from preference. First, exclude options that cannot satisfy the request. Then score the remaining routes against the current traffic class.
1. Apply hard constraints before any score
Hard constraints are non-negotiable. A model that fails one should not remain in the candidate set just because it is cheap or fast.
Typical filters include:
| Constraint | Example question | Why it comes first |
|---|---|---|
| Modality | Does the route accept text, image, audio, or video input? | A text-only endpoint cannot recover a multimodal request. |
| Contract | Does it support the required tool calls, streaming shape, or structured output? | A syntactically valid response can still break the application. |
| Data policy | Is the provider, region, or retention setting approved for this request? | Policy cannot be traded for a lower token price. |
| Capacity | Does the route have enough context, quota, or healthy capacity? | A candidate that will immediately reject the request is not eligible. |
| Product policy | Is this model allowed for this tenant, environment, or experiment? | Makes governance auditable instead of implicit. |
This order prevents a common routing mistake: treating every model as interchangeable. They are not. Compatibility needs to be proved for the exact endpoint and traffic class you intend to route.
2. Score the eligible candidates
After filtering, a route can use a transparent weighted score. One simple pattern is:
route score =
quality_weight × expected_quality
+ latency_weight × (1 - normalized_latency)
+ cost_weight × (1 - normalized_cost)
+ reliability_weight × recent_route_health
This formula is a decision framework, not a universal benchmark. The important part is that the terms are normalized consistently, the weights belong to a traffic class, and the policy can be explained after an incident.
For example, a background tagging job may put more weight on cost and throughput. A customer-facing support agent may put more weight on quality, tool compatibility, and tail latency. A regulated workflow may give data residency an absolute filter rather than a score.
3. Execute, observe, and use a bounded fallback
The router should log the selected route, score inputs, policy version, request ID, latency, cost estimate or usage record, and result status. If the primary route fails, the fallback decision needs its own rules:
- Is the error transient enough for a retry on the same route?
- Is the request idempotent, or could a retry duplicate an external action?
- Does the fallback preserve the output and tool contract?
- Is the fallback allowed to exceed the request’s budget or data boundary?
- If not, should the system queue, degrade gracefully, or return a controlled error instead?
That is why fallback is a routing technique, not a synonym for routing. Our model fallback strategy playbook covers the recovery decisions in more detail.
Six LLM routing techniques to use deliberately
There is no single best router. Start with the simplest technique that can satisfy the workload, then add complexity when your measurements justify it.
1. Static routing
Every request in a workload goes to one named model or endpoint.
Use it when: you are establishing a baseline, need a stable evaluation target, or have a narrow workload with one approved model.
Watch for: a static route can hide the cost of using a high-capability model for trivial work, or the reliability risk of depending on one upstream path.
2. Rule-based routing
Explicit conditions choose a route: send requests with images to a vision-capable path; send requests with tool calls only to models that support the approved tool contract; send internal batch summaries to an economical route.
Use it when: the decision is explainable from request metadata and the rules are few enough to review.
Watch for: rule sprawl. Put rules in a versioned policy file or service instead of scattering provider checks through application code.
3. Capability-based routing
The router starts from a model capability registry rather than prompt text alone. It filters on endpoint family, context, supported modalities, tool behavior, structured-output support, region, and account allowance.
Use it when: an application has several endpoint types or models with materially different contracts.
Watch for: stale catalog data. Treat model IDs, capabilities, and deprecations as operational data that needs regular verification.
4. Weighted cost, quality, latency, and reliability routing
Eligible routes receive a score based on the current objective. This is the common answer to what is LLM routing when a team wants more nuance than a fixed rule but still needs an explainable policy.
Use it when: you can define a measurable acceptance signal and have enough route telemetry to compare candidates.
Watch for: false precision. A score is only as useful as the quality data, cost assumptions, health signals, and weights behind it. Do not imply that a decimal score proves one model is universally better.
5. Classifier-based routing
A small classifier or an LLM-based classifier labels the request—such as simple extraction, code generation, complex reasoning, or safety-sensitive content—then hands it to a predefined policy for that class.
Use it when: prompt intent strongly predicts the appropriate model tier and you can evaluate classifier errors separately from model errors.
Watch for: adding an opaque decision before the model call. Keep a low-confidence path, log the classification, and evaluate the classifier against representative requests.
6. Health-aware fallback routing
The primary path is selected normally, but the router reacts to timeouts, overload, rate limits, or degraded health with a bounded retry, an equivalent backup, a queue, or a controlled failure.
Use it when: availability matters and you have identified which backups are contract-compatible.
Watch for: silently crossing a semantic boundary. A fallback may be appropriate for a retryable text completion but not for a tool-using agent, a state-changing workflow, or an output that must match a strict schema. See our guide to multimodal agent routing for examples where modality changes make fallback rules stricter.
Build a cost-aware policy without becoming quality-blind
Cost-aware does not mean “always select the cheapest model.” It means give cost a defined role after required capability, policy, and contract checks have passed.
A useful pattern is to create traffic classes before setting weights:
| Traffic class | First priority | Common routing approach | Safe fallback posture |
|---|---|---|---|
| Customer-facing assistant | Quality and predictable contract | Capability filter + quality/latency weights | Equivalent approved route only |
| Background extraction | Throughput and unit cost | Rules + cost/latency weights | Retry, queue, or lower-cost approved route |
| Evaluation suite | Reproducibility | Pin the model and policy version | Usually no hidden fallback |
| Tool-using agent | Tool compatibility and idempotency | Capability filter + risk rules | Stop and reconcile on uncertain side effects |
| Media generation | Endpoint and output requirements | Modality/endpoint filters + budget rules | Fail closed unless the backup was tested |
Then compare the number that matters: cost per accepted outcome, not just cost per request. A lower-priced route that causes extra retries, human review, schema repairs, or failed tool calls may cost more per completed task.
Amazon’s routing documentation makes the same practical caveat in a different context: automated routing has limits, including route-specific training data and support boundaries. It recommends ongoing review of performance and cost rather than assuming a routing decision remains optimal forever. Your own policy should be evaluated against your own request mix.
A safe routing policy, in pseudocode
This simplified example keeps the decision auditable. It does not assume that every route can handle every request.
const candidates = catalog
.filter(route => route.isHealthy)
.filter(route => route.supports(request.modality))
.filter(route => route.supportsTools(request.requiredTools))
.filter(route => route.allows(request.dataClass))
.filter(route => route.withinBudget(request.budget));
if (candidates.length === 0) {
return failClosed("No approved route for this request");
}
const primary = candidates
.map(route => ({
route,
score: score(route, request.trafficClass),
}))
.sort((a, b) => b.score - a.score)[0];
const result = await call(primary.route, request);
if (isTransient(result.error) && request.isIdempotent) {
return retryWithinBudget(primary.route, request);
}
if (canUseEquivalentFallback(result.error, request, primary.route)) {
return call(selectApprovedFallback(candidates, primary.route), request);
}
return returnControlledError(result.error);
The policy should emit a route reason such as tool_contract_required, budget_preferred, primary_unhealthy, or no_contract_safe_fallback. That small detail makes the system much easier to debug than a generic “model failed” message.
How to measure whether LLM routing is helping
Routing is not successful because it produced a model name. It is successful when it improves a defined workload outcome without creating unacceptable risk.
Track results by traffic class, policy version, and selected route:
- Accepted-outcome rate: the share of responses that pass the business, schema, or human-review acceptance test.
- Cost per accepted outcome: total model, retry, and remediation cost divided by accepted tasks.
- P50, P95, and P99 latency: segmented by request size and route; an average can hide painful tail latency.
- Fallback recovery rate: how often a fallback produces an acceptable result, not merely an HTTP 200.
- Contract failure rate: invalid structured output, unsupported tools, streaming incompatibility, or missing required fields.
- Route distribution: whether a new policy is unexpectedly sending most traffic to an expensive or degraded path.
- Policy rejection rate: how often no safe candidate exists—often a signal to expand approved capabilities rather than relax a guardrail.
For a practical scorecard, see AI routing API metrics that actually matter. The metrics are especially useful when a routing change appears to lower token cost but reduces accepted outcomes.
When do you need LLM routing?
You may not need a router when one provider, one endpoint, and one stable workload meet your requirements. Pin the model, set clear timeouts and retries, and measure the baseline first.
LLM routing becomes valuable when at least one of these is true:
- Different request types need different capabilities, cost envelopes, or latency objectives.
- You need to keep provider, model, and regional choices behind a policy boundary.
- An outage, quota issue, or deprecation needs a safe response path.
- You want to compare candidates without rewriting every caller.
- You need a record of which route handled a request and why.
If that sounds like your stack, a gateway can make the operational boundary easier to manage. Flatkey’s documentation describes an OpenAI-compatible base URL and a single API key for supported models; callers choose a model ID per request, while the platform documents automatic upstream failover. Start by pinning a model and logging results, then introduce an explicit routing policy for the traffic classes that truly need it. For the next architectural step, read our AI API gateway requirements checklist.
For routing strategies, a policy template, and production code, read our model routing guide.
Frequently asked questions
What is LLM routing versus model fallback?
LLM routing is the broader decision process for selecting an execution path. A model fallback is one recovery action within that process, used only when the primary route fails and an alternate route is approved for the same request contract.
What is LLM routing in a multi-provider API?
In a multi-provider API, LLM routing can select a provider path as well as a model. The policy should verify endpoint compatibility, data policy, model capability, pricing units, health, and fallback behavior before treating two providers as equivalent.
Which LLM routing technique should a new team start with?
Start with static routing plus explicit capability checks and observability. Add rules for clearly different traffic classes. Move to weighted or classifier-based routing only after you have representative tests and a way to measure accepted outcomes.
Can an LLM router choose the cheapest model automatically?
It can, but cost should never bypass hard constraints. Filter for approved capability and contract compatibility first, then optimize price among the eligible routes. Validate the result with cost per accepted outcome, not token price alone.
The practical answer to “what is LLM routing?”
The practical answer to what is LLM routing is not “pick a clever model.” It is “encode the choices your team can defend.” Start with approved capabilities and hard constraints; use weights only for real tradeoffs; keep fallbacks bounded and contract-safe; and record every decision well enough to improve it later.
If you are centralizing model access, use that boundary to run controlled routing experiments instead of scattering provider logic through every service. Learn more in the Flatkey API documentation.
