Routing Requests Between a Cheap and a Capable Model

The situation

A product team runs a customer-facing assistant on Amazon Bedrock. Every request goes to one large capable model, and the bill has grown faster than usage because the model is priced for its hardest job and asked to do its easiest one thousands of times a day. The traffic is a mix. A large share is simple: classify an incoming message into one of a handful of intents, pull an order number out of a sentence, answer a factual question the knowledge base already contains. A smaller share is genuinely hard: reconcile a multi-part complaint, work through a returns policy with three conditions, plan a sequence of steps and explain the reasoning.

When the team samples the logs, the split is roughly eighty-twenty. Four in five requests are the kind a small model answers correctly and in a fraction of the time, at a fraction of the per-token price. One in five actually exercises the large model’s reasoning. Running everything through the flagship means the eighty per cent subsidises the twenty, both in cost and in the extra latency the big model carries on requests that never needed it.

The obvious move is to send easy requests to a cheap small model and hard ones to a large capable model. The obvious risk is getting the split wrong: route a hard request to the weak model and you get an answer that is fast, cheap, and wrong, delivered with the same confidence as a right one. The decision is how to classify each request, how much a misroute costs, and whether to build the router or let Bedrock run it.

What actually matters

The first thing to name is that model size is a cost lever, not a quality dial you turn up for safety. A bigger model is not uniformly better at everything; it is better at the tasks that need what it has, which is depth of reasoning and breadth of knowledge. On a single-step classification, a well-chosen small model matches it and answers quicker. So the question is never “is the big model better” in the abstract, it is “does this particular request need what the big model has”. Where the answer is no, the flagship is paying for capability the request will not use.

The shape of the workload decides how much routing can save. If almost every request is hard, there is little to route away and the machinery earns nothing. If the traffic is genuinely mixed, with a large easy majority and a smaller hard tail, routing captures most of the flagship’s cost as savings on the majority while preserving quality on the tail. The eighty-twenty split is the case routing is built for; a workload that is uniformly hard, or uniformly trivial, does not need a router at all, it needs the one model that fits.

The cost of a misroute is asymmetric and it runs one way harder than the other. Send an easy request to the big model and you overpay by a few tokens and a little latency; the answer is still correct. Send a hard request to the small model and you can get a wrong answer that reads as authoritative, which reaches the customer and costs far more than the tokens you saved. The escalation threshold has to be set with that asymmetry in mind: it is cheaper to occasionally send a borderline-easy request to the big model than to occasionally send a hard one to the small model. When in doubt, route up.

That means the thing you actually have to measure is quality per route, not quality in aggregate. An overall accuracy number hides the failure that matters, because the misroutes are a minority inside a mostly-correct stream. You want to know how often the small model was handed something it got wrong, which means sampling the requests that went to the cheap path and checking them against what the capable model would have produced. Without per-route measurement you cannot tell a healthy router from one that is quietly degrading answers to save money.

And routing is not the only cost lever, so it should not be the only one you pull. A cache in front of the whole thing removes the repeated identical and near-identical requests before either model runs; trimming a bloated prompt cuts the per-call cost on both paths. Routing decides which model a request reaches; caching and prompt trimming decide whether it needs a model at all and how much it costs when it does. They compound, and the cheapest request is the one the cache answers.

What we’ll filter on

1. Workload variety, is the traffic a genuine mix of easy and hard, or mostly one kind?
2. Misroute cost, how bad is a wrong answer from the weak model, and how asymmetric is it against overpaying on the strong one?
3. Classification signal, can the request’s difficulty be told from cheap signals (task type, length, a small classifier) or does it need the model to look?
4. Measurability, can quality be measured per route, so a degrading cheap path is visible?
5. Build versus managed, does a managed router within a model family fit, or does the split need custom logic across families?
6. Stacking, does routing sit alongside caching and prompt trimming rather than replacing them?

The routing landscape

A rules or heuristic router classifies each request from cheap signals before any model runs. Route by task type when the caller already knows what it is asking for, so a classification endpoint goes to the small model and a reasoning endpoint goes to the large one. Route by input length as a rough proxy, since short inputs are more often simple and very long ones more often need the bigger context and reasoning, though length is a weak signal on its own. Route on the output of a tiny classifier, a small cheap model or a lightweight text classifier whose only job is to label the request easy or hard. Heuristic routing is transparent, free of an extra model call when it keys on task type or length, and easy to reason about; its weakness is that a hand-written rule cannot see difficulty that is not visible in the surface of the request.

A small-model-decides-then-escalates router runs the cheap model first and promotes to the capable one when the cheap answer looks weak. The small model attempts every request; if it signals low confidence, or a validator finds the answer malformed or failing a check, the request is retried on the large model. This adapts to difficulty the request’s surface does not reveal, because the cheap model has actually attempted the task. The cost is that hard requests pay twice, once for the failed cheap attempt and again for the capable retry, so it wins when the easy majority is large enough that the double-paid tail stays cheap overall, and it depends on having a reliable signal that the cheap answer was inadequate.

Amazon Bedrock Intelligent Prompt Routing is the managed option. It routes each request within a single model family to the member it predicts will meet your quality bar at the lowest cost, so an easy request goes to the smaller cheaper model in the family and a hard one to the larger. You select a router, either one of the Bedrock-provided defaults or a custom router built from two or more models in the same family, and set the response-quality tolerance, how much difference from the strongest model’s response you are willing to accept; a fallback model catches anything the router cannot confidently place. It works through the Converse and InvokeModel APIs by pointing the request at the router instead of a specific model, and Bedrock predicts per request which model will do. AWS reports cost reductions of up to thirty per cent on suitable workloads without a meaningful quality drop, and there is no separate charge for the routing itself; you pay for whichever model actually serves the request. The constraint is that it routes within a family, not across arbitrary models, so it fits when a single family spans the range of capability you need.

Underneath all three is the same measurement discipline. Whichever router you run, you sample the cheap path and check its answers against the capable model, and you tune the threshold from what you find rather than from a guess.

Side by side

Approach	Sees hidden difficulty	Extra call on easy path	Hard requests pay twice	Managed	Tuning surface
Rules by task type	✗	✗	✗	✗	Route table
Rules by input length	✗	✗	✗	✗	Length cut-off
Small classifier router	Partly	✓ (tiny)	✗	✗	Classifier and threshold
Small-model-then-escalate	✓	✗	✓	✗	Confidence and validator
Bedrock Intelligent Prompt Routing	✓	✗	✗	✓	Quality tolerance, fallback

The picks in depth

Start with the workload before touching a router, because the whole case rests on the split. Sample real traffic, sort it into easy and hard by hand, and get the ratio. A genuine eighty-twenty is the sweet spot: a large easy majority to move off the flagship and a real hard tail to protect. If the sample comes back mostly hard, routing saves little and adds a moving part for nothing, and the honest answer is to keep the one model that fits. If it comes back almost all trivial, drop to the small model outright and skip the router. Routing earns its complexity only on a mixed stream.

For a mixed stream where the difficulty is legible from the request itself, a rules or classifier router is the least machinery. When each endpoint already knows its task, a route table by task type is transparent and adds no extra model call: the intent classifier and the extraction job point at the small model, the reasoning endpoint points at the large one. Where task type is not enough, a tiny classifier that labels the request easy or hard gives a cheap signal without running the expensive model first. Length can supplement this as a coarse tiebreak, but do not lean on it alone; a short input can still be a hard reasoning problem and a long one can be a simple extraction from a wall of text.

When difficulty is not visible on the surface, the small-model-decides-then-escalates pattern earns its place, with the escalation threshold set against the asymmetry of a misroute. The cheap model attempts everything; a low-confidence signal or a failed validation promotes the request to the capable model. Because a wrong cheap answer costs more than an unnecessary escalation, tune the threshold to escalate readily: it is better to send some easy requests up than to let hard ones through on the cheap path. The trade is that promoted requests pay for both models, so this pattern depends on the easy majority being large enough that the double-paid tail stays cheap, and on a promotion signal you trust.

Amazon Bedrock Intelligent Prompt Routing is the pick when a single model family spans the capability range and you would rather not build and maintain the router. Point the request at a router instead of a named model, and Bedrock predicts per request which family member will meet your quality tolerance at the lowest cost, falling back to a nominated model when it cannot place the request confidently. You tune one dial, the response-quality tolerance, and set the fallback; there is no separate routing charge, so you pay for the model that actually serves each request, and AWS reports up to thirty per cent savings on suitable workloads. Its boundary is the family: it will not route from one vendor’s small model to another’s large one, so it fits when the family you are on already offers a cheap member and a capable member. When the split you need crosses families, or hangs on business logic Bedrock cannot see, the custom router is the one that fits.

Whichever router runs, the non-negotiable is per-route measurement. Aggregate accuracy hides the misroutes because they are a minority inside a mostly-correct stream, so sample the requests that took the cheap path and check them against what the capable model produces. That sample tells you whether the threshold is set right and catches a router that has started quietly trading quality for cost. Then stack the other levers: a cache in front removes repeated requests before any model runs, and trimming the prompt cuts the per-call cost on both paths. Routing, caching, and trimming are separate cuts at the same bill, and they compound.

A worked example: the support assistant, split three ways

The team samples a day of traffic and sorts it. Roughly sixty per cent is intent classification and short extraction, twenty per cent is factual questions the knowledge base already answers, and twenty per cent is the hard tail of multi-condition policy reasoning. Three shapes, and the flagship was serving all of them.

The classification and extraction slice is legible from the endpoint, so it takes a rules route straight to the small model with no extra call; the task type is the signal. The factual-question slice goes through the cache and the knowledge base first, and only the residue that needs generation reaches the small model, so most of it never pays for a large-model call at all. The hard policy slice is where difficulty hides inside ordinary-looking questions, so it runs small-model-first with escalation: the cheap model attempts the answer, and a validator that checks the policy conditions were all addressed promotes the weak ones to the capable model. The threshold is set to escalate on any unmet condition, because a wrong policy answer reaching a customer costs far more than a spare capable-model call.

After a fortnight the per-route sample tells the story. The cheap path handles the classification and factual slices with accuracy matching the old flagship-only numbers, and the policy path escalates about a third of its requests, which is the tail that genuinely needed reasoning. The flagship now runs on the twenty-odd per cent of traffic that exercises it, the cache absorbs the repeats, and the bill falls by more than half, most of it from moving the easy majority off the big model rather than from any single clever trick.

What’s worth remembering

1. Model size is a cost lever, not a safety dial; a big model is better at what needs its depth and merely more expensive at what does not.
2. Routing pays off on a genuinely mixed workload with a large easy majority and a smaller hard tail; a uniformly hard or uniformly trivial stream does not need a router.
3. The misroute cost is asymmetric: overpaying on an easy request loses a few tokens, but a hard request on the weak model returns a confident wrong answer, so route up when in doubt.
4. Rules and classifier routers are cheapest and most transparent when difficulty is visible from task type, length, or a tiny classifier, but they cannot see difficulty the request’s surface hides.
5. Small-model-then-escalate adapts to hidden difficulty because the cheap model actually attempts the task, at the cost of hard requests paying for both models.
6. Amazon Bedrock Intelligent Prompt Routing routes each request within a single model family to the cheapest member that meets your quality tolerance, with a fallback model and no separate routing charge.
7. Bedrock’s router works through the Converse and InvokeModel APIs by targeting a router instead of a named model, and it routes within a family, not across arbitrary vendors.
8. Measure quality per route, not in aggregate; sample the cheap path against the capable model, because misroutes are a minority hidden inside a mostly-correct stream.
9. Set the escalation or quality threshold from what the sampling shows, tuned toward escalating readily because a wrong cheap answer costs more than a spare capable call.
10. Routing is one lever; put a cache in front to kill repeated requests and trim bloated prompts to cut per-call cost, and the three savings compound.