Small models are quietly eating the inference bill

Key takeaways

• —Most production traffic is classification, extraction and routing — work that does not need a frontier model.
• —Routing cheap requests to a small model typically removes 60-80% of token spend before any prompt optimization.
• —The failure mode is silent quality drift, which is why routing needs evaluation before it needs cost analysis.

Every team that has run a large language model in production for more than a quarter arrives at the same discovery, usually via the finance department: the model you evaluated on is not the model your traffic needs.

Evaluation happens on the hard cases. Production is mostly easy cases. The gap between those two facts is where most inference budgets are quietly wasted.

What production traffic actually looks like

Instrument a mature LLM feature and the distribution is consistently lopsided. A rough shape, drawn from what teams report and what shows up in trace data:

Request type	Share of volume	Needs frontier capability?
Classification and routing	30–40%	No
Structured extraction	20–30%	Rarely
Short-form generation	15–25%	Sometimes
Long-form reasoning	5–15%	Yes

The bottom row is the reason you chose the model. The top two rows are the reason your bill is what it is.

Routing, and the order to do it in

The obvious move is a router: classify the incoming request, send the easy ones somewhere cheap. The non-obvious part is that this is dangerous to do cost-first.

A router that mis-routes 5% of hard requests to a small model does not fail loudly. It produces plausible, slightly worse output on a slice of traffic, and nobody notices until a customer does. Cost dashboards improve. Quality dashboards, if they exist at all, move within noise.

So the sequence matters:

1. Build the evaluation set first. Fifty to two hundred real requests per category, with known-good outputs. Without this, everything downstream is guesswork.
2. Measure the small model on that set before routing anything to it. You are looking for categories where it matches, not where it is close.
3. Route only the categories that matched. Leave everything else alone.
4. Then look at the bill.

Teams that run this in order typically find 60–80% of token volume can move without a measurable quality change. Teams that run it cost-first find the same number and then spend a quarter discovering which parts of it were a mistake.

Where the savings actually come from

Three places, in descending order of size:

Model choice. The largest single lever, and the one above. Order-of-magnitude differences in per-token price between tiers mean this dominates everything else.

Prompt length. Enormous system prompts get sent on every request. Caching helps substantially where it is available, but the cheapest token remains the one not sent. Auditing a system prompt that has accreted for a year usually removes a third of it with no behavior change.

Output length. Under-specified output constraints produce models that pad. max_tokens plus an explicit format instruction is a rounding error to implement and routinely cuts generation cost by a quarter.

The part that is genuinely hard

None of the above is difficult. The hard part is that a routed system has more failure modes than a single-model system, and most teams do not have the observability to see them.

You need per-category quality tracking, not aggregate. You need to know when the input distribution shifts, because a router calibrated on last quarter's traffic degrades silently as usage patterns move. And you need someone to own the evaluation set, because an evaluation set nobody maintains stops describing production within a few months.

That is the real cost of the 70% saving, and it is worth being honest that it is an ongoing engineering cost rather than a one-off migration. For most teams at any scale it is still comfortably worth it. For a team running a few thousand requests a day, it may genuinely not be — and that is a legitimate answer rather than a failure of ambition.