Small-Model Economics: Why the Frontier Model Isn't Always the Right Call
When a smaller model beats the frontier model in production. Cost, latency and failure-mode trade-offs from applied AI R&D in Australia, finance to voice.
The most expensive default in applied AI is reaching for the biggest model every time. It feels safe. It is rarely the right call once a system is in production and the bill arrives per call rather than per demo.
This is not an argument against frontier models. We use them daily. It is an argument for treating model choice as an engineering decision with a cost function, rather than a reflex.
The cost that matters is per outcome, not per token
Per-token pricing is the number everyone looks at and the one that matters least on its own. What matters is the cost of a completed outcome: a call handled end to end, a document classified correctly, a record extracted and validated.
Take an AI phone-answering system we built for professional services firms. A single inbound call is not one model call. It is a chain: transcribe, detect intent, decide whether to answer or hand off, generate a reply, check the reply against guardrails, log a structured summary. Six to ten inferences per call, hundreds of calls a day per client.
Run every link in that chain on a frontier model and the per-call cost lands somewhere between ten and thirty times what it costs with a well-chosen small model doing the narrow steps. At that multiple, unit economics stop being a finance question and become a product question: the thing either has margin or it doesn't.
Latency compounds the same way. A frontier model adding 800 milliseconds to each of eight sequential steps is six seconds of dead air on a phone call. A small model doing the classification steps in under 150 milliseconds keeps the conversation feeling like a conversation.
Where the frontier model earns its keep
Frontier models are worth paying for when the task is open-ended, the input is messy and novel, and the cost of a wrong answer is high relative to the cost of the call.
In a financial data platform we're building, the step that reads a lender's policy document and reasons about whether a borrower scenario fits is exactly that kind of task. Ambiguous source material, multi-step reasoning, a wrong answer that costs someone real money. We run that on the strongest model we can get and we don't argue about the price.
The same is true for low-volume, high-judgement work: drafting a first pass of a client-facing summary, reconciling conflicting records, anything where a human is going to read the output closely anyway. Volume is low, stakes are high, spend is a rounding error.
The frontier model is also the right tool while you are still learning what the task actually is. Early in a build, you don't yet know which inputs are hard. Running everything on the best model for the first few weeks buys you a labelled dataset of real traffic and a clear view of where the difficulty concentrates. That data is what lets you make the small-model decision properly later.
Where the small model wins outright
Most production inference is not open-ended reasoning. It is classification, extraction, routing and formatting on a narrow domain the system sees thousands of times a day.
Intent detection on a phone call. Is this a new enquiry, an existing client, a supplier, spam. Pulling a name, a property address and a loan amount out of an email into a fixed schema. Deciding which of five workflows a document belongs to. Rewriting a retrieved paragraph into a two-sentence answer with a fixed tone.
On these tasks, a small model with a tight prompt and a few dozen examples in context matches the frontier model on accuracy in our evaluations, at a fraction of the cost and latency. Not close to. Matches. The frontier model's advantage is breadth, and breadth is not what a routing step needs.
Small models are also easier to constrain. Narrower capability means narrower failure surface. A model that cannot write an essay is also a model that will not decide to write one in the middle of a JSON response.
The routing pattern
The practical architecture that falls out of this is tiering, not choosing.
The system runs a small model by default. It escalates to the frontier model only when a specific condition is met: the small model's confidence is below threshold, the input falls outside the distribution it was evaluated on, or the step is one that has been flagged as high-stakes by design. In our systems the frontier model ends up handling somewhere between five and fifteen percent of calls, and that fraction is where nearly all the value of the frontier model lives.
The frontier model has a second job in this pattern. It generates the evaluation sets and the labelled examples the small model is tuned against. You use the expensive model to teach the cheap one, then you use the cheap one at scale. That is the whole economic argument in one sentence.
How we actually make the decision
Model choice without an evaluation set is guessing. The first thing we build for any new step is a few hundred real or realistic inputs with known correct outputs, drawn from the domain the system will actually see. Not a public benchmark. Our traffic.
Then we measure four things for each candidate model: task accuracy on that set, p95 latency under realistic load, cost per completed outcome including retries and guardrail passes, and the cost of a failure when it does get one wrong.
That fourth number is the one that bounds how small you can go. In regulated work, a misclassified document that sends a client down the wrong path is not a cheap error. The downside of a wrong answer sets a floor on model capability, and that floor is task-specific. The floor for "which queue does this go in" is low. The floor for "does this scenario meet the lender's servicing policy" is not.
The trap: small models fail differently
The honest limit of the small-model argument is that smaller models don't just fail more often on hard inputs. They fail differently, and sometimes more quietly.
A frontier model given an out-of-distribution input tends to hedge or ask. A small model tends to produce a confident, well-formatted, wrong answer. That is worse in production, because it passes every structural check and lands in front of a human or a downstream system looking like a good result.
This is why the routing threshold and the guardrails are not optional extras on a small-model architecture. They are the thing that makes it safe. Confidence scoring, schema validation, out-of-distribution detection and a clear escalation path to either a bigger model or a person are load-bearing components, not polish.
What to take from this
Default to the smallest model that clears your accuracy floor on your own evaluation set, and escalate to the frontier model on a defined condition rather than by habit. Use the frontier model to build the evals and the examples that make the small model good. Measure cost per outcome, not per token, and let the cost of a wrong answer decide where the floor sits.
The frontier model is the right call for the hard ten percent. Paying frontier prices for the easy ninety is the most common way we see applied AI projects lose their margin before they find their market.
CPS 230 When Your Service Provider Is an AI System
APRA's operational risk standard never mentions artificial intelligence. It doesn't need to. If your AI sits inside a bank's critical operation, the obligations arrive through the contract — and the transition period has already expired.
AI and the Best Interests Duty: Where Automation Stops in Mortgage Broking
No Australian law addresses AI use by mortgage brokers. That doesn't mean the question is open — the best interests duty already answers most of it, and a privacy deadline in December closes the rest.
Prompt to Production: Shipping AI Features Without Breaking Things
A prompt that works in a playground is not a shipped feature. Here's the internal pipeline we use to take AI from a working prompt to something safe to run in production.