Somebody notices the invoice. There is a feature that pulls six fields out of every incoming contract, it runs against the largest model on the price list, and it runs on every document all day. Asking whether it has to run there is a reasonable question, and it takes about ten minutes to answer. The cheap model goes in, the output falls apart, the original goes back. The model name gets a comment next to it in the config saying do not change this. The ticket closes.
The example is invented and deliberately ordinary. The ten minutes are not. What happened in them is that a system was tested once, failed once, and got diagnosed without ever being opened. Everybody concluded the task was hard. The same evidence supports a different conclusion just as well, which is that the task was never properly specified, and the large model had been covering the difference the whole time, quietly, on every call, since the day it shipped.
Nobody opened it
That reaction is not laziness, it is a lack of anywhere to look. The feature is one call. A document goes in with a prompt wrapped around it, JSON comes back, and the JSON was wrong. There is no stage that failed, no intermediate value to inspect, no point you can name where the problem entered. The only component in the whole thing with a name is the model.
So the model becomes the dial. It is the one part you can turn without understanding anything, and turning it up usually works, which is what makes it a trap. The symptom goes away and takes the reason to investigate with it. The system stays exactly as opaque as it was and now it costs more.
Four things reach the model, and only one of them is a dropdown
Open that call up and you find four things you built and one thing you picked. The instruction says what to do. The context is the material the answer has to come from. The tools are what the model reaches for when the answer is not in front of it. The output shape is the contract the result has to satisfy. Then there is the model, which is a string in a config file.
Reaching for a bigger model means reaching for the fifth one because the other four are harder to think about. That is the whole pattern, and it is the path of least resistance wearing the clothes of a technical decision.
It is also a claim that can be wrong, so it is worth saying what would prove it. Specify the instruction, select the context instead of dumping it, give the model tools it can use without guessing, constrain the output, and if the small model still sits below your threshold on a set of cases you graded by hand, then the task really does need the capability and you should go and buy it. That happens. It happens far less often than the config files suggest.
What the big model was covering
Go back to the contract extractor and look at what was actually being sent.
An instruction that described a wish. Find the six important fields and return them as JSON. Nothing in there says which date counts as the start date when the document names three, or what to do when a field is missing rather than unclear. A large model makes a decent guess, which is why nobody notices that the decision was never made by a person. A small model guesses too, and its guess will be different next Tuesday.
Everything, in case. The whole contract goes in, all ninety pages of it, because filtering is work and the window is large. This feels safe and is the opposite of safe. A retrieval problem you could have tested became a retrieval problem happening invisibly inside the model, once per call, where nobody can see it.
A tool it had to guess at. The extractor can look up a counterparty in the customer register. The tool is called lookup_party, it takes a string called name, and when nothing matches it returns null. The large model reads the surrounding intent and calls it correctly anyway. The small one sends the trading name instead of the legal name, gets nothing back, learns nothing from that, and tries again.
An output nobody constrained. JSON was requested in the prompt rather than enforced by the schema, so something downstream parses whatever arrives. That parser is the real interface now, and it was never designed, it accreted. Every bit of the model’s capability gets spent producing text that a regular expression has to survive.
None of these are model problems. All four get blamed on the model, because the model is where they surface.
A million tokens is not an index
The second one deserves its own paragraph, because the largest windows made a bad habit feel like an architecture.
A context window is storage. It is not a search index, and putting ninety pages into it does not mean the model reads ninety pages the way you would. This is one of the better studied failure modes in the field. Lost in the Middle found that accuracy is highest when the relevant passage sits near the beginning or the end of the input and drops noticeably when it sits in the middle. RULER showed that models which look flawless on a simple find-the-sentence probe degrade substantially on harder long-context tasks, well before their advertised limit. NoLiMa added the part that matters most in practice: performance falls off sharply when the question and the passage that answers it do not share obvious vocabulary, which describes almost every real question anybody asks of a real document.
A large window raises the ceiling on what you can pass in. It does not decide what to pass in, and that decision is work you can actually see: a retrieval step has an input and an output, and you can tell at a glance whether the answer was even in the room. Anthropic have published on this directly, reporting large reductions in retrieval failure simply by giving each chunk enough surrounding context to be identifiable. Structuring beat scaling. When retrieval is a step you own, a wrong answer is traceable. When it is an emergent property of ninety pages in a prompt, every wrong answer just looks like the model being stupid.
Taking the feature apart
So somebody opens it, and the rebuild is not clever. Locate the sections likely to hold each field. Ask for one field at a time from the two or three pages that could contain it, against a schema that says what that field is allowed to look like. Check dates and amounts in code, because code is better at that than any model will ever be. Where nothing is found, return not found instead of the most plausible looking string on the page. Escalate only the fields that failed.
The tool gets rewritten at the same time, and this is the part that changes behaviour most for the least effort. The model is the caller and the description is its only documentation.
{ "name": "lookup_party", "description": "Looks up a party", "parameters": { "name": { "type": "string" } }}{ "name": "find_customer_by_legal_name", "description": "Matches a counterparty against the customer register by its full legal name including the legal form, as printed in the contract preamble. Not the trading name, not the address block. Returns no_match with the closest three candidates when there is no exact match.", "parameters": { "legal_name": { "type": "string", "minLength": 3 } }}Nothing in the second version needed a better model. It needed somebody to decide what the tool was for. The same applies to the output: a schema enforced by the API removes a whole category of failure rather than making it less likely, because an answer that cannot be expressed does not have to be checked for. OpenAI reported, on their own eval of their own feature, that constrained structured outputs took schema compliance to the top of the scale where prompting alone sat well below it. Vendor evidence about a vendor product, so weigh it accordingly, but the mechanism is not controversial. We wrote about this shape from the product side in payloadcms-mcpx.
Every step in the rebuilt version is boring, and that is the property that matters. Boring, narrow steps are what small models are good at, and each one is inspectable when it goes wrong. The claim here is about the structure and not about a number. We are not saying this is cheaper by some percentage. We are saying you can now tell which part is broken.
Which is also the point at which you can stop guessing. Take a few dozen real contracts, including the ones that went wrong, write down the right six fields for each by hand, run both models, and count. That is not an evaluation framework and does not need to be. It turns “the cheap one feels worse” into a number somebody can disagree with, and it tells you the more useful thing, which is not which model won but which cases both of them failed. Those cases are your specification, in the only handwriting that never lies.
Where the big model earns its price
None of which means the frontier tier is a mistake, and this argument is worth nothing if it cannot say when it is right.
Long-horizon agentic work is the clearest case. When a run is thirty tool calls, per-step reliability compounds, and a rate that looks respectable in isolation is unusable across a chain. What the larger model buys there is often consistency rather than peak ability, and those are different products. The tau-bench authors made that measurable by asking whether an agent solves the same task repeatedly rather than once, and the gap between solving something once and solving it every time is where a lot of demos live.
Then there is the arithmetic nobody in our position should skip. Two engineer-days of retrieval plumbing cost more than an enormous number of calls at the top of the price list. If the contract extractor ran fifty times a day inside one team, the frontier model would be the right answer and optimising it would be a hobby. The crossover is set by volume, by how much latency matters, and by what a wrong answer costs. Where one wrong answer is expensive or hard to reverse, paying an order of magnitude more per call is not waste, it is the cheapest insurance on the menu. Genuinely open-ended work belongs up there too: architecture, ambiguous research, anything where the shape of a good answer is not known in advance. That is precisely the work the tier gap is measured on, and it is the opposite of what most production features do.
The compensating architecture is not free either. Retrieval, decomposition, validation and retries all cost latency, and three attempts at a small model can lose to one success at a large one in both money and wall clock. Prompts do not port cleanly between tiers, so moving down is a re-tuning exercise and not a one-line diff. And since we build software for a living, an argument that recommends engineering over vendor spend is an argument that recommends us. Read it with that in mind.
Ship the smallest thing that passes
Which leaves a working order rather than a verdict. Build on the largest model, because it is the best instrument you have for finding out whether the task is possible at all, and it takes capability off the table while you are still working out what you are actually asking for. If it fails there too, you have learned something more valuable than any benchmark: the specification is broken, and nothing on the price list will fix it.
Then step down and read the failures. Teams skip that part, and it is the informative one. The small model does not just fail, it fails specifically, and each failure names the piece of the system that was leaning on capability. Ambiguous instruction, missing context, tool called wrong, output that did not have to conform. Fix those, try again, and stop when it passes the set you graded, whichever tier that turns out to be.
Everything specific here expires. Tiers move every few months, today’s small model is last year’s frontier, and any recommendation naming a model is wrong by the time it is interesting, which is why there are none in this post. What does not move is the question. When a feature only works at the top of the price list, you are holding evidence about your own system, and it says the same thing in every generation: something the surrounding software should have decided is being decided by the model instead. Sometimes that is a deliberate trade. Usually it is a decision nobody made, sitting in a config file behind a comment telling everyone not to touch it.