The cheapest token is not the cheapest task

Move a math agent off a frontier model onto an open model on fast silicon, and the cost per correct answer drops 85% while accuracy goes up a point. Move a customer-service agent onto the same model, on the same chip, and it fails one more conversation in ten. Same switch, opposite verdict.

Most teams compare inference options by price per million tokens. It is the number on every pricing page, and it leaves out the two things that decide what you actually pay: how many tokens a task burns, and how often it succeeds. The number that matters is:

Price per token, $p$, is one input. The others move every time you change the model or the chip. A reasoning model spends output tokens you never see, so $n_{\text{out}}$ grows. A weaker model lowers the success rate $s$, and a lower $s$ divides straight into the bill. A flaky endpoint drops requests, and a dropped request is a cost with no answer attached. So we measured all of it, on three agent workloads, against a frontier baseline.

The baseline is what a team ships by default: Claude Sonnet 4.6 for the steps that reason and Claude Haiku 4.5 for the light ones. The candidate is one open model, gpt-oss-120b, served on two accelerators: Cerebras and SambaNova. The workloads are GSM8K (math, 80 problems), HotpotQA (multi-hop question answering, 80 questions) and $\tau$-bench retail (a tool-using customer-service agent talking to a simulated customer, 20 conversations). Each ships its own grader, so success is the benchmark's definition, not ours.

The price sheet ranks the options one way. Cost per successful task ranks them another.

The cheaper token loses

On the price sheet SambaNova wins. It lists gpt-oss-120b at $0.22 in and $0.59 out per million tokens, against $0.35 and $0.75 on Cerebras: 21–37% cheaper for identical weights. Per successful task it loses. On $\tau$-bench it costs $3.4\times$ as much per completed conversation, because it burned far more tokens to finish the same nine conversations out of twenty. On GSM8K it costs 10% more per correct answer, solves seven fewer problems in eighty, drops three requests outright, and takes $3.5\times$ as long on the requests that complete.

Same weights, different serving stack, and the cheaper token is the more expensive answer. A per-token comparison cannot see this, because it treats tokens per task and success rate as properties of the model. They are properties of the model on a particular stack.

Same model, same chip, opposite verdict

Against the frontier baseline, gpt-oss-120b on Cerebras is a clear win on two workloads. On GSM8K it solves 79 of 80 problems (the baseline solves 78) at 85% lower cost per correct answer and 54% lower latency. On HotpotQA it answers 46 of 80 correctly (baseline 47) at 85% lower cost and 59% lower latency.

On $\tau$-bench the same lane completes 9 conversations in 20, against the baseline's 11. Cost per successful task falls 99%: the baseline spent $0.62 per completed conversation, the open model under one cent. By equation (1) it is the best route we measured. For most teams it is also the wrong one, because the two extra failures are customers whose orders did not get changed. Raising gpt-oss's reasoning effort from low to medium made no difference; the same conversations failed. It is a capability gap on multi-turn tool use, not a setting.

So cost per successful task is the objective and the quality bar is the constraint. You minimize the first only over routes that clear the second:

The margin $\delta$ is a business decision: how many extra failures per hundred tasks you will accept for the savings. It differs by workload. A homework helper might accept three points. An agent that edits customer orders might accept none.

Split the trace only where cost and fragility differ

An agent is not one call. Our HotpotQA agent runs three steps per question (retrieve, reason, answer), and the reasoning step carries 87% of the baseline bill. That is the step you most want to move. In a step-by-step test it was also the one step that broke when moved: routing only the reasoning step to gpt-oss dropped exact match from 0.525 to 0.475, while the retrieval and answer steps moved without loss.

So we kept reasoning on the frontier and moved the other two. On 40 questions, accuracy rose from 0.500 to 0.675, and the bill rose 79%. The steps we were allowed to move were already cheap, and gpt-oss spent more tokens on them than Haiku had. Moving the whole trace to gpt-oss instead held accuracy within a point and cut cost 85%. The open model's reasoning held up when it also handled the steps around it.

Splitting a trace across models pays off only when the expensive steps and the fragile steps are different steps. When they are the same step, keep the trace together and decide it as a whole.

What we measured vs. what we cite

The takeaway

Price per token is an input, not an answer. Where an agent should run is a per-workload decision with four terms (tokens per attempt, price, failed requests and success rate) and a quality bar you set before you look at the savings. That is the decision Natum makes for each production workload, and makes again every time the model or the stack underneath it changes.

The teams that cut inference cost did not find the cheapest token. They priced the task.