Change the model without breaking production

A cheaper model can pass every spot check and still lose most of the work that matters. In a replay of 175 recent programming tasks, a candidate that serves 2.5× the throughput of the current model came within 7 points of it on easy tasks, 93% against 100%, and fell from 50% to 12% on hard ones. Overall it solved 38.3% against 71.2%. Speed and cost show up on the first day of a swap; a quality regression shows up later, in production. This paper measures both before the switch and sizes the replay that catches the regression: about 538 paired tasks for a five-point drop.

Model swaps happen for good reasons: a smaller model frees GPUs, a newer one is faster, a different family is cheaper to license or to run. The standard way to qualify one is to send it the traffic the current model receives and compare the results before any user sees them, a shadow test, the evaluation half of a canary release^[5]. We ran one as a replay: the same fixed set of tasks to the current model, gpt-oss-120b, and the candidate, Qwen3-30B-A3B, graded by running the code each one wrote.

The setup

The tasks are the 175 problems of LiveCodeBench release v6^[1], published on AtCoder and LeetCode between January and April 2025, after gpt-oss's stated knowledge cutoff of June 2024^[7]. Each answer is graded against the problem's full hidden test set, typically about 40 tests, with a 6 s limit per test; an answer passes only if every test passes, the pass@1 criterion^[2]. 112 problems read standard input, 63 are LeetCode functions.

Both open models ran on one NVIDIA H100 80GB with vLLM 0.11.0, 32k context and an FP8 KV cache, each with its publisher's recommended sampling: gpt-oss-120b at temperature 1.0 and medium reasoning effort, up to 24,576 output tokens; Qwen3-30B-A3B-Instruct-2507 (FP8 weights) at temperature 0.7, top-p 0.8, top-k 20, up to 16,384 output tokens^[6]. Each model answered every problem three times, for 525 answers each. Claude Sonnet 5.5 answered every problem once through the Anthropic API as a reference, not as a candidate.

What the swap changes

The candidate is 33.0 points behind (95% CI 27–39). Qwen's own model card reports 43.2 on LiveCodeBench v6 over February to May 2025^[6], in line with our 38.3% over January to April. Of Qwen's failures, 63 answers ran to the 16,384-token output cap, 48 of them without producing code, and 108 exceeded the time limit, against 13 for gpt-oss: the candidate more often writes an algorithm too slow for the test sizes. Per problem, gpt-oss-120b had the higher pass rate on 85 problems and the candidate on 3.

The same model disagrees with itself

Sampling makes a single answer a noisy measurement. Comparing gpt-oss-120b's three answers to each problem pairwise, it passes one and fails the other on 12.2% of problems. For Qwen the figure is 8.4%. Any comparison between two models has to resolve a difference against this floor, which is why one run of a small evaluation set cannot qualify a swap.

How many replays you need

Pairing each candidate answer with the current model's answer to the same task, the comparison is a McNemar test^[3] on the discordant pairs: tasks where exactly one model passes. With $p_{10}$ the share where only the current model passes and $p_{01}$ the share where only the candidate does, the number of paired tasks needed to detect a drop $\delta = p_{10} - p_{01}$ is^[4]:

In this replay $p_{10}$ = 34.5% and $p_{01}$ = 1.5%, and a difference that large shows within 24 paired tasks. A smaller regression is harder to see. Taking gpt-oss-120b's own 12.2% disagreement as the baseline discordance, with $\alpha$ = 0.05 two-sided and 80% power:

This study's 525 paired answers resolve a drop of about five points. A two-point regression takes 2,783 paired tasks, and one point takes 10,351. A replay should be sized for the smallest drop the workload can tolerate, and run on tasks drawn from the workload, before the decision is made.

Where averages hide damage

The candidate gives up 100−93 points on easy problems, 81−34 on medium and 50−12 on hard. A replay sampled from routine traffic would report a small regression. A workload whose value sits in the hard tail would lose most of it. Breaking results down by the categories that carry the workload's value, here difficulty, is part of the test, not an afterthought.

What the swap buys

The candidate is cheaper to run. Ramped with 8k-token agent prompts under a p95 time to first token of 2 s, as in paper 04, it held 32 concurrent requests at 1,165 output tokens per second, against 8 at 458 for gpt-oss-120b in its tuned configuration. The full replay took 17 minutes against 41. Divided by the problems each model solved, 100,000 solved tasks take 143 GPU-hours on the candidate and 183 on gpt-oss-120b: 2.4× faster, 22% less GPU time per solved task, because it solves far fewer of them. The ratio holds at any GPU price. The cheapest token is not the cheapest task^[8].

What we measured vs. what we cite

The takeaway

A model swap trades quality for cost, and both sides of the trade can be measured before production. Throughput is a load test. Quality is a replay, sized by equation (1) for the regression the workload can absorb, broken down by the categories that carry its value, and compared against how much the current model already disagrees with itself. On this workload the candidate model would have cut GPU time per solved task by a fifth and given up three quarters of the hard problems.

The inputs are a set of production tasks with a way to grade them (tests, a reference answer or a rubric), the current model, the candidate, and the largest regression the workload can tolerate. The output is a sized replay, pass rates with confidence intervals by category, the throughput and cost of each model on the target hardware, and a go or no-go. Natum runs the same replay again whenever a model, a quantization or the serving stack changes.

References

1. Jain et al. LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code. ICLR 2025.
2. Chen et al. Evaluating Large Language Models Trained on Code. 2021.
3. McNemar. Note on the sampling error of the difference between correlated proportions or percentages. Psychometrika 12(2):153–157, 1947.
4. Connor. Sample size for testing differences in proportions for the paired-sample design. Biometrics 43(1):207–211, 1987.
5. Google. The Site Reliability Workbook, chapter 16: Canarying Releases.
6. Qwen. Qwen3-30B-A3B-Instruct-2507-FP8 model card.
7. OpenAI. gpt-oss-120b, chat template system message.
8. Natum Research. The cheapest token is not the cheapest task. 2026.