Two lines of engine configuration double the number of concurrent agent requests a GPU serves inside a 2 s time-to-first-token target. On gpt-oss-120b with 8k tokens of context, an FP8 KV cache and a 2,048-token prefill budget raise capacity from 4 to 8 concurrent requests and throughput inside the SLO by 22%. At 4k context, capacity goes from 8 to 16. At 16k, a load that stalled the default configuration at 11 tokens per second runs at 253. The model, the hardware and the price per hour stay the same.

A serving engine batches requests together. Larger batches produce more tokens per second and make every request in the batch wait longer. In a closed loop with $c$ requests in flight, throughput follows from Little's law:

where $L_{\text{out}}$ is the output length and $T_{\text{e2e}}$ the end-to-end latency of one request. Raising $c$ raises $X$ until $T_{\text{e2e}}$ grows faster than $c$. The number that matters for a production deployment is not peak throughput but goodput: the highest throughput at which latency still meets the SLO^[4]. We measured it on one GPU, at three context lengths, at defaults and after tuning.

The setup

The deployment is gpt-oss-120b (MXFP4 weights) on one NVIDIA H100 80GB, vLLM 0.11.0, 32k context, prefix caching off, 95% of GPU memory for the engine (the minimum at which this model gets any KV cache on 80 GB). The traffic is agent-shaped: each prompt is a real GitHub issue with the source files its fix touches, from SWE-bench Lite^[5], cut with the model's own tokenizer to 4,165, 8,261 or 16,453 tokens while keeping the issue and the closing instructions. Every request generates exactly 512 tokens, so each load level does the same work. Prompts are distinct (200, 171 and 100 per length), so no request reuses another's cache.

Each level holds $c$ requests in flight. The first $c$ are discarded as warm-up, the next $\max(16, 2c)$ are measured, and filler requests keep the engine at $c$ until every measured request completes. The SLO is a p95 time to first token (TTFT) of at most 2 s and a p95 time per output token of at most 50 ms. The knee is the highest concurrency that meets both.

At defaults, the knee arrives early

With 4k prompts, the default configuration meets the SLO up to 8 concurrent requests, where it produces 629 output tokens per second at a p95 TTFT of 1.47 s. At 12 the median request still sees its first token in 1.76 s, but the slowest wait 9.0 s for cache memory to free up, and throughput stops rising. With 8k prompts the knee is 4 (374 tokens per second). With 16k it is 2 (211 tokens per second).

At 16k tokens and four concurrent requests the default configuration collapsed. The engine ran two requests and held two waiting, with the KV cache 65–81% full, and generated 10.8 tokens per second for the 16 minutes the level ran: a p95 TTFT of 97 s and 184 ms per output token, against 166 tokens per second for a single request. The KV cache at defaults holds 25,440 tokens, about one and a half 16k requests with their output.

Two settings double the knee

We changed one setting at a time and ramped each configuration with 8k prompts.

Storing the KV cache in FP8 doubles the tokens it holds^[6] and lifts throughput at eight concurrent requests by a third. It does not move the knee: each new 8k prompt still enters as one large prefill step that stalls the requests already decoding, and the p95 TTFT at eight stays above 2 s. Capping each engine step at 2,048 prefill tokens splits long prompts into chunks that are scheduled alongside ongoing decodes^[3]. With both settings the p95 TTFT at eight falls to 1.44 s and the knee doubles. vLLM's own guidance is that a smaller budget improves inter-token latency and a larger one improves TTFT^[7]; under this concurrent long-context load, the smaller budget gave the lower TTFT. A 16,384-token budget behaved like FP8 alone.

With 4k prompts the tuned configuration holds a p95 TTFT of about 0.95 s from 12 to 16 concurrent requests, where the default had already failed, and produces 790 tokens per second at its knee of 16: 26% more than the default at its knee. At 16k, both configurations meet the 2 s SLO at two concurrent requests. At four, the load that collapsed the default, the tuned configuration runs at 253 tokens per second with a p95 TTFT of 2.30 s: 24× the throughput, just outside the SLO.

What tuning costs

Two things. Each request decodes more slowly because it shares steps with prefill chunks: at the 4k knee, p95 time per output token rises from 10.3 ms at defaults to 19.0 ms tuned, still well inside the 50 ms bound. And past its knee the tuned configuration degrades faster. At twelve concurrent 8k requests it produced 163 tokens per second, with a p95 TTFT of 14.7 s and one gap of 8.4 s between streamed tokens. A tuned deployment needs admission control at the knee, so excess requests queue at the gateway instead of inside the engine.

The knee also depends on where the SLO is set:

For a fleet, the knee sets how many GPUs a workload needs, and throughput at the knee sets the GPU-hours behind every token. Both are independent of what a GPU-hour costs, so the percentage holds on any contract, cloud or owned cluster:

At 4k and 8k context, tuning halves the fleet and cuts the GPU time behind each token by a fifth. At 16k the knee stays at two concurrent requests on both configurations, and the tuned one spends 9% more GPU time per token there; its value at 16k is the load above the knee, where the default stalls.

What we measured vs. what we cite

The takeaway

A GPU's capacity for an agent workload is the knee, not the peak, and the knee moves with three things: context length, the SLO, and two lines of engine configuration. On this deployment those two lines doubled the concurrent requests one H100 serves within a 2 s TTFT at 4k and 8k context, and turned a collapse at 16k into a working, slightly slower service. The setting that helped most ran against the general guidance for it, which is why the knee has to be measured on the traffic the deployment will carry.

The inputs are a sample of production prompts or traces, the model and serving stack, and the latency targets. The output is the knee at defaults and tuned, the configuration that produced it, the admission limit to set at the gateway, and the fleet size and GPU-hours per token at that limit. Natum runs the same ramp again when the model, the engine version or the traffic mix changes.

References

1. Yu, Jeong, Kim, Kim, Chun. Orca: A Distributed Serving System for Transformer-Based Generative Models. OSDI 2022.
2. Kwon et al. Efficient Memory Management for Large Language Model Serving with PagedAttention. SOSP 2023.
3. Agrawal et al. Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve. OSDI 2024.
4. Zhong et al. DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving. OSDI 2024.
5. Jimenez et al. SWE-bench: Can Language Models Resolve Real-World GitHub Issues? ICLR 2024.
6. vLLM. Quantized KV Cache.
7. vLLM. Optimization and Tuning: Chunked Prefill.