Private inference fails in software, not silicon
Kill a vLLM server running gpt-oss-120b on an H100 and it serves again in 59 s. Start the same server on a fresh machine and it takes 400 s. Same model, same GPU, same flags: a 6.7× difference, set by what the machine already has on disk and in cache. Recovery time is not a property of the model. It is a property of the restart.
Teams that move an agent off a hosted API onto their own GPUs plan for the first question: does the model fit, and what does a task cost. The second question decides availability: when the server stops, how long until it serves again, and what is lost in between. The time to recover is a sum:
detection, process start-up, loading the weights onto the GPU, and engine initialization (compilation, CUDA graph capture, KV cache allocation). We measured the last three on our own deployment, crashing it on purpose under live traffic, and took the first and the failure rates from published production data. Every number below is marked as measured or cited.
Most of that sum is set by deployment decisions, not by the model.
One full-context request of headroom
The deployment is gpt-oss-120b (MXFP4 weights) on one NVIDIA H100 80GB, vLLM 0.11.0, 32k context, prefix caching off. The weights occupy 66 GiB. What remains holds 25,488 tokens of KV cache, enough for 1.24 requests at full context. Storing the cache in FP8 doubles that to 51,056 tokens (2.48 requests) and slows a single coding task from 8.5 s to 9.4 s. On the default configuration the server passed a code generation suite 5 of 5 times on every run, returned its first token in 503 ms at the median, and cost $0.0071 per task at $3.00 an hour. An alternative attention backend moved the median 2%, inside run-to-run spread, so the defaults stayed.
For a single latency-bound agent, 1.24× is enough. For concurrent traffic it is the first limit the deployment hits, and it is a configuration choice, not a hardware one.
A crash costs a minute. A cold start costs nearly seven.
We sent one streaming request per second, killed the vLLM process group with SIGKILL, and restarted it in place with identical arguments, three times. With the weights already on local disk and in the page cache, the server answered its health check 59 s after the kill at the median (range 58–60 s), and the next request succeeded at the same moment. Each crash failed 59–61 requests: every request sent during the outage, plus the one in flight at the kill. In-flight work is not recovered. Its KV cache dies with the process.
The restart breaks down as 14 s loading the weights, 21 s initializing the engine (12 s of it capturing CUDA graphs, 4 s loading compiled kernels from cache), and 24 s of process and API server start-up. The first boot on the fresh pod took 400 s. Most of it, 319 s, was the model loading phase; vLLM attributes 22 s of that to reading the weights and does not break down the rest. Compilation took 35 s before it was cached.
These numbers do not transfer between deployments. NVIDIA measured warm vLLM restarts of 106 to 157 s on 8-GPU B200 nodes, with 94 s of the 157 spent capturing CUDA graphs[3]; on our single H100 graph capture took 12 s. Loading the same gpt-oss-120b weights took 42.3 s with vLLM's default loader and 12.8 s with a streaming loader from Azure blob storage on 8× H100[8]; from local disk with a warm page cache it took us 14 s. The term that dominates depends on the node, the model, the storage path and what is cached, so it has to be measured on the deployment that will run.
Our test detected the crash in under 0.1 s because the probe ran beside the server. A default Kubernetes liveness probe checks every 10 s and fails after 3 misses[7], which adds about 30 s to every recovery.
What actually breaks
Training clusters fail mostly in hardware: about 78% of 419 unexpected interruptions in Meta's 54-day Llama 3 pre-training run were confirmed or suspected hardware faults[1]. Serving fails mostly in software. In Microsoft's study of 156 high-severity incidents in AI inference services, about 60% were inference engine failures and about 20% infrastructure[2]. Over 18 weeks of production serving, NVIDIA recorded 5,762 in-place restarts of replicas; most engine failures raised no GPU error at all, and only three episodes required resetting a GPU[3]. Across 929 bugs in five inference engines, including vLLM, crashes were 65% of symptoms[4].
Not every failure is a crash. 26% of Microsoft's inference incidents were not caught by automated
detection[2], and in a separate Microsoft study of generative AI cloud services, people rather than
monitors reported 38.3% of incidents, against 13.7% for other services[5]. Anthropic's postmortem of
three quality regressions in 2025 states that its evaluations simply didn't capture the degradation users
were reporting
[6]. A health check detects a dead process. It does not detect wrong answers.
What the restart rate costs
NVIDIA's 5,762 restarts came from about 150 concurrent replicas over 18 weeks[3], or 0.30 restarts per replica per day: about 2 a week. Applying that rate to our measured recovery times gives the expected downtime of a single replica:
| Restart path | Per failure | Per week | Availability |
|---|---|---|---|
| restart in place, instant detection | 59 s | 2.1 min | 99.98% |
| restart in place, default probe | 89 s | 3.2 min | 99.97% |
| fresh machine, default probe | 430 s | 15.3 min | 99.85% |
The rate comes from a cluster of B200 and RTX PRO 6000 GPUs serving a different model mix, so it is an order of magnitude, not a forecast for this deployment. Two things hold regardless of the exact rate. A single replica has short, regular outages that the agent above it sees in full. And a replica that comes back on a new machine, without the weights on disk or compiled kernels in cache, takes 6.7× longer to serve again than one that restarts in place.
There are three ways to cover the gap:
| Strategy | Outage per failure | Cost | Data stays inside |
|---|---|---|---|
| Restart in place | detection + 59 s | 1× GPU | yes |
| Hot standby replica | detection | 2× GPU ($3 → $6 an hour here) | yes |
| Fail over to a hosted API | detection | hosted price per task while failed over | no |
Failing over to a hosted model is the cheapest to build; our own gateway does it by default. For a workload that moved in-house for data reasons, it sends that data back out exactly when the private deployment is failing. In all three, the requests in flight at the failure are lost. Live migration of in-flight requests between replicas has been demonstrated in research systems, at 20 to 30 ms of downtime[9], but is not part of a standard vLLM deployment.
What we measured vs. what we cite
| Claim | Value | Source |
|---|---|---|
| gpt-oss-120b on one H100: correctness, TTFT p50, cost per task | 5/5; 503 ms; $0.0071 | measured |
| KV cache headroom, FP16 vs. FP8 | 25,488 vs. 51,056 tokens (1.24× vs. 2.48×) | measured |
| Kill to serving, restart in place | 59 s median, 58–60 s ($n$ = 3) | measured |
| First boot on a fresh pod | 400 s ($n$ = 1, from log timestamps) | measured |
| Requests failed per crash at 1 req/s | 59–61, including 1 in flight | measured |
| Training interruptions from hardware | ~78% of 419 (Llama 3, training) | cited [1] |
| Inference incidents by cause | engine ~60%, infrastructure ~20% ($n$ = 156) | cited [2] |
| Production restarts, replicas, recovery time | 5,762 in 18 weeks, ~150 replicas, median 3.5 min | cited [3], preprint |
| Warm vLLM restart on 8× B200 | 106–157 s; CUDA graphs 94 s of 157 | cited [3], preprint |
| Crashes as share of engine bugs | 65% of 929 | cited [4] |
| Incidents reported by people, not monitors | 38.3% vs. 13.7% | cited [5] |
| Kubernetes probe defaults | 10 s period, 3 failures | cited [7] |
| gpt-oss-120b weight load from blob, default vs. streaming | 42.3 s vs. 12.8 s (8× H100) | cited [8] |
| Per-replica downtime and availability | cited rate × measured restart time | derived |
The takeaway
Whether a model fits on a GPU is one measurement. How long it is down when it fails depends on four choices made before the first failure: how often health is checked, whether a replacement starts with the weights already on disk, whether a standby is paid for, and whether a fallback may send data outside the boundary. That is the part of a private deployment Natum qualifies before it carries production traffic, and measures again when the model or the stack underneath it changes.
The hardware will mostly hold. The process on top of it will restart, and how long that takes is a number to measure before production, not after.
References
- Llama Team, Meta. The Llama 3 Herd of Models, §3.3.4, Table 5. 2024.
- Ranganathan, Zhang, Wu (Microsoft). Enhancing reliability in AI inference services: an empirical study on real production incidents, §4.4, §5.1. 2025.
- Saereesitthipitak et al. (NVIDIA). Fast Recovery for LLM Serving via Decoupled Device Memory Lifetime in Dynamo, §2.2, §5.2. Preprint, 2026.
- Liu et al. A First Look at Bugs in LLM Inference Engines, §4.2. 2025.
- Yan et al. An Empirical Study of Production Incidents in Generative AI Cloud Services, Table I. 2025.
- Anthropic. A postmortem of three recent issues. 2025.
- Kubernetes. Pod v1 API reference: Probe.
- Microsoft Azure SDK Blog. Eliminate LLM cold starts with Azure Blob Storage and Run:ai Model Streamer. 2026.
- Sun et al. (Alibaba). Llumnix: Dynamic Scheduling for Large Language Model Serving, §6.2. OSDI 2024.