Xindan Zhang and Fei Wang trace a model server that stays overloaded after a traffic burst ends. Clients keep retrying, and those retries create work that gets cancelled just as it begins running.
Their 137 experimental runs use vLLM 0.30.0, three models and one H100 GPU type. A 107-second burst reaches 2.7 times measured capacity. Traffic then falls to 60% of capacity, with a 120-second deadline per attempt and up to three retries.
- With retries enabled, cancellation alone fails to restore service during the 720-second recovery window. About 12–38% of subsequent calls receive complete answers within the overall 840-second delivery limit.
- In Qwen2.5-7B traces, scheduled retries typically begin processing less than half a second before their own deadline. Every recorded abort reaches the engine within half a second of client departure.
- Eventually aborted requests consume 79–85% of computed prompt tokens. The normal prompt counter misses roughly one-third to one-half of all computed prompt tokens because it records prompts only after output begins.
- Attaching retries to their existing execution delivers every burst and recovery-period call across the three models when cancellation is disabled. Some answers arrive late.
A separate Qwen test delays cancellation for ten seconds so retries can reconnect. It restores delivery, but the proxy never actually cancels anything. Cleanup after a permanently departed client remains untested.
Caveat: These are author-reported experiments with synthetic traffic on one server version and GPU type, not evidence from a multi-server production deployment.
Retry budgets can restore service by dropping work. With cancellation enabled, they favor timely answers for later arrivals, while reattachment preserves earlier calls. Measure complete answers received by clients, and choose explicitly whether late delivery is preferable to losing a call.