A serving study connects live task testing, complete request recording and repeatable load replay. Its main lesson is to size a model server with workloads resembling the agents it will actually serve.
The released Exgentic dataset contains 10,056 sessions and 241,674 model requests. The Inference Perf replay guide describes reconstructing request dependencies and feeding new model responses into later requests.
- In two forty-task comparisons, replay’s 95th-percentile input-token load stays within 1.18 times the original recording. Fresh live runs reach 4.98 times. These are workload-volume ratios, not speedups.
- Live, replayed and synthetic tests reach different Python serving paths in one measured vLLM setup. A targeted synthetic workload subsequently reaches a missed cache-restore path.
- In the authors’ four-model rollout, captured workloads reach about half the optimal concurrent-request count and 60% of the token-processing rate measured with public traces.
Replay controls serving load while live task runs check end-to-end quality. Those tests answer different questions and belong in the same release process.
Caveat: Replay reuses recorded tool results and can force tool selection, partly explaining its stability. Production capacity claims use a proprietary final configuration, and Python coverage does not prove correctness.