Xiuyuan Qi and colleagues address a reproducibility problem in model serving: neighboring requests can change an answer even with a fixed prompt and random seed. Different batch shapes change floating-point sums, sometimes flipping the next generated token.
BlockLock fixes the protected request’s position and processing dimensions while sharing the initial prompt pass. Its Rows mode generates answers for every request. Pack generates only the protected answer, so its timings describe less delivered work.
- Tests cover five dense models on A6000, A100 and H100 graphics processors, plus initial two-processor experiments. The reported tests preserve generated sequences and the exact bits of the top twenty scores.
- With four requests and 64 generated tokens each, Rows cuts execution time by 53.5–55.0% versus vLLM’s batch-invariant mode. Both sides deliver the same amount of output.
- Against ordinary PyTorch batching, Rows adds 0.5–0.8% execution time in that suite. Its overhead against ordinary vLLM is 5.4–6.9%, a different baseline.
- Longer prompts can erase the advantage over processing requests separately. At a 4,096-token class width and 2,048 generated tokens, Rows takes 141.141 seconds versus isolation’s 135.103.
The formal guarantee assumes fixed geometry, repeat-stable arithmetic and an exact handoff of the model’s stored prompt state. A batch-one audit also requires restricting unstable backend operations; geometry alone is insufficient.
Caveat: Reproducibility holds within a fixed, validated hardware and software setup. Models that route between different experts remain untested, and the code repository is withheld for review.
The measurements cover GPU execution, not network queueing or tail response times. Treat the speedup as a workload-specific result, and revalidate after changing libraries, hardware or batch settings.