Horace He (Thinking Machines Lab) on one of the most annoying facts about LLMs: ask the same model the same question twice and you can get different answers — even at temperature 0, where the math says the model should always pick the same token.

The common explanation is wrong

  • The usual suspect: “concurrency + floating point” — GPU threads racing to accumulate results in different orders, so the same kernel gives different answers run to run
  • But the same matrix multiplication on the same data is bitwise identical 1000 times in a row. GPUs are definitely concurrent and floating point is definitely involved — so that’s not the whole story
  • The real foundation is floating-point non-associativity: (a+b)+c ≠ a+(b+c). Adding numbers in a different order genuinely changes the result. The question is what changes the order

The actual culprit: your batch size

Four things are simultaneously true: some GPU kernels are nondeterministic; every kernel in an LLM’s forward pass is deterministic; the inference server is deterministic; and yet users see nondeterminism.

The resolution: LLM kernels are not batch-invariant. The result of your request depends on how many other requests are being processed at the same time — because the reduction strategy (how numbers get added together) changes with batch size. A simple demo: the first row of a matrix product computed alone differs from the same row computed as part of a full batch (by up to 1669 in their example).

Since server load is effectively random from any individual user’s perspective, load → batch size → your numerics. That’s the whole trick:

  • The forward pass has no atomic-add races — reductions are done with data-parallel or split strategies that are deterministic per run
  • But split-reduction strategies (Split-K matmuls, FlashDecoding-style Split-KV attention) change how each element is reduced depending on how much parallelism is available — i.e. on the batch size
  • So the same prompt at 1 AM vs noon can produce different tokens, deterministically

The fix: batch-invariant kernels

For the three reduction-heavy operations:

  • RMSNorm: assign each batch element to its own core (data-parallel) so each reduction is always done the same way; accept idle cores at small batch sizes
  • Matmul: compile one kernel configuration and use it for all shapes — split-k only pays off when both matrix dims are small, and in LLM inference the model dimension is usually large; costs ~20% vs cuBLAS
  • Attention: the hard one — the KV cache must be laid out identically regardless of prefill vs decode state, and decode-stage parallelism needs a fixed split-size strategy (fixed-size KV splits instead of fixing the number of splits), plus internal FlexAttention changes

They shipped it as thinking-machines-lab/batch_invariant_ops, swapped into vLLM via torch.Library.

The receipts

  • Qwen3-235B, temperature 0, 1000 completions of “Tell me about Richard Feynman”: 80 unique completions — identical for the first 102 tokens, then 992 say “Queens, New York” and 8 say “New York City”
  • With batch-invariant kernels: all 1000 identical
  • Performance: vLLM default 26s → unoptimized deterministic 55s → with improved attention kernel 42s
  • True on-policy RL: sampling and training never agree bitwise, which silently turns “on-policy” RL into off-policy RL; deterministic inference makes them bitwise identical (KL divergence flat at 0, no reward collapse without importance weighting)

The closing argument: don’t paper over nondeterminism with looser test tolerances — “with a little bit of work, we can understand the root causes and even solve them.”