Kaoutar El Maghraoui of IBM Technology walks through a debugging method for models that produce different results on a new accelerator.

Localize before diagnosing

  • Run identical model weights and controlled inputs on the target and a reference implementation whose behavior has been validated.
  • Save intermediate tensors at corresponding boundaries, such as embedding, attention, normalization and logits.
  • Find the first checkpoint outside the allowed tolerance; investigate the region between it and the last matching checkpoint.
  • This is the first observed divergence: the actual cause may lie between checkpoints, and the failing boundary does not identify the faulty component.

Decide what counts as a mismatch

  • Check tensor shape and data type, then compare values using measures such as maximum absolute difference, relative error or cosine similarity.
  • Choose tolerances for the operation and precision: floating-point order and lower precision can produce expected differences.
  • A final output can amplify an earlier small difference; an asynchronous device may report an error only at a later synchronization point.

Isolate and retest

  • Compare the suspect operation’s inputs, isolate it and make a minimal reproducer.
  • For a normalization mismatch, inspect accumulation precision, reduction behavior, broadcasting, layout and compiler lowering.
  • After a fix, compare checkpoints again, rerun the complete model and keep the reproducer as a regression test.

“First we localize. Then we diagnose.” — Kaoutar El Maghraoui