Kaoutar El Maghraoui of IBM Technology walks through a debugging method for models that produce different results on a new accelerator.
Localize before diagnosing
- Run identical model weights and controlled inputs on the target and a reference implementation whose behavior has been validated.
- Save intermediate tensors at corresponding boundaries, such as embedding, attention, normalization and logits.
- Find the first checkpoint outside the allowed tolerance; investigate the region between it and the last matching checkpoint.
- This is the first observed divergence: the actual cause may lie between checkpoints, and the failing boundary does not identify the faulty component.
Decide what counts as a mismatch
- Check tensor shape and data type, then compare values using measures such as maximum absolute difference, relative error or cosine similarity.
- Choose tolerances for the operation and precision: floating-point order and lower precision can produce expected differences.
- A final output can amplify an earlier small difference; an asynchronous device may report an error only at a later synchronization point.
Isolate and retest
- Compare the suspect operation’s inputs, isolate it and make a minimal reproducer.
- For a normalization mismatch, inspect accumulation precision, reduction behavior, broadcasting, layout and compiler lowering.
- After a fix, compare checkpoints again, rerun the complete model and keep the reproducer as a regression test.
“First we localize. Then we diagnose.” — Kaoutar El Maghraoui