IBM Technology’s Avery Blanchard explains what happens after wrapping a PyTorch model with torch.compile.

Three stages

  • Dynamo captures eligible tensor operations in a graph and records guards: assumptions about inputs and program state.
  • AOT Autograd prepares the graph for the compiler and, during training, captures the backward computation used for gradients.
  • Inductor, the default backend, generates hardware-specific code and can fuse compatible operations to reduce launches and memory traffic.
  • Specialized operations can still use separate optimized library kernels.

When it does not help

  • Compilation is normally lazy: the first call incurs tracing and compilation costs.
  • A graph break runs unsupported code in ordinary Python between captured regions; fullgraph=True instead requires a single graph and raises an error when capture fails.
  • Changed shapes or other guard assumptions can trigger recompilation. Dynamic shapes can reduce some recompilations.
  • Measure compilation separately from steady-state execution, and synchronize GPU work when timing it.
  • Speedup depends on the model and workload; compilation is an opportunity to optimize, not a guarantee.

“Whether that opportunity turns into a speed up depends on the model and the workload.”