Sebastian Raschka explains how language models keep track of word order. His illustrated tutorial follows the math and code from GPT-2’s position vectors to the methods used in more recent models.

  • Absolute positions: GPT-2 adds a learned position vector to each token’s representation before processing the sequence.
  • Rotary position embeddings, or RoPE: The model rotates parts of its attention calculations according to position. This makes the distance between tokens part of their comparison.
  • Other choices: Some models rotate only part of each representation, skip explicit rotations in some layers, or add a learned position-based adjustment to attention scores.

The guide includes worked examples and PyTorch code. Raschka’s architecture gallery suggests that RoPE remains common, though its sample of model designs is small.

One important qualification: shifting two tokens together preserves their RoPE comparison only if their underlying representations stay fixed. A real added prefix can change those representations.