Zhang Chong proposes mixing nearby token representations through geometry. Each pair defines a plane; coordinates describing that plane replace the attention weights that normally connect positions. The paper tests this idea in small language models and a sentence-pair classifier.

The reported results have narrow scope:

  • Next-token prediction: a 13.00-million-parameter model scores 275.7 validation perplexity against a 12.59-million-parameter Transformer’s 248.4. Perplexity measures prediction error; lower is better. A deeper model also trails its baseline, 261.1 against 235.2.
  • Sentence classification: the geometric head scores 85.38% test accuracy against 85.11%. Both systems retain an attention-using DistilBERT backbone. No repeated-run uncertainty is reported.
  • Computation: with the geometric dimensions and local windows fixed, work grows linearly with sequence length. The actual implementation is slower per step than the Transformer at the tested lengths, up to 256 tokens.

The written method has an unresolved causality problem. Section 3.2 pairs each position with future tokens and feeds those features back into that position. It calls this “causal,” although ordinary left-to-right prediction must exclude future inputs. Both the PDF and HTML contain the inconsistency; whether the implementation corrects it remains unknown.

The geometry offers an architectural idea worth testing. Claims about easier interpretation remain proposals: the experiments include no global explanatory invariants or direct interpretability study. The language-model numbers need a clarified information boundary before they can support the paper’s broader claims.

What the thread adds

The 61-comment thread on Hacker News adds scrutiny of the evaluation and an unanswered engineering question.

  • mapontosevenths — flags the future-token pairing and asks whether the problem is confined to the manuscript. magicalhippo disputes the interpretation in a reply. The thread supplies no implementation evidence that settles it.
  • seamossfet — asks whether the geometric projection adds computation and how the representation affects training-data needs. The paper acknowledges slower steps but supplies no quantitative data-efficiency comparison.
  • HexPhantom — pushes back on dismissing the small experiment entirely: “I think there’s a middle ground between "attention is obsolete" and "this proves nothing"”. That leaves room for the idea without treating it as a validated replacement.

Handles are pseudonymous; HN exposes no per-comment scores. Ordering follows HN’s ranking, so this is a slice of the thread, not a consensus.