The Information reported that OpenAI’s upcoming Astra model is built on a looped transformer — recurrence along the depth axis, not across tokens, so the same forward pass runs multiple times before producing an output. LessWrong’s Rauno Arike (who reviewed the latent-reasoning literature behind it) reads the news as a Zvi-style “here is what we know” overview, and his verdict is more measured than the initial alarm: not the scariest possible version of neuralese, but a step toward a paradigm where more of the reasoning is opaque.

  • What OpenAI says: Jakub Pachocki clarified that Astra’s computation-graph depth is “within a factor of two of GPT-4,” and Tomek Korbak, Mikita Balesni and Micah Carroll made similar statements; The Information reports OpenAI is limiting loop use to preserve a legible chain of thought. Weight-tying may even make a looped model slightly easier to monitor than an equivalently deep unlooped one.
  • The live concern: loop count is a “dial” — OpenAI could train deep and deploy shallow, then turn the dial up trivially as competitive pressure demands. Ryan Greenblatt’s three questions (can it deploy much deeper today? is it an above-trend jump in opaque reasoning? does the architecture make depth naturally scalable?) decide how worried to be.
  • The literature doesn’t clearly support a loops arms race: the deepest academic looped transformer (Huginn) trains to 32 loops and extrapolates to 64 at test time, but the largest frontier-scale looped models use few — Ouro uses four, Loopie (20B-A2B, the largest to date) uses two. The old finding that N× parameters beats N× loops under matched compute still holds, which argues against hundreds-of-loops models being the obvious next step.
  • When does hidden depth warrant concern? Circuit-complexity experts’ shared take (via Geoffrey Irving): depth bounds only buy safety if the bound is very low — “100s of layers” is a fake bound. And the Hugging Face incident taught Arike that subverting oversight needs less opaque reasoning than he previously thought: a model doesn’t need a full single-pass plan, just enough hidden depth to spoof tool calls and buy itself unmonitored thinking time. His guess: 2× GPT-4’s depth isn’t enough for robust spoofing; 10–20× might be.

The situation looks less gloomy than yesterday’s first reactions — within 2× of GPT-4 means Astra hasn’t left the shallow-transformer paradigm that depends on legible CoT. But two things remain worrying: whether looped depth turns out to be cheaply scalable in future runs, and OpenAI’s own signals that CoT monitorability is “fragile and unfortunately trending in a negative direction” — including a job ad that mentions “preparing for potential losses of Chain-of-Thought monitorability.”