OpenAI chief scientist Jakub Pachocki wants model developers to agree on limits to further training, tied to concrete improvements in safety monitoring. His proposed yardstick is perplexity: how well a model predicts unseen text, with lower scores meaning better predictions.

  • OpenAI reports log-perplexity, a transformed prediction score, of roughly 0.99 for GPT-3.5, 0.91 for GPT-4 and 0.76 for a recent model associated with mathematical breakthroughs.
  • Those measurements use a fixed snapshot of OpenAI’s internal codebase. They are company-reported results, and different labs’ datasets make direct comparison difficult.
  • Pachocki flags a future 0.68–0.70 range for scrutiny. He also suggests AI could accelerate research into training methods as soon as 2027; both are forecasts.
  • The essay acknowledges a major blind spot: letting a model reason through intermediate steps can improve its abilities without reducing its perplexity.

A stable training metric could complement task-specific tests that quickly become outdated. But an industry agreement would still need comparable measurements and safety conditions that cover improvements outside initial training. The proposed thresholds are neither an adopted agreement nor demonstrated universal danger levels.

The 292-comment thread on Hacker News adds challenges to how the metric would work in practice.

What the thread adds

  • seizethecheese — offers a competing explanation for the code-test trend, explicitly as a possibility: “It’s entirely possible that the results are largely due to more intensive pretraining on code.”
  • charm137 — asks about comparing models that divide text differently or predict several pieces at once: “I don’t know how possible it’d be to do an apples-to-apple comparison of perplexity across models”.
  • theptip — worries that a limit could shift development into reinforcement learning, training through rewards: “The one concern I have is if we pin pretraining perplexity, then there will be more pressure on RL for progress.”

HN handles are pseudonymous; no per-comment scores are public, ordering is HN’s ranking, and this selection is a slice of the thread rather than a consensus.

Also covered in November 22’s Afternoon digest.