Chief scientist Jakub Pachocki publishes log-perplexity measurements, a score for how well models predict unseen text. Lower scores mean better predictions on that dataset.
The test uses a fixed snapshot of OpenAI’s internal codebase. Reported scores include about 0.99 for GPT-3.5, 0.91 for GPT-4 and 0.76 for a recent model associated with new mathematical results.
Pachocki proposes limits that developers would jointly adopt before further training advances. Crossing them would require concrete improvements in understanding and monitoring model behavior.
He identifies 0.68–0.70 as a future score range deserving advance scrutiny. His suggestion that AI could accelerate training research in 2027 is a forecast, rather than an observed result.
The article acknowledges that datasets differ across labs and that additional reasoning can improve task performance without lowering this score.
Caveat: These are company-reported measurements on private code, and the proposed limits are neither an adopted industry agreement nor a demonstrated universal danger threshold.
(HN 370 points · 240 comments · Techmeme · primary disclosure)
New runtime support connects smaller Inkling files to applications#
A November 22 merge adds Nvidia execution and the shared lamq serve interface. This Morning’s Inkling item covers downloads and quality tradeoffs. These new features were merged after Morning’s collection.
/v1/chat/completions supports streamed replies, separate reasoning text and structured tool calls. A tool call is the model asking a program to do something.
Tool selection supports auto and none. Requests demanding a call or forcing a particular function receive HTTP 400, an invalid-request response.
The server accepts one generation request at a time and returns HTTP 503 when busy. It binds locally by default, rejects browser-origin requests and requires a key for other network addresses.
The CUDA guide covers Nvidia’s GPU programming platform and limits execution to single-sequence text generation. Its native engine requires Ampere-generation GPUs or newer, and accelerated drafting remains unsupported for the two largest mixed-format packs.
Caveat: Passing interface tests with a simulated model establishes neither real GPU performance nor reliable completion of extended coding work.
(HN 565 points · 135 comments · merged source, runtime guide and local interface tests)
Muru’s AI-assisted Python server → LOW_UTILITY: the alpha server and detailed engineering account are real, but AI-assisted authorship alone does not establish a new model-building or coding-assistant method. Its general web-serving features are omitted from this short AI edition.
Worker 755’s AI-manager game → LOW_UTILITY: inspectable source and a local-model option make it more than a trailer, but playing the manager’s subordinate supplies no measured reliability result or new engineering integration for this edition.
Anil Madhavapeddy’s software-complexity essay → LOW_UTILITY: formal specifications and implementation diversity are substantive proposals, but the essay supplies no newly measured deployment outcome or released method to add to today’s news.