OpenAI proposes measuring training limits with a private code test

  • OpenAI publishes prediction scores and proposes shared training limits — The disclosure makes a proposed safety measure more concrete, while exposing why one lab’s private test cannot yet serve as an industry standard.
    • Chief scientist Jakub Pachocki publishes log-perplexity measurements, a score for how well models predict unseen text. Lower scores mean better predictions on that dataset.
    • The test uses a fixed snapshot of OpenAI’s internal codebase. Reported scores include about 0.99 for GPT-3.5, 0.91 for GPT-4 and 0.76 for a recent model associated with new mathematical results.
    • Pachocki proposes limits that developers would jointly adopt before further training advances. Crossing them would require concrete improvements in understanding and monitoring model behavior.
    • He identifies 0.68–0.70 as a future score range deserving advance scrutiny. His suggestion that AI could accelerate training research in 2027 is a forecast, rather than an observed result.
    • The article acknowledges that datasets differ across labs and that additional reasoning can improve task performance without lowering this score.
    • Caveat: These are company-reported measurements on private code, and the proposed limits are neither an adopted industry agreement nor a demonstrated universal danger threshold. (HN 370 points · 240 comments · Techmeme · primary disclosure)

New runtime support connects smaller Inkling files to applications

  • LookAtMyQuant adds Nvidia support and a local chat server — Developers can now connect the packed Inkling models to existing chat clients and coding assistants, with explicit hardware and request limits.
    • A November 22 merge adds Nvidia execution and the shared lamq serve interface. This Morning’s Inkling item covers downloads and quality tradeoffs. These new features were merged after Morning’s collection.
    • /v1/chat/completions supports streamed replies, separate reasoning text and structured tool calls. A tool call is the model asking a program to do something.
    • Tool selection supports auto and none. Requests demanding a call or forcing a particular function receive HTTP 400, an invalid-request response.
    • The server accepts one generation request at a time and returns HTTP 503 when busy. It binds locally by default, rejects browser-origin requests and requires a key for other network addresses.
    • The CUDA guide covers Nvidia’s GPU programming platform and limits execution to single-sequence text generation. Its native engine requires Ampere-generation GPUs or newer, and accelerated drafting remains unsupported for the two largest mixed-format packs.
    • Caveat: Passing interface tests with a simulated model establishes neither real GPU performance nor reliable completion of extended coding work. (HN 565 points · 135 comments · merged source, runtime guide and local interface tests)
All gathered items - what was cut and why (9)