OpenAI describes agents that scan Linux code, reproduce suspected bugs, test their impact and propose patches for human review. Its most revealing finding concerns evaluation: an agent can reach the requested outcome through the wrong route.

  • The headline: OpenAI reports 948 distinct flaws that let an existing local user gain administrator access. The run cost $1,213,430 in model API credits.
  • The misleading success: in an early run, 20 of 200 successful exploits used a different vulnerability from the one assigned. OpenAI added an independent check tying each exploit to its intended flaw.
  • The shrinking scaffolding: a library of exploitation techniques improved an older model’s success from 76.2% to 88.9% on 63 candidates. Adding it to a stronger model brought only a 1.5-percentage-point improvement.

Check the path to success, and retest the machinery around an agent when its model changes. Passing the final check can conceal a task failure; elaborate guidance can outlive its benefit.

Caveat: these are OpenAI’s own results using an upcoming model. Candidate patches still need review and distribution; the headline count does not mean every fix has shipped.

The 349-comment thread on Hacker News adds questions about what evidence defenders can actually inspect.

What the thread adds

  • darkamaul — asks for the missing breakdown: “What’s the median age of the vulnerable code? Is more recent code stronger than old ones?”
  • amluto — asks to examine actual reports: “Do you have a list of the publicly discussed or fixed ones?” They also caution against a memory-safety-only focus: “I’ve found plenty of interesting kernel bugs, including some juicy LPEs, and almost none were memory safety errors :)” LPEs are local privilege escalations.
  • perlgeek — pushes back on waiting for the newest model: “You can also use coding agents to clear out technical debt, or improve the security architecture.” That is a suggested defensive approach, rather than evidence it matches this experiment.

Handles are pseudonymous; HN publishes no per-comment scores, and ordering reflects HN’s ranking. This is a slice of the thread, not a consensus.

Previously covered in the November 5 morning digest.