Arjun Panickssery describes supervising AI on game-playing and philosophy research projects. The recurring failure is choosing what matters: which source to trust, which experiment can answer a question, and which direction deserves more work.

  • The model found an important blog post but, in his account, missed its relevance compared with older papers.
  • It ran comparisons with too little training data to tell competing model designs apart.
  • There was a striking success too: he reports a tenfold improvement in listing legal Scrabble moves. Human suggestions changed both how performance was measured and how the problem was divided up.

Fast implementation still leaves someone responsible for deciding whether an experiment will teach them anything. Panickssery also describes questioning the model repeatedly to understand its reports and establish what it actually did.

Caveat: these are personal observations, including the tenfold gain, without enough experimental detail to reproduce or generalize them.

The 161-comment thread on Hacker News adds ways to organize the work and challenges the strength of the evidence.

What the thread adds

  • nfw2 — recommends making decisions one at a time, with only the relevant context, after finding long autonomous runs unproductive.
  • skew-aberration — suggests dividing a project into modules with explicit objectives and tests. They still get useful extra research done by running agents in the background.
  • ainch — reports weak research prioritization alongside useful diagnostic tools, visualizations and experiment scaffolding. An assistant can help even when the person keeps choosing the questions.
  • janalsncm — asks what the model actually suggested and why it failed. Without those details, readers have little basis to judge the author’s standard for ingenuity.
  • wbolt — reports the opposite experience after giving an agent a dedicated environment and time to develop working patterns. That counterexample is anecdotal too.

HN handles are pseudonymous and expose no per-comment scores. This is a slice of HN’s ranking, not a consensus.