Ben Swerdlow built a version of StarCraft: Brood War that you can only play through an AI agent, as an experiment to play with friends. The friends did better than he expected, and when he asked why, they said they hadn’t done much — they had told their agent to attack, and the agent had built a small army and carried out the attack. That made him wonder how far the agents could get on their own.

Brood War Bench is the answer. Nineteen model-and-effort configurations — Codex variants, Claude Fable, Opus, Sonnet and Haiku, and Grok 4.6 at three effort levels — played a round robin in which every configuration faced every other. The matches ran in parallel on rented machines, with the game data and each agent’s logs saved.

The scoreboard

  • Codex Astra at its highest effort setting won every game it played: 18–0, at $10.54 a game. Astra at medium effort went 16–2, Claude Fable 15–3, Claude Opus 5 12–6.
  • The bottom is worse than “not great.” Claude Haiku lost all sixteen games it played, averaging 0.3 actions per minute. Grok 4.6 at low effort also went 0–16.
  • Actions per minute — a rough measure of how often a player does anything at all — ranged from 0.3 to about 48 across the table.
  • Cost did not track skill. The cheapest configuration came in at $0.16 a game and the most expensive at $21.07, and both ends of that range sit below the leaders.
  • The author’s own summary: none of the models “played beyond a beginner level,” Codex Astra is the clear leader, and Grok models “are not smart enough to play Brood War yet.”

Why they lost

  • Older models played a real-time game as if it were turn-based, and so “got destroyed while they were thinking.” Newer models were more aware of what deliberation costs them in game time — which, the report suggests, may be why some lower-effort settings outperformed higher-effort ones.
  • Disruption came before production. Codex often sent a single worker unit across the map to harass, and it worked far better than it should have, because the opposing agent would spend dozens of seconds deciding what to do about one probe. Sustained production was much weaker: delayed technology, units trickled one at a time into defended bases.
  • The coordination failure was inside the agent, not between agents. Codex frequently spawned separate subagents for economy, army production and army control, and they barely talked to each other — the army subagent kept sending each new unit straight into an attack, unaware of the larger force the other agents were planning. The author compares it to the classic beginner mistake of attacking in drips instead of waiting for a critical mass.
  • Grok spent the game between actions rather than playing it. One game logged 11,138 reasoning tokens and six batches of commands across 43 minutes, and never produced a combat unit.
  • Claude Fable, the author notes with some affection, actually tried to play: it built an economy, climbed the tech tree, and in one game reached Mutalisks and won. Ambition did not guarantee execution.

Why it still matters

The verdict is harsher than the leaderboard alone suggests: a beginner who rushed with basic units would win every one of these games. But the failure modes are the interesting part, because none of them are exotic. Thinking time competing with acting time, a group of subagents that never share a plan, a unit-sending cadence too slow to matter — that is roughly the same list you would write after watching an agent work any long task, minus the spaceships.

Swerdlow’s own conclusion is optimistic in a specific way: he calls the benchmark “nowhere near exhausted,” and says watching the agents play made him more excited than he has been in a while.

What the thread adds

The 121-comment thread on Hacker News splits between people asking how the harness actually works, people arguing about what the game proves, and a large amount of Brood War nostalgia.

  • gadtfly — asked the question the report does not answer: did the agents play “by looking at screenshots and sending clicks, or was there other mediation/symbolization?” and noted that this changes what the result means. Commenters who say they know the harness fill it in: callmekit says the agents get a purpose-built API rather than the game interface, one that also hands them things a human player would not see — invisible units are reported, and a unit can be selected from underneath other units. loeg agrees the games were played in real time rather than in turns.
  • nullc — the sharpest objection to the scores: an LLM’s StarCraft ability may partly be recall, since “there will be a HUGE difference in performance that depends purely on this game being in the LLM’s training.” A better test, in their view, is having the models write a game engine under a strict compute budget — “a skill that is more general and they should all have.”
  • aswegs8 and faeyanpiraat — point to a counterexample to the beginner-level ceiling: a reinforcement-learning Brood War bot that is currently beating strong players on the ladder. ericpruitt supplies the qualifier — it is not limited by human actions-per-minute and is likely using map hacks, “both of which make the bot significantly less impressive.”
  • alembic_fumes — argues the valuable axis here is not skill but real-timeliness, since the question people actually face is whether a slower, stronger model beats a faster, dumber one. They want Google’s Gemini models and a Cerebras-served open model added, and eventually a sweep across workloads from speed chess to turn-based games with clocks.
  • GodelNumbering — independent corroboration on a smaller board: their friend’s GoBench pits LLMs against Go engine anchors on 9×9 boards and shows the same ordering at the top, with Astra ahead of everything else.
  • mcteamster — offers the thread’s best metaphor for agent orchestration, borrowed from the game’s three factions: Protoss as expensive frontier models you micromanage for the hardest tasks, Terran as a team of specialists you delegate well-defined work to, Zerg as swarms of cheap custom agents you evolve for speed and cost.
  • Svenstaro — actually ran it against a local model, and hit an error the report never mentions: “another match is already active,” apparently because only one match can run at a time.

The question the report leaves open

The report describes the matrix and the observations in detail, but never says how the agent perceives and acts — whether it reads a game API or a screen, and what a human player would be missing in the same position. gadtfly asked directly; the answers came from other commenters rather than the report. The follow-up question those answers raise — how much of this is a model reasoning about StarCraft versus remembering it — was raised by nullc and left unresolved by everyone.

A note on reading comments as evidence: HN handles are pseudonymous and the site publishes no per-comment scores, so the ordering here is HN’s own ranking, not a vote. The claims about the harness, the map hacks and the training-data hypothesis come from commenters, not from the report. This is a slice of the thread, not a consensus.