Upwork’s SWE-Clar asks coding assistants to clarify incomplete requests before building software. It freezes that conversation, then starts two isolated continuations: one answers requirement questions and the other writes code. Assessment questions never enter the coding branch.
The 100 tasks reconstruct Upwork jobs with at least $4,000 in total payments each. They contain 512 requirement questions and separate acceptance tests. Starter workspaces can already contain runnable structure.
- Across nine models, GPT-6 Astra passes every test on 67 of 100 tasks. Its 95% uncertainty interval spans 58–76 tasks, limiting confident rankings.
- GPT-5.6 Luna succeeds on 6% without interaction, 49% after clarification and 75% with complete requirements. All conditions use the same 100 tasks.
- Correct requirement answers still accompany failing implementations in 6.1–7.8% of all 512 requirement units per model. Explaining a rule and implementing it are different tests.
One example makes the distinction concrete. Luna correctly explains that malformed history must raise an error, while valid empty history returns empty results. Its code converts an invalid empty dictionary into a valid empty list and accepts it.
Caveat: The tasks are selected local reconstructions with simulated clients, so their scores do not measure autonomous delivery of live paid commissions. Workspaces use synthetic records and controlled services.
GPT-6 Astra helped construct the tasks, tests and references, with human cross-checking. The MIT-licensed artifact is public; the reported model results remain the authors’ measurements. For spec-driven work, clarified requirements still need executable boundary checks.