Fei Ding audits a mismatch in coding tests: instructions describe behavior, but the grader requires exact names or paths found only in the reference implementation. A solution can do what was requested and fail because it chose a different constant name.

The detailed study examines SWE-Milestone, a fixed collection of software changes used to compare coding assistants. It covers seven repositories, 98 milestones and 35 published runs. The audit looks for requirements absent from the instructions, starting code and established naming conventions.

  • Twenty-five milestones contain hidden names or paths. These affect only 0.6% of required tests overall, but can prevent an entire milestone from passing.
  • One name can block a package. Event-streaming tests require valueSSE and headerAccept, although the task never names them. Missing either can prevent all 84 required package tests from running.
  • Affected units fail more often. Their failure rate was 80.6%, versus 48.4% for comparable units testing new code. Renaming one hidden identifier in reference code prevented compilation in 126 of 128 Go cases.
  • The overall ranking largely survives correction. Among 28 leaderboard runs, removing problematic tests with the broader filter changed score rankings by at most two places. Repository-level scores moved more.

The last result matters. The paper identifies a measurable defect without showing that the leaderboard’s ordering collapses. A hidden dependency also does not guarantee failure: some assistants happen to reproduce the expected name.

The proposed repairs are concrete. State interfaces that tests require, or revise tests that depend on private implementation details. Check reference solutions in different permitted milestone orders, isolate compilation failures and publish per-test results.

Caveat: The failure-rate comparison is observational, and the controlled renaming experiment covers Go only; proposed specification addenda still need fresh coding runs.

The paper reports analyses of public benchmark artifacts and existing runs, rather than a new trial of agents given better instructions. For spec-driven development, an exact interface belongs in the specification whenever an existing caller or acceptance test requires it.

Daily coverage