Haichao Zhu and colleagues test what coding assistants follow when existing code conflicts with a new requirement. Their paper, Precedent or Intent?, separates passing functional tests from choosing the implementation the developer intended.
The study runs 5,625 trials across four assistants and six implementation choices. Each choice admits two functionally acceptable answers, such as returning an identifier as a number or a string.
- Existing code wins over defaults. When every endpoint demonstrates the opposite of an assistant’s usual choice, the assistant follows precedent in 359/360 trials.
- Nearby examples matter more. Three supporting files for the resource being changed yield 347/360 target adoptions. Three files for a neighboring resource yield 13/360.
- State priority explicitly. A written rule achieves both functional correctness and the intended choice in 59/60 trials, versus 38/60 with an example alone. This comparison covers two assistants and two choices.
- The effect extends beyond the toy repository. Across four public repositories in Python, Go and TypeScript, a rule in the assistant’s native instruction file raises joint success from 0/120 to 117/120.
A migration instruction should say which old convention is being superseded and why. An extra example leaves the assistant to resolve the conflict itself. A targeted test should verify the intended representation as well as general functionality.
Caveat: These are author-run tests of narrow implementation choices. They do not measure long-term maintainability, and native-file experiments supply rules through the wrapper rather than testing whether assistants discover them.