Elmira Onagh and Maleknaz Nayebi tested how explicit requirements shape ChatGPT-4o code. A detailed list of behavioral obligations helped much more than a lightweight specification template.

Their controlled study assigned 72 participants to three conditions, with 24 in each. Participants implemented one of two small Python tasks, then revised it after a requirement change. Each task had 33 hidden tests.

  • Initial failures: Mean failed tests fell from 21.54 without a required specification to 19.67 with a lightweight template and 11.38 with explicit obligations. The authors report reductions of 8.7% and 47.2%.
  • What became explicit: The detailed template required functional requirements, constraints, invariants, forbidden behaviors, edge cases and tie-breaking priorities. An invariant is a rule that must always hold.
  • Complete correctness stayed rare: Only one of the detailed group’s 24 submissions passed every test. Neither other group produced a fully correct submission.
  • Change introduced a tradeoff: Four detailed-group submissions regressed, versus two in each other group. The detailed group lost less previously passing behavior than the lightweight group.
  • The adjusted result is weaker: The detailed-versus-unstructured failure model had an estimated rate ratio of 0.48, but its corrected p-value was 0.205. It lost statistical significance after adjustment for multiple comparisons.

The replication package supplies data, hidden tests and analysis notebooks. These are author-reported results, with no independent replication established here.

Caveat: The treatment includes planning and reflection, not just better wording. Small ChatGPT tasks do not establish reliability in autonomous production-repository work.

The practical starting point is to connect each required or forbidden behavior to a test, then keep both updated as requirements change. The study supports investigating that workflow; its rare perfect submissions show why a specification still needs verification.

Daily coverage