Shervin Shahidi and colleagues describe AutoRefine, an internal Meta system that rewrites a code-review model’s instructions. A teacher model sees the review outcome and explains labels it can support from the judge’s inputs. It sets aside examples whose outcome those inputs cannot explain.
The prompt optimizer learns from disagreements and selects its final instructions on separate validation data. The paper reports several distinct tests:
- On an 85-task test set with execution-based labels, a rubric judge’s chance-corrected agreement rises from 0.36 to 0.60. It also sees reference patches; about 5% of tasks per replay are excluded for errors or timeouts.
- That experiment uses the reliable labels directly, without the teacher. Its gain cannot be credited to teacher filtering.
- Without reference patches, the teacher abstains on 12 of 13 validation failures. Their defects depend on hidden tests rather than the visible changes.
- On 17 independent examples from the current production workflow, agreement rises from 0.16 to 0.51. On 16 older examples it falls from 0.50 to −0.02.
- Filtering deliberately corrupted review outcomes cuts label noise from 24.9% to 20.4%, but retains 19 of 22 failing changes incorrectly labeled successful.
Abandonment can mean a defect, a duplicate, superseded work or an unpublished draft discarded during internal triage. The label’s meaning depends on the workflow. A teacher that preserves apparently good changes can leave false approvals intact even while reducing overall noise.
Caveat: The production test is small, comes from one internal agent and uses merged or abandoned changes as imperfect substitutes for code quality.
These are the authors’ industrial results, not independent replication. For teams tuning evaluators, frequent abstention can signal missing evidence rather than a weak prompt. Keep a current independent test set, and audit false approvals separately from aggregate agreement.