VerifierBench tests whether agents can write executable task checkers. An agent is a model running a multi-step task with tools. A checker must recognize different valid solutions, including ones its author never chose.
The authors report a recurring failure: models add restrictions that the task never required, then test examples that already satisfy those restrictions.
- The benchmark, a fixed task set for comparing models, contains 222 tasks and 4,993 hidden cases across tool requests, application states and code changes.
- Across 19 models in the same surrounding software, checkers accepted 21.1–70.2% of valid solutions and rejected 79.2–96.3% of invalid ones.
- In the analyzed nine-model archive, checkers rejecting every tested solution caused 87.9–96.0% of false rejections.
- One checker banned legal Django changes in
__init__.py. Its successful self-test changed another file, leaving that restriction unchallenged.
Verifier4Verifier audits the reference checks through replay, repeatability, source tracking, label consistency, trivial baselines and deliberate mutations. Source defects excluded 45 of the initial 267 tasks.
Caveat: The detailed failure cases were selected to explain mechanisms. They do not measure prevalence across every configuration, and the suggested testing improvements remain unevaluated.