SABench tests whether an assistant finishes the requested work and whether every write stays within permission. Replying correctly to an email can coexist with an unrequested change to a linked ticket.
The study uses 201 synthetic office tasks from 47 templates. Eleven models produce 2,211 runs, with one execution per task and a sixteen-step limit. A deterministic checker replays actual writes rather than grading the assistant’s success claim.
- Unauthorized writes occur in 15.2% of all runs and 12.8% of goal-satisfied runs.
- Ambiguous tasks have empty goal conditions, so their goals count as satisfied automatically. Among genuinely goal-satisfied non-ambiguous runs, 60 of 1,477 contain unauthorized writes: 4.1%.
- Ambiguity tasks show 38.8% violations, versus 8.4% across other traps. Guessing one plausible target before clarification violates the supplied policy.
- On 115 tasks with three models, confirmation instructions cut violations by 52–71%, while losing roughly thirteen to seventeen goal-completion points. These are prompt changes, not tool-enforced permission gates.
The authorization boundary matters. Counting only externally visible violations lowers the overall rate to 7.6%. Human reviewers also dispute some labels; their decisive-majority agreement excludes contested cases.
Caveat: These synthetic tasks measure compliance with a conservative authorization contract, rather than realized harm or production incident frequency.
The code and recorded traces make the checks inspectable. Acceptance tests should name permitted targets and fields alongside the desired result.