Tanmay Sah and Dolly Sah present their UndoBench and EvoUndo research on recovery after coding-agent failures.
A timeout is not necessarily a failed action
- An external service may create a ticket even if its acknowledgement never reaches the agent. Retrying blindly can create another ticket.
- Git can restore repository changes, but runtime configuration, database writes, deployments and notifications can require separate recovery steps.
- The response depends on what actually happened: retry, restore saved state, compensate for an external effect, or stop and escalate.
Check recovery as well as capability
- EvoUndo captures evidence of the original state before a change and tests a recovery plan in a test environment.
- An independent verifier checks the resulting state; running an undo command alone is not proof that recovery succeeded.
- The speakers report that 197 of 478 capability-improving changes in their experiment had recovery defects, about 42%. This is a result for their tested setup, not a universal failure rate.
- They describe improved recovery after providing richer recovery operations. These are research results, not proof that every real-world side effect can be reversed.
Why it matters
- Treat recoverability as an acceptance condition for agent changes, alongside the capability they add.
- Save the original values and test the actual recovery procedure before relying on it in production.
- Irreversible or unverifiable actions still need a human escalation path.