Sachin Gupta’s AgentFaultBench tests whether recovery instructions help agents, models running multi-step tasks with tools, finish after errors. LangGraph and CrewAI operate on a real isolated GitLab installation, with controlled failures inserted around their program requests.

The tasks supply the exact issue and report text. This tests execution and required procedure, rather than the ability to write a report from source documents.

  • Primary advice hurts GPT-4o mini: Across a 1,944-planned-run comparison, guided GPT-4o mini completes 124/216 tasks, or 57.4%. Neutral feedback completes 142/216, or 65.7%, with the same conditional opportunity to continue.
  • The observed failure is stopping: All twenty neutral-only wins involve temporary rate limits. Guided agents receive feedback, request no further tool action and report incompletion. The bundled message prevents isolating one causal sentence.
  • Correct state differs from full success: Scoring separates files and issues from required source reads, genuine verification and final JSON reporting. Among 1,879 jointly known outcomes, 318 have correct state but fail the full requirements.
  • A later policy helps in a different experiment: On six related new instances, a revised execution-information/advice bundle yields 161 correct states plus one unknown in 192 guided runs, versus 142/192 neutral. Missing-outcome bounds put the gain between 9.90 and 10.42 percentage points.
  • Restored issue IDs cannot explain the entire gain: Removing IDs gives 47/48 successes versus 46/48 with them. Other execution information remains, so this still does not isolate advice wording.

The later policy includes automatic recovery and uses different feedback from the primary experiment. It does not erase the earlier adverse result. A small wording follow-up matches neutral completion but does not beat it.

Caveat: Code and complete execution records are unavailable publicly, and no independent human validation was collected. The constructed GitLab workflow does not establish a generally effective recovery policy.

The engineering lesson is to test advice against a neutral continuation and a bounded retry, then check actual application state separately. Confirm that the task is feasible under its retry limits and that the tested feedback was actually delivered.

Daily coverage