Allos AI’s Ayush Bhardwaj treats agents’ shell commands as usability feedback for the tools they call. His measurements come from his own coding-session traces, not a representative study of all CLIs.

Look at what happens after the call

  • In roughly 20,000 recorded agent calls across 14 CLIs, he found repeated output filters such as head, tail, grep and Python. A single pipe is ordinary; repeated copies of the same workaround reveal missing interface options.
  • A normal shell pipeline may hide a failed test’s exit code or clip the actual error. Show a short failure summary at the end and preserve the full log.
  • His session-history CLI frequently returned far more context than needed. He added relevant-message slices, compact search output and batch inspection across sessions.

Teach and test the changed interface

  • Agents kept using an obsolete no-op flag because its skill still recommended it. After the skill changed, usage moved to the new flags; help text and examples are part of the interface.
  • Replay recorded calls on old and new versions. Use what the agent did next as a rough clue to intent, and reserve cases to test the change without overfitting.
  • He reports one agent-mode example that cut displayed output by 59% but saved only about 0.5% of overall input tokens, with no cost or runtime gain. Measure correct answers and turns, not only shorter text.

“The workaround is the bug report.” — Ayush Bhardwaj