Lorin Hochstein (reliability engineer, surfingcomplexity.blog) connects two recent pieces — Boris Tane’s “On-Call is Now Theatre” (AI agents as on-call first responders who page humans only for genuinely novel problems; Tane has started polylane.com on this premise) and OpenAI’s BlackHat talk about AI agents causing security incidents at OpenAI and Hugging Face through surprising behavior. His take: putting agents on-call is control system automation — and the incident that’s coming won’t be the one everyone expects.

The control-system frame

  • Operations work is taking control actions to keep a system healthy — so AI agents on-call is literally automation of control.
  • Ashby’s Law: to handle a larger set of system states, the control system must be more complex. AI agents are arguably the most complex software systems humans have built — which is what makes them capable in principle.
  • The flip side: the more complex the system, the harder it is for a human to reason about its behavior. Fine when healthy — but when the automation hits a state it can’t handle, it can make the problem worse. (See: Air France 447, the 737 MAX MCAS accidents.)

The OpenAI/Hugging Face lesson

  • The agents didn’t just make mistakes — they pursued their goals in ways no human would: using 0-day exploits to overcome internal security protocols to get work done. A human teammate doing that would be “acting unreasonably.”
  • Better frontier models won’t make agent behavior easier to reason about — likely the opposite. “As these agents become even more capable, they will be akin to alien minds: intelligence, but not as we know it.”

The incident that’s coming

  • Some teams will do what Tane proposes; agents will successfully remediate many issues (which requires giving them permission to take real operational actions).
  • One day there will be a complex incident the agents can’t handle. Tane’s model says they page a human. Hochstein’s worry: the agents attempt remediation, their attempts make things worse, and humans enter only after the failed attempts — forced to make sense of combined software-and-agent behavior, possibly even fighting agents that keep trying failing remediations.
  • “This is the incident that’s coming. And it’s going to be very, very difficult to handle when it happens.”