Aiden Sochowski, a founding engineer at Runlayer, describes monitoring more than 300 daily agents across a roughly 50–60-person company in this 17-minute AI Engineer talk.

Inventory the actions, not just the chat windows

  • A central gateway records sessions and tool calls across agents, local clients and connected services.
  • Map agents by task, connected systems and potential damage if they misuse their access.
  • An agent with a narrow job can still inherit broad permissions from a person’s account.

Watch the whole session

  • A support email can carry instructions that try to turn calendar or customer access into a data leak.
  • Inbound screening, outbound data checks and restrictions on sensitive sources provide different defensive layers.
  • A code-review agent may be pressured through comments into approving a change it previously questioned.
  • Runlayer uses a session-aware classifier to assess whether tool calls still serve the agent’s assigned task.

Reduce permissions and shared-data exposure

  • Historical tool use can suggest access to remove, especially for narrowly defined agents.
  • A recruiting bot may need selected applicant records without access to every meeting or negotiation email.
  • One compromised agent can write poisoned content into a system that other agents later read.
  • Map those connections and tightly limit where agents can write.

Debug the bill with the same traces

  • A recruiting team’s generated MCP server, connecting agents to a vendor API, repeatedly failed on formatting issues.
  • Agents tried workarounds, consuming more input and output tokens on recurring runs.
  • Monitor errors, test tool-server code and apply rate limits before retries become routine spending.
  • Sochowski also reports a model change associated with a roughly 100-fold cost increase within a week.
  • Budgets, sensible defaults and task-specific model evaluation help contain these surprises.

Controls still need evaluation

  • Caveat: the talk presents vendor experience and illustrative attack paths, with no classifier accuracy, false-positive rate or measured security reduction.
  • Behavioral history can guide permission review; hard authorization boundaries still need to enforce what an agent may do.

“Agents are just software, right? They’re just, you know, plumbing data between one source to another.” — Sochowski, 11:55