Anthropic’s Lance Martin walks through prompt cleanup, caching, evaluation design, and durable agent infrastructure, followed by an extensive Q&A on AI Engineer.

Remove old instructions that create new work

  • Martin recommends auditing repeated thinking boosters, verification rituals, scratchpads, and contradictory rules when moving to stronger models.
  • His support-ticket example improves after a prompt audit, but remains a small synthetic evaluation with deliberately inserted problems.
  • Preserve task requirements and measured checks while testing whether old reasoning instructions still help the chosen model.

Keep reusable prompt prefixes stable

  • Prompt caching reuses the repeated beginning of a request across agent turns.
  • Put stable instructions and tools before project context, session state, and changing messages.
  • Martin reports up to 72% lower cost on internal evaluations after fixing cache use, with no meaningful performance change.
  • Check cache-hit rates in real traffic; a changing prefix can erase the saving.

Build a test the agent can improve against

  • Martin looks for useful headroom, low run-to-run variance, and sensible scaling with model capability and reasoning effort.
  • Inspect raw traces with a human before promoting examples into an evaluation.
  • Define clear pass/fail rubrics; use pairwise comparisons where choosing between two outputs is easier than assigning a score.
  • Sample for production relevance rather than collecting only one model’s peculiar failures.

Optimize with a held-out check

  • Let the agent edit a scoped surface, such as a prompt or skill, and inspect failures from the training split.
  • Track held-out results separately; training gains with flat test results warn of overfitting.
  • Martin reports one 90% cost reduction while improving quality by searching model, effort, and prompt combinations.
  • Stronger models at lower effort are worth testing; neither that saving nor automatic convergence is guaranteed.

Make long runs recoverable

  • Martin separates the agent controller, execution sandbox, and append-only session log in Claude Managed Agents.
  • The independent log lets a replacement controller recover history after a process or sandbox failure.
  • Vault-managed secrets reduce exposure of credential bytes; permitted actions still need appropriate controls.
  • Preserve the original history behind compacted context so the agent can retrieve details that become important later.

Keep people responsible for the objective

  • People decide what good work means, review suspect data, and check whether improvements transfer to production.
  • Multi-turn agents may need both trajectory checks and final-outcome checks.
  • Small datasets, ambiguous graders, repeated test inspection, and long evaluation cycles all complicate optimization.
  • Martin describes a practical loop: production traces → reviewed failures → offline tests → changes → regression coverage.

“Prompts accumulate debt just like code.” — Lance Martin, 1:37