Maddy-Zhang

Maddy Zhang uses a customer-support agent to explain the engineering behind reliable multi-step work. This 11-minute overview covers nine layers, from the agent loop to production controls. The loop and model choices (00:51) An agent runs a loop: plan, call a tool, inspect the result, then finish or continue. Checking an order takes fewer steps than canceling and refunding it. Every additional step creates another opportunity for failure. Keep workflows short and check intermediate results. Model routing assigns different models to different tasks. Zhang suggests small, fast models for routine classification and stronger models for ambiguous complaints, balancing quality against cost and response time. Tools with clear boundaries (02:56) Give each tool a precise name, description, input format and structured errors. A larger tool menu also makes choosing the right tool harder. Separate reading an order from issuing a refund. The latter moves money and needs stronger checks. MCP, the Model Context Protocol, standardizes connections between agents, tools and data sources. State, memory and retrieval (05:21) State records the current task: order ID, current step, customer confirmation and whether the refund succeeded. Memory holds longer-lived information, such as customer history and past conversations. Store task state explicitly. Retrieve only the memory needed for the next step rather than repeatedly sending the entire history to the model. RAG, answering with retrieved documents, lets the agent fetch the relevant refund-policy passages. Search quality depends on how documents are split, filtered, ranked and kept current. If retrieval supplies the wrong policy, a fluent answer can still apply the wrong rule. Test retrieval as well as the final response. Start with a defined workflow (07:24) A known support process can follow a fixed sequence: classify, retrieve, act, confirm and respond. Add open-ended planning when the task actually needs it. Use multiple agents when work separates into distinct specialties; each handoff introduces another place to lose context. Tests and traces answer different questions (08:17) Evals are repeatable tests of model behavior. Include straightforward requests, ambiguous cases, tool failures and upset customers; rerun them after changing a prompt or model. Check tool choices and arguments, not just whether the final answer looks plausible. A successful API call with the wrong order ID is still a failure. A trace records one run step by step. Log model and prompt versions, tool inputs and results, errors, fallbacks, cost and response time to find where a real incident began. Put action controls in application code (09:53) Match permissions and approval requirements to the damage a mistaken action could cause. Looking up an order and changing billing should have different safeguards. The model proposes an action; application code validates it. Require human approval for the highest-risk operations. Treat customer messages and retrieved documents as untrusted data. Prompt injection can hide instructions inside them; restrict tool permissions and keep business rules outside the model’s discretion. “Your business logic should live in your application code, never in whatever the model happened to infer this time.” — Maddy Zhang, 10:24 ...