How to Automate AI Evals (Correctly) — Shreya Shankar
Shreya Shankar (Stanford CS professor, co-creator of the AI evals course with Hamel) kicks off the 12-part AI product engineering series. 27 minutes on Hamel Husain’s channel. Why this matters Output quality is the biggest barrier to productionizing agents (LangSmith annual report) — and figuring out how to evaluate models is genuinely hard Vendors (LangChain, Braintrust, Arize) are selling end-to-end automated eval tools: point an LLM at your traces, it finds and fixes your bugs The catch is epistemic: what “good” means lives in your head, not in the traces — if a tool could fully fix your product, it could fix everyone’s, and there’d be nothing left to differentiate yours AI’s real job: help you express and apply your judgment faster, not replace it The eval lifecycle (analyze → measure → improve) Error analysis — the hardest step: take traces and find failure modes. No perfect definition of “mistake” (you can’t define slop, but you know it when you see it) Measure — how prevalent is each failure mode? Pareto applies: ~80% of issues come from ~20% of failure modes — prioritize those Improve — fix the product: prompt instructions, model switch, fine-tuning. Iterate forever AI is weak at the front (taste-specific error analysis) and strong at the back (measurement, prompt optimization, hill-climbing). ...