How To Turn Evals Into A Better Model — Will & Florian (Prime Intellect)
Evals are the bottleneck: the three parts of an evaluation, the hidden variables that skew scores, and when to reach for fine-tuning and RL instead of prompt tweaks.
Evals are the bottleneck: the three parts of an evaluation, the hidden variables that skew scores, and when to reach for fine-tuning and RL instead of prompt tweaks.
ContractSim separates reaching a deal from honoring it. Private earnings can reward missed payments, and changing the instructions changes the result.
Lucas Rocha, an engineer at Brazilian edtech Nova Escola and alum of Hamel Husain's AI Evals course, tells his evals rollout story backwards: from a team that now runs LLM judges with no tickets and no engineer in the loop, back to the messy first spreadsheet and the three barriers — data, methodology, and judgment — that every team hits along the way.
Before worrying about evals, ask if you are building the right thing: AI that dumps an unverifiable answer creates more work. Hamel walks through design patterns — provenance, progressive disclosure, expert signals, modular steps — across three worked examples.
Part 1 of Hamel and Shreya's AI product engineering series: the analyze → measure → improve eval lifecycle, and three mistakes to avoid when using AI to automate evals.
When a product seems impossible to eval, that is usually a sign the product itself is weak. Hamel suggests infusing user-facing signals into the product so users — and you — can assess quality.
Hamel Husain counters the 'evals take too long' objection with one principle: if you can't eval your product, your users can't either — so evals are how you figure out how to convey trust in AI answers.
Hamel Husain explains how to scientifically measure an LLM judge's noise against ground-truth human labels — and why every LLM judge is a black-box classifier with edge cases, so you have to decide how much noise is tolerable.
Hamel Husain argues that ROUGE/BLEU string-similarity scores and 1-to-10 rating scales are holdovers from traditional NLP that make evals harder to act on — nobody knows what a 4.2 means, and binary pass/fail evals are what actually let you ship.
Hamel Husain explains why he converts every 1-to-5 rating scale eval into binary pass/fail checks, scoping each failure into a specific, actionable criterion instead of hiding noise behind middle values.
Maddy Zhang demonstrates skills for interface design, requirements, test-first development, code review and turning repeated engineering work into reusable instructions.
Jane Street's Ron Minsky on strong types, readable tests, and code review that preserves human understanding as agents write more software.
Hamel Husain on evals for non-experts: why generic metric dashboards actively hurt you, how error analysis beats top-down guesswork, what guardrails and MCPs actually are, and the argument that AI competency can never be outsourced.
Wolfe turns his own writing into a training set, tunes a model in Nebius, and shows how messy transcripts and reply-heavy data distort the results.