How To Turn Evals Into A Better Model — Will & Florian (Prime Intellect)

Evals are the bottleneck: the three parts of an evaluation, the hidden variables that skew scores, and when to reach for fine-tuning and RL instead of prompt tweaks.

Published:  · Added:  · 3 min

Models can profit from agreements by breaking their promises

ContractSim separates reaching a deal from honoring it. Private earnings can reward missed payments, and changing the instructions changes the result.

Published:  · Added:  · 2 min

How To Build AI Evals — Lucas Rocha

Lucas Rocha, an engineer at Brazilian edtech Nova Escola and alum of Hamel Husain's AI Evals course, tells his evals rollout story backwards: from a team that now runs LLM judges with no tickets and no engineer in the loop, back to the messy first spreadsheet and the three barriers — data, methodology, and judgment — that every team hits along the way.

Published:  · Added:  · 5 min

How To Build AI That Earns User Trust — Hamel Husain

Before worrying about evals, ask if you are building the right thing: AI that dumps an unverifiable answer creates more work. Hamel walks through design patterns — provenance, progressive disclosure, expert signals, modular steps — across three worked examples.

Published:  · Added:  · 3 min

How to Automate AI Evals (Correctly) — Shreya Shankar

Part 1 of Hamel and Shreya's AI product engineering series: the analyze → measure → improve eval lifecycle, and three mistakes to avoid when using AI to automate evals.

Published:  · Added:  · 4 min

How to Eval an AI Product That Seems Impossible to Eval — Hamel Husain

When a product seems impossible to eval, that is usually a sign the product itself is weak. Hamel suggests infusing user-facing signals into the product so users — and you — can assess quality.

Published:  · Added:  · 1 min

If You Can't Eval Your AI, Your Users Can't Trust It — Hamel Husain

Hamel Husain counters the 'evals take too long' objection with one principle: if you can't eval your product, your users can't either — so evals are how you figure out how to convey trust in AI answers.

Published:  · Added:  · 1 min

Why There's No LLM Judge You Can Trust 100% of the Time — Hamel Husain

Hamel Husain explains how to scientifically measure an LLM judge's noise against ground-truth human labels — and why every LLM judge is a black-box classifier with edge cases, so you have to decide how much noise is tolerable.

Published:  · Added:  · 2 min

Why AI Rating Scales Make It Harder to Ship — Hamel Husain

Hamel Husain argues that ROUGE/BLEU string-similarity scores and 1-to-10 rating scales are holdovers from traditional NLP that make evals harder to act on — nobody knows what a 4.2 means, and binary pass/fail evals are what actually let you ship.

Published:  · Added:  · 2 min

Why the 1 to 5 Scale Is Where AI Evals Break Down — Hamel Husain

Hamel Husain explains why he converts every 1-to-5 rating scale eval into binary pass/fail checks, scoping each failure into a specific, actionable criterion instead of hiding noise behind middle values.

Published:  · Added:  · 1 min

Five reusable Claude Code workflows — Maddy Zhang

Maddy Zhang demonstrates skills for interface design, requirements, test-first development, code review and turning repeated engineering work into reusable instructions.

Published:  · Added:  · 3 min

Now More Than Ever: Building Reliable Software in the Age of Agents — Ron Minsky

Jane Street's Ron Minsky on strong types, readable tests, and code review that preserves human understanding as agents write more software.

Published:  · Added:  · 4 min

The AI Testing Framework Every Business Needs (But Few Use)

Hamel Husain on evals for non-experts: why generic metric dashboards actively hurt you, how error analysis beats top-down guesswork, what guardrails and MCPs actually are, and the argument that AI competency can never be outsourced.

Published:  · Added:  · 4 min

Fine-tuning a writing style starts with cleaner data — Matt Wolfe

Wolfe turns his own writing into a training set, tunes a model in Nebius, and shows how messy transcripts and reply-heavy data distort the results.

Published:  · Added:  · 4 min