VerifierBench: AI-written checkers reject valid solutions

VerifierBench tests whether agents can write executable task checkers. An agent is a model running a multi-step task with tools. A checker must recognize different valid solutions, including ones its author never chose. The authors report a …

Published:  · Added:  · 1 min

Automatic prompt tuning helps local math tests but can hurt question answering

A two-model GEPA and SkillOpt study separates answer correctness, formatting, server failures, waiting time and token use.

Published:  · Added:  · 2 min

Combining two instruction files can make a model ignore one

SkillClash isolates failures after both skills are loaded. A correct combined example repairs much of the lost rule compliance.

Published:  · Added:  · 2 min

How To Get Your Team To Invest In Evals — Hamel Husain

Review 50–100 traces, find recurring failures, and show how fixes reduce errors. Build support for evaluation through visible product improvements.

Published:  · Added:  · 1 min

How To Decide Who Owns AI Evals — Hamel Husain

Keep evaluation inside the development loop, give a trusted domain expert the final call on quality, and build annotation tools around their work.

Published:  · Added:  · 2 min

Scale and Reflection repair coding tests and grade patches in clean containers

SWE-Bench Pro V2 separates bad instructions from evaluation exploits. Its strongest systems already approach the repaired test set's ceiling.

Published:  · Added:  · 2 min

Arcade measures when searching for tools cuts the bill

Arcade compares finding tools on demand with listing 100 tools in every request. Savings depend on search frequency, prompt caching and how the client handles large results.

Published:  · Added:  · 2 min

Munin lets research groups question papers on their own servers

Munin combines local paper search, full-text reading and citation checks with a rerunnable evaluation suite. Its strongest result permits web access and depends on a private reference corpus.

Published:  · Added:  · 2 min

SchemaShield-Lite repairs broken requests after tool descriptions change

Validation and a constrained retry recover tool-request errors in controlled Qwen tests. The strongest results depend on resolving a usable tool name, with weaker gains outside that model family.

Published:  · Added:  · 2 min

TokenEconomizer learns where coding assistants waste money

An external controller learns short repository instructions, hides unused tools and redirects stalled work. Reported savings include its own model requests and prompt-cache discounts.

Published:  · Added:  · 2 min

Can One Model Run Your App And Evals?

Hamel Husain argues for validating AI judges against human labels before adding another model provider: measure precision and recall, then optimize cost.

Published:  · Added:  · 2 min

FEB-30 checks stored facts alongside an agent's answers

A memory test follows facts as they change, exposing errors that answer scores can hide or misdiagnose.

Published:  · Added:  · 2 min

ToolBottleneckBench traces agent failures to execution limits

A larger tool menu looked like worse tool use until researchers raised the model's response allowance.

Published:  · Added:  · 2 min

MAI-Gym trains and tests screen-operating agents through one loop

A shared interaction loop and executable task checks matter more than the nine-billion-parameter model's headline scores.

Published:  · Added:  · 2 min

How to Ace AI Eval Interview Questions (for Engineers & PMs)

Hamel Husain's interview playbook: audit imperfect data, establish a baseline, run honest comparisons, challenge vague metrics, and explain product trade-offs.

Published:  · Added:  · 2 min

Can Claude Build Your Evals? — Hamel Husain and Isaac Flath

A live test finds real failures in 100 leasing-assistant conversations, but premature test design, brittle grading and a poor review interface undermine the workflow.

Published:  · Added:  · 2 min

DeRef helps tool-call judges check values they must derive

A model can reject the right tool argument when its source is missing, or when using it requires an inference.

Published:  · Added:  · 2 min

Memory Echo: When an Agent Mistakes Repetition for Evidence

Copies of one old claim can crowd out a correction. A memory study tests source tracking, and shows when filtering makes answers worse.

Published:  · Added:  · 1 min

7 Most Common AI Eval Mistakes (From Teaching 5K+ Engineers & PMs)

Hamel Husain's seven eval pitfalls: imaginary failures, generic scores, unvalidated judges, opaque interfaces, rating scales, the wrong reviewers, and over-automation.

Published:  · Added:  · 2 min

TurnCost: Count Cached Input Before Cutting Agent Costs

An analysis of 600 coding-agent runs finds that repeated steps explain much more cost variation than tool-output size, under simulated cached pricing.

Published:  · Added:  · 2 min

How To Test AI Agents With Simulations

Ben Hylak explains change-focused agent testing: replay production traces in a reconstructed environment, inspect behavioral differences, and validate the simulation before trusting it.

Published:  · Added:  · 3 min

Is Opus 5.5 Better at Writing? — Hamel Husain and Isaac Flath

A blind, small-sample writing comparison finds some improvements over Opus 5, but generic phrasing, misplaced detail, and bot-like replies survive.

Published:  · Added:  · 2 min

How to Do Error Analysis When You Can't Look at the Data — Hamel Husain

If customer traces are sensitive, seek limited authorized access, recruit cleared domain experts, or carefully redact; synthetic examples are a last resort.

Published:  · Added:  · 1 min

I Ranked Every AI Eval Concept From Worst to Best — Hamel Husain

Hamel ranks 28 evaluation practices: inspect real failures, use human-validated product-specific checks, and stop mistaking generic dashboard scores for progress.

Published:  · Added:  · 2 min

Why We Tell People to Read 100 Traces — Hamel Husain

The 100-trace target is a starting cue, not a magic sample size; keep inspecting until examples stop teaching you something new.

Published:  · Added:  · 1 min

Your AI Tests Are Supposed to Go Stale — Hamel Husain

Refresh AI eval datasets through error analysis; use product metrics, not static tests, for long-term comparison.

Published:  · Added:  · 1 min

How to Evaluate OCR for AI Agents — Isaac Flath

A document-agent answer can be wrong because OCR misread the source, not because the LLM hallucinated. Trace the failure back before changing the model.

Published:  · Added:  · 3 min

Do You Need a Rubric Before Annotating Data? — Hamel Husain

Writing an evaluation rubric before examining real traces can lock in the wrong criteria. Hamel Husain argues for open-ended error analysis first, then rubrics and reference answers grounded in observed failures.

Published:  · Added:  · 1 min

Why 90% Agreement Is a Bad Sign for Your LLM Judge — Hamel Husain

Overall agreement can make a useless LLM judge look accurate when failures are rare. Hamel Husain shows why product teams need class-specific error metrics instead of one reassuring percentage.

Published:  · Added:  · 1 min

Why AI Benchmarks Don't Tell You If Your Product Works — Hamel Husain

Model benchmarks answer how a model performs in general. Product evals answer the operational question that builders actually own: whether their specific system does the right thing on their users’ tasks.

Published:  · Added:  · 1 min

The Best Jev Use Cases for AI Coding — Cole Medin

Four places a fast decision model can sit inside an AI coding workflow: screening tool calls, playtesting games, driving browser tests, and routing GitHub work to the right process and model.

Published:  · Added:  · 3 min

8 Claude Code Skills for Better AI Evals — Hamel Husain

Hamel Husain tours an open-source eval toolkit built with Shreya Shankar: discover real failure modes before writing tests, audit weak eval pipelines, calibrate LLM judges against human labels, and build review interfaces tailored to your traces.

Published:  · Added:  · 3 min

Why the Prompt Matters Less Than the Context — Hamel Husain

Hamel Husain and Isaac Flath compare three writing setups and get a messier result than expected: the elaborate anti-slop prompt, a plain-writing prompt, and no system prompt each win examples. Their small test points away from prompt mythology and toward context, real outputs, and editing effort.

Published:  · Added:  · 2 min

Designing reliable AI agents — Maddy Zhang

A customer-support example walks through tool design, selective memory, testing, tracing and safeguards for agents that take real actions.

Published:  · Added:  · 3 min

Behind Navigator n2: How Yutori Made Computer-Use Agents 10x Cheaper — Skanda Vaidyanath

Yutori founding researcher Skanda Vaidyanath on training a 27B computer-use model: trajectories that weave GUI, code and tool calls, a data flywheel with agents in every loop, rejection sampling then async RL, and stress-testing verifiers for reward hacks.

Published:  · Added:  · 6 min

How to Get Agents to Do Evals Well — Hamel Husain

Hamel Husain's answer to how you get an agent to do evals well starts with the human, not the agent: understand what good looks like first, because a published skill is generic by design and only pays off once you customize it.

Published:  · Added:  · 2 min

Opus 5.5 and the gap between token price and task cost — Matt Wolfe

Wolfe compares Opus 5.5 with GPT-6 Sol and Luna through launch scores, animation demos and the bill for completing a task.

Published:  · Added:  · 2 min

Building a Personal Agent That Actually Works and Sounds Like You — Ahmet İlten, Sentience

Sentience founding engineer Ahmet İlten on the four principles behind their personal-agent product: complexity masks the real problem, start from the user experience and work backwards, keep the system prompt visible as one monster you can stare at, and run evals before every change.

Published:  · Added:  · 6 min

Evals for AI Writing — Hamel Husain and Isaac Flath

Three model-written X posts about the same document-QA pipeline, judged by reading the drafts aloud. The ones that won carried a real number and one fewer fact than the rest.

Published:  · Added:  · 3 min

Putting Claude's "AI Slop" Solution to the Test — Hamel Husain

Anthropic's fix for mannered prose is itself written in mannered prose. Hamel Husain and Isaac Flath run it side by side against a plain-writing prompt and a no-prompt baseline on the same question, then judge the drafts by reading them out loud.

Published:  · Added:  · 4 min

Why The Future Of Content Is Born Multilingual — Olga Beregovaya

Smartling's VP of AI Olga Beregovaya on why one-to-one translation is dying, generating target content straight from briefs instead of a source, why models hallucinate more in under-resourced languages, interpretation outliving translation, translateese and modelese, the build-versus-buy line, and the 12 metrics her team scores translation quality with.

Published:  · Added:  · 9 min

Aligned to whom? — Ryan Lopopolo

A short essay aimed at people building agents: your confidence in them is mostly a statement about your own expertise, and the one thing you're actually trusting where you're not the expert is the model's priors — which are bad.

Published:  · Added:  · 3 min

Trying the New Claude Eval Tool — Hamel Husain (live)

Hamel Husain streams Anthropic's new `claude plugin eval` against a plain-writing skill and calls it a failure: the eval it generates is one ablation against the skill's own rules, the numbers have no provenance, and it asks you for exactly the error analysis that is the hard part.

Published:  · Added:  · 5 min

Building an AI-video detector exposes the hard part of agent work — Matt Wolfe

Wolfe’s detector takes hours of agent work, a specialist API and repeated tests. The polished interface arrives well before reliable judgments.

Published:  · Added:  · 2 min

AI Evals: A Hands-On Guide for Product Teams — Teresa Torres

Teresa Torres on error analysis, four eval types, and baseline-driven experiments for AI products and workflows.

Published:  · Added:  · 2 min

How To Build And Evaluate Search Agents — Nandan Thakur

Nandan Thakur (BEIR, MIRACL, TREC-RAG; now Microsoft Research India) on agentic search: BrowseComp-Plus as a reproducible eval, why the retriever moves accuracy more than the model does (60% → 88% on the same LLM), ORBIT's zero-budget 20k-question synthetic data pipeline, and HawkEye trajectory analysis for the leaderboard trap where 95% accuracy hides 210 search calls.

Published:  · Added:  · 7 min

How to Stop Building Products Nobody Wants — Teresa Torres & Hamel Husain

Teresa Torres (Continuous Discovery Habits) on why 'talk to your customers' fails for engineers, story-based interviewing as the reliable feedback loop, the Opportunity Solution Tree, how AI synthesis and her Vistily partnership put evals at the center of discovery, and a three-week eval war story about upstream errors and counterbalancing judges.

Published:  · Added:  · 6 min

Stop Shipping AI Nobody Can Verify — Hamel Husain

Hamel Husain on why 'it's hard to eval' is a product smell, verification-first design (the data agent, workers' comp report, and PE lesson planner redesigns), sharing skill shapes not skills, transition failure matrices, and why data science is more valuable than ever in the agentic era.

Published:  · Added:  · 4 min

How To Build Better AI Evals with Claude Code — Shreya & Hamel

Shreya Shankar and Hamel Husain walk through the practical evals loop: top-down vs bottom-up criteria, an open-source Claude Code "error discovery" workflow that turns your taste into a rubric, and why automated eval tools get you a baseline but never replace reading your own data.

Published:  · Added:  · 5 min

The Benchmarkpocalypse — Dan Luu

Dan Luu let a coding agent build a regex engine for a month. It claimed a 40% speedup, then fell apart on data it had never seen — the benchmarkpocalypse.

Added:  · 2 min